From b75dc89a93793ca1c93fce2a16e15ee4496b2fd6 Mon Sep 17 00:00:00 2001 From: avsm Date: Tue, 18 Oct 2022 17:03:40 +0000 Subject: [PATCH] deploy: 6b0fa01c9eb9e081eb66fc65c3d3815734cb1d7c --- classes.html | 501 +++-------- command-line-parsing.html | 475 ++--------- compiler-backend.html | 674 +++------------ compiler-frontend.html | 761 ++++------------- concurrent-programming.html | 1005 +++++----------------- css/app.css | 2 +- data-serialization.html | 412 ++------- error-handling.html | 515 +++-------- faqs.html | 4 +- files-modules-and-programs.html | 613 +++----------- first-class-modules.html | 326 ++----- foreign-function-interface.html | 702 ++++----------- functors.html | 392 ++------- gadts.html | 725 ++++------------ garbage-collector.html | 537 ++---------- guided-tour.html | 883 ++++--------------- images/book-cover.jpg | Bin 55707 -> 76024 bytes imperative-programming.html | 1130 +++++-------------------- index.html | 16 +- install.html | 4 +- json.html | 523 +++--------- lists-and-patterns.html | 529 +++--------- maps-and-hashtables.html | 523 +++--------- objects.html | 442 ++-------- parsing-with-ocamllex-and-menhir.html | 433 ++-------- platform.html | 605 +++---------- prologue.html | 317 ++----- records.html | 424 ++-------- runtime-memory-layout.html | 490 ++--------- testing.html | 605 +++---------- toc.html | 5 +- variables-and-functions.html | 687 +++------------ variants.html | 561 +++--------- 33 files changed, 3111 insertions(+), 12710 deletions(-) diff --git a/classes.html b/classes.html index fa369aeb9..9e8374785 100644 --- a/classes.html +++ b/classes.html @@ -1,20 +1,12 @@ -Classes - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Classes - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Classes

This chapter was written by Leo White and Jason Hickey.

-

Programming with objects directly is great for encapsulation, but one -of the main goals of object-oriented programming is code reuse through -inheritance. For inheritance, we need to introduce classes. In -object-oriented programming, a class is essentially a recipe for -creating objects. The recipe can be changed by adding new methods and -fields, or it can be changed by modifying existing methods.  

+

Programming with objects directly is great for encapsulation, but one of the main goals of object-oriented programming is code reuse through inheritance. For inheritance, we need to introduce classes. In object-oriented programming, a class is essentially a recipe for creating objects. The recipe can be changed by adding new methods and fields, or it can be changed by modifying existing methods.  

OCaml Classes

-

In OCaml, class definitions must be defined as toplevel statements in -a module. The syntax for a class definition uses the keyword -class:  

+

In OCaml, class definitions must be defined as toplevel statements in a module. The syntax for a class definition uses the keyword class:  

open Base;;
 class istack = object
@@ -38,20 +30,8 @@ 
OCaml Classes

 >  end
-

The class istack : object ... end result shows that we -have created a class istack with class type -object ... end. Like module types, class types are -completely separate from regular OCaml types (e.g., int, -string, and list) and, in particular, should -not be confused with object types (e.g., -< get : int; .. >). The class type describes the -class itself rather than the objects that the class creates. This -particular class type specifies that the istack class -defines a mutable field v, a method pop that -returns an int option, and a method push with -type int -> unit.

-

To produce an object, classes are instantiated with the keyword -new:

+

The class istack : object ... end result shows that we have created a class istack with class type object ... end. Like module types, class types are completely separate from regular OCaml types (e.g., int, string, and list) and, in particular, should not be confused with object types (e.g., < get : int; .. >). The class type describes the class itself rather than the objects that the class creates. This particular class type specifies that the istack class defines a mutable field v, a method pop that returns an int option, and a method push with type int -> unit.

+

To produce an object, classes are instantiated with the keyword new:

let s = new istack;;
 >val s : istack = <obj>
@@ -63,33 +43,18 @@ 
OCaml Classes

 >- : int option = Some 5
-

You may have noticed that the object s has been given -the type istack. But wait, we’ve stressed classes are -not types, so what’s up with that? In fact, what we’ve said is -entirely true: classes and class names are not types. However, -for convenience, the definition of the class istack also -defines an object type istack with the same methods as the -class. This type definition is equivalent to:

+

You may have noticed that the object s has been given the type istack. But wait, we’ve stressed classes are not types, so what’s up with that? In fact, what we’ve said is entirely true: classes and class names are not types. However, for convenience, the definition of the class istack also defines an object type istack with the same methods as the class. This type definition is equivalent to:

type istack = < pop: int option; push: int -> unit >;;
 >type istack = < pop : int option; push : int -> unit >
-

Note that this type represents any object with these methods: objects -created using the istack class will have this type, but -objects with this type may not have been created by the -istack class.

+

Note that this type represents any object with these methods: objects created using the istack class will have this type, but objects with this type may not have been created by the istack class.

Class Parameters and Polymorphism

-

A class definition serves as the constructor for the class. -In general, a class definition may have parameters that must be provided -as arguments when the object is created with new.   

-

Let’s implement a variant of the istack class that can -hold any values, not just integers. When defining the class, the type -parameters are placed in square brackets before the class name in the -class definition. We also add a parameter init for the -initial contents of the stack:

+

A class definition serves as the constructor for the class. In general, a class definition may have parameters that must be provided as arguments when the object is created with new.   

+

Let’s implement a variant of the istack class that can hold any values, not just integers. When defining the class, the type parameters are placed in square brackets before the class name in the class definition. We also add a parameter init for the initial contents of the stack:

class ['a] stack init = object
   val mutable v : 'a list = init
@@ -113,13 +78,8 @@ 
Class Parameters and Polymorphism

 >  end
-

Note that the type parameter ['a] in the definition uses -square brackets, but for other uses of the type they are omitted (or -replaced with parentheses if there is more than one type parameter).

-

The type annotation on the declaration of v is used to -constrain type inference. If we omit this annotation, the type inferred -for the class will be “too polymorphic”: init could have -some type 'b list:

+

Note that the type parameter ['a] in the definition uses square brackets, but for other uses of the type they are omitted (or replaced with parentheses if there is more than one type parameter).

+

The type annotation on the declaration of v is used to constrain type inference. If we omit this annotation, the type inferred for the class will be “too polymorphic”: init could have some type 'b list:

class ['a] stack init = object
   val mutable v = init
@@ -146,25 +106,12 @@ 
Class Parameters and Polymorphism

 >       The method pop has type 'a option where 'a is unbound
-

In general, we need to provide enough constraints so that the -compiler will infer the correct type. We can add type constraints to the -parameters, to the fields, and to the methods. It is a matter of -preference how many constraints to add. You can add type constraints in -all three places, but the extra text may not help clarity. A convenient -middle ground is to annotate the fields and/or class parameters, and add -constraints to methods only if necessary.

+

In general, we need to provide enough constraints so that the compiler will infer the correct type. We can add type constraints to the parameters, to the fields, and to the methods. It is a matter of preference how many constraints to add. You can add type constraints in all three places, but the extra text may not help clarity. A convenient middle ground is to annotate the fields and/or class parameters, and add constraints to methods only if necessary.

Object Types as Interfaces

-

We may wish to traverse the elements on our stack. One common style -for doing this in object-oriented languages is to define a class for an -iterator object. An iterator provides a generic mechanism -to inspect and traverse the elements of a collection.      

-

There are two common styles for defining abstract interfaces like -this. In Java, an iterator would normally be specified with an -interface, which specifies a set of method types:

+

We may wish to traverse the elements on our stack. One common style for doing this in object-oriented languages is to define a class for an iterator object. An iterator provides a generic mechanism to inspect and traverse the elements of a collection.      

+

There are two common styles for defining abstract interfaces like this. In Java, an iterator would normally be specified with an interface, which specifies a set of method types:

// Java-style iterator, specified as an interface.
 interface <T> iterator {
@@ -173,10 +120,7 @@ 
Object Types as Interfaces

   void Next();
 };
-

In languages without interfaces, like C++, the specification would -normally use abstract classes to specify the methods without -implementing them (C++ uses the “= 0” definition to mean “not -implemented”):

+

In languages without interfaces, like C++, the specification would normally use abstract classes to specify the methods without implementing them (C++ uses the “= 0” definition to mean “not implemented”):

// Abstract class definition in C++.
 template<typename T>
@@ -188,20 +132,14 @@ 
Object Types as Interfaces

   virtual void next() = 0;
 };
-

OCaml supports both styles. In fact, OCaml is more flexible than -these approaches because an object type can be implemented by any object -with the appropriate methods; it does not have to be specified by the -object’s class a priori. We’ll leave abstract classes for -later. Let’s demonstrate the technique using object types.

-

First, we’ll define an object type iterator that -specifies the methods in an iterator:

+

OCaml supports both styles. In fact, OCaml is more flexible than these approaches because an object type can be implemented by any object with the appropriate methods; it does not have to be specified by the object’s class a priori. We’ll leave abstract classes for later. Let’s demonstrate the technique using object types.

+

First, we’ll define an object type iterator that specifies the methods in an iterator:

type 'a iterator = < get : 'a; has_value : bool; next : unit >;;
 >type 'a iterator = < get : 'a; has_value : bool; next : unit >
-

Next, we’ll define an actual iterator for lists. We can use this to -iterate over the contents of our stack:

+

Next, we’ll define an actual iterator for lists. We can use this to iterate over the contents of our stack:

class ['a] list_iterator init = object
   val mutable current : 'a list = init
@@ -228,10 +166,7 @@ 
Object Types as Interfaces

 >  end
-

Finally, we add a method iterator to the -stack class to produce an iterator. To do so, we construct -a list_iterator that refers to the current contents of the -stack:

+

Finally, we add a method iterator to the stack class to produce an iterator. To do so, we construct a list_iterator that refers to the current contents of the stack:

class ['a] stack init = object
   val mutable v : 'a list = init
@@ -259,8 +194,7 @@ 
Object Types as Interfaces

 >  end
-

Now we can build a new stack, push some values to it, and iterate -over them:

+

Now we can build a new stack, push some values to it, and iterate over them:

let s = new stack [];;
 >val s : '_weak1 stack = <obj>
@@ -284,10 +218,7 @@ 
Object Types as Interfaces

Functional Iterators

-

In practice, most OCaml programmers avoid iterator objects in favor -of functional-style techniques. For example, the alternative -stack class that follows takes a function f -and applies it to each of the elements on the stack:  

+

In practice, most OCaml programmers avoid iterator objects in favor of functional-style techniques. For example, the alternative stack class that follows takes a function f and applies it to each of the elements on the stack:  

class ['a] stack init = object
   val mutable v : 'a list = init
@@ -315,15 +246,8 @@ 
Functional Iterators

 >  end
-

What about functional operations like map and -fold? In general, these methods take a function that -produces a value of some other type than the elements of the set.

-

For example, a fold method for our -['a] stack class should have type -('b -> 'a -> 'b) -> 'b -> 'b, where the -'b is polymorphic. To express a polymorphic method type -like this, we must use a type quantifier, as shown in the following -example:

+

What about functional operations like map and fold? In general, these methods take a function that produces a value of some other type than the elements of the set.

+

For example, a fold method for our ['a] stack class should have type ('b -> 'a -> 'b) -> 'b -> 'b, where the 'b is polymorphic. To express a polymorphic method type like this, we must use a type quantifier, as shown in the following example:

class ['a] stack init = object
   val mutable v : 'a list = init
@@ -351,19 +275,12 @@ 
Functional Iterators

 >  end
-

The type quantifier 'b. can be read as “for all -'b.” Type quantifiers can only be used directly -after the method name, which means that method parameters must be -expressed using a fun or function -expression.

+

The type quantifier 'b. can be read as “for all 'b.” Type quantifiers can only be used directly after the method name, which means that method parameters must be expressed using a fun or function expression.

Inheritance

-

Inheritance uses an existing class to define a new one. For example, -the following class definition inherits from our stack -class for strings and adds a new method print that prints -all the strings on the stack:   

+

Inheritance uses an existing class to define a new one. For example, the following class definition inherits from our stack class for strings and adds a new method print that prints all the strings on the stack:   

class sstack init = object
   inherit [string] stack init
@@ -381,9 +298,7 @@ 
Inheritance

 >  end
-

A class can override methods from classes it inherits. For example, -this class creates stacks of integers that double the integers before -they are pushed onto the stack:

+

A class can override methods from classes it inherits. For example, this class creates stacks of integers that double the integers before they are pushed onto the stack:

class double_stack init = object
   inherit [int] stack init as super
@@ -400,21 +315,12 @@ 
Inheritance

 >  end
-

The preceding as super statement creates a special -object called super which can be used to call superclass -methods. Note that super is not a real object and can only -be used to call methods.

+

The preceding as super statement creates a special object called super which can be used to call superclass methods. Note that super is not a real object and can only be used to call methods.

Class Types

-

To allow code in a different file or module to inherit from a class, -we must expose it and give it a class type. What is the class type? - 

-

As an example, let’s wrap up our stack class in an -explicit module (we’ll use explicit modules for illustration, but the -process is similar when we want to define a .mli file). In -keeping with the usual style for modules, we define a type -'a t to represent the type of our stacks:

+

To allow code in a different file or module to inherit from a class, we must expose it and give it a class type. What is the class type?  

+

As an example, let’s wrap up our stack class in an explicit module (we’ll use explicit modules for illustration, but the process is similar when we want to define a .mli file). In keeping with the usual style for modules, we define a type 'a t to represent the type of our stacks:

module Stack = struct
   class ['a] stack init = object
@@ -426,10 +332,7 @@ 
Class Types

   let make init = new stack init
 end
-

We have multiple choices in defining the module type, depending on -how much of the implementation we want to expose. At one extreme, a -maximally abstract signature would completely hide the class -definitions:

+

We have multiple choices in defining the module type, depending on how much of the implementation we want to expose. At one extreme, a maximally abstract signature would completely hide the class definitions:

module AbstractStack : sig
    type 'a t = < pop: 'a option; push: 'a -> unit >
@@ -437,16 +340,8 @@ 
Class Types

    val make : 'a list -> 'a t
 end = Stack
-

The abstract signature is simple because we ignore the classes. But -what if we want to include them in the signature so that other modules -can inherit from the class definitions? For this, we need to specify -types for the classes, called class types.

-

Class types do not appear in mainstream object-oriented programming -languages, so you may not be familiar with them, but the concept is -pretty simple. A class type specifies the type of each of the visible -parts of the class, including both fields and methods. Just as with -module types, you don’t have to give a type for everything; anything you -omit will be hidden:

+

The abstract signature is simple because we ignore the classes. But what if we want to include them in the signature so that other modules can inherit from the class definitions? For this, we need to specify types for the classes, called class types.

+

Class types do not appear in mainstream object-oriented programming languages, so you may not be familiar with them, but the concept is pretty simple. A class type specifies the type of each of the visible parts of the class, including both fields and methods. Just as with module types, you don’t have to give a type for everything; anything you omit will be hidden:

module VisibleStack : sig
 
@@ -461,24 +356,13 @@ 
Class Types

   val make : 'a list -> 'a t
 end = Stack
-

In this signature, we’ve chosen to make everything visible. The class -type for stack specifies the types of the field -v, as well as the types of each of the methods.

+

In this signature, we’ve chosen to make everything visible. The class type for stack specifies the types of the field v, as well as the types of each of the methods.

Open Recursion

-

Open recursion allows an object’s methods to invoke other methods on -the same object. These calls are looked up dynamically, allowing a -method in one class to call a method from another class, if both classes -are inherited by the same object. This allows mutually recursive parts -of an object to be defined separately.    

-

This ability to define mutually recursive methods from separate -components is a key feature of classes: achieving similar functionality -with data types or modules is much more cumbersome and verbose.

-

For example, consider writing recursive functions over a simple -document format. This format is represented as a tree with three -different types of node:

+

Open recursion allows an object’s methods to invoke other methods on the same object. These calls are looked up dynamically, allowing a method in one class to call a method from another class, if both classes are inherited by the same object. This allows mutually recursive parts of an object to be defined separately.    

+

This ability to define mutually recursive methods from separate components is a key feature of classes: achieving similar functionality with data types or modules is much more cumbersome and verbose.

+

For example, consider writing recursive functions over a simple document format. This format is represented as a tree with three different types of node:

type doc =
   | Heading of string
@@ -495,13 +379,8 @@ 
Open Recursion

   { tag: 'a;
     text: text_item list }
-

It is quite easy to write a function that operates by recursively -traversing this data. However, what if you need to write many similar -recursive functions? How can you factor out the common parts of these -functions to avoid repetitive boilerplate?

-

The simplest way is to use classes and open recursion. For example, -the following class defines objects that fold over the document -data:

+

It is quite easy to write a function that operates by recursively traversing this data. However, what if you need to write many similar recursive functions? How can you factor out the common parts of these functions to avoid repetitive boilerplate?

+

The simplest way is to use classes and open recursion. For example, the following class defines objects that fold over the document data:

class ['a] folder = object(self)
   method doc acc = function
@@ -520,13 +399,8 @@ 
Open Recursion

   | Quote doc -> self#doc acc doc
 end
-

The object (self) syntax binds self to the -current object, allowing the doc, list_item, -and text_item methods to call each other.

-

By inheriting from this class, we can create functions that fold over -the document data. For example, the count_doc function -counts the number of bold tags in the document that are not within a -list:

+

The object (self) syntax binds self to the current object, allowing the doc, list_item, and text_item methods to call each other.

+

By inheriting from this class, we can create functions that fold over the document data. For example, the count_doc function counts the number of bold tags in the document that are not within a list:

class counter = object
   inherit [int] folder as super
@@ -542,23 +416,12 @@ 
Open Recursion

 
 let count_doc = (new counter)#doc
-

Note how the super special object is used in -text_item to call the [int] folder class’s -text_item method to fold over the children of the -text_item node.

+

Note how the super special object is used in text_item to call the [int] folder class’s text_item method to fold over the children of the text_item node.

Private Methods

-

Methods can be declared private, which means that they may -be called by subclasses, but they are not visible otherwise (similar to -a protected method in C++).     

-

For example, we may want to include methods in our -folder class for handling each of the different cases in -doc and text_item. However, we may not want to -force subclasses of folder to expose these methods, as they -probably shouldn’t be called directly:

+

Methods can be declared private, which means that they may be called by subclasses, but they are not visible otherwise (similar to a protected method in C++).     

+

For example, we may want to include methods in our folder class for handling each of the different cases in doc and text_item. However, we may not want to force subclasses of folder to expose these methods, as they probably shouldn’t be called directly:

class ['a] folder2 = object(self)
   method doc acc = function
@@ -595,14 +458,8 @@ 
Private Methods

     list_item : 'a . int -> 'a list_item -> int;
     text_item : int -> text_item -> int >  = new folder2
-

The final statement that builds the value f shows how -the instantiation of a folder2 object has a type that hides -the private methods.

-

To be precise, the private methods are part of the class type, but -not part of the object type. This means, for example, that the object -f has no method bold. However, the private -methods are available to subclasses: we can use them to simplify our -counter class:

+

The final statement that builds the value f shows how the instantiation of a folder2 object has a type that hides the private methods.

+

To be precise, the private methods are part of the class type, but not part of the object type. This means, for example, that the object f has no method bold. However, the private methods are available to subclasses: we can use them to simplify our counter class:

class counter_with_private_method = object
   inherit [int] folder2 as super
@@ -614,13 +471,7 @@ 
Private Methods

     acc + 1
 end
-

The key property of private methods is that they are visible to -subclasses, but not anywhere else. If you want the stronger guarantee -that a method is really private, not even accessible in -subclasses, you can use an explicit class type that omits the method. In -the following code, the private methods are explicitly omitted from the -class type of counter_with_sig and can’t be invoked in -subclasses of counter_with_sig:

+

The key property of private methods is that they are visible to subclasses, but not anywhere else. If you want the stronger guarantee that a method is really private, not even accessible in subclasses, you can use an explicit class type that omits the method. In the following code, the private methods are explicitly omitted from the class type of counter_with_sig and can’t be invoked in subclasses of counter_with_sig:

class counter_with_sig : object
   method doc : int -> doc -> int
@@ -639,10 +490,7 @@ 
Private Methods

Binary Methods

-

A binary method is a method that takes an object of -self type. One common example is defining a method for -equality:    

+

A binary method is a method that takes an object of self type. One common example is defining a method for equality:    

class square w = object(self : 'self)
   method width = w
@@ -670,10 +518,8 @@ 
Binary Methods

 >  end
-

Note how we can use the type annotation (self: 'self) to -obtain the type of the current object.

-

We can now test different object instances for equality by using the -equals binary method:

+

Note how we can use the type annotation (self: 'self) to obtain the type of the current object.

+

We can now test different object instances for equality by using the equals binary method:

(new square 5)#equals (new square 5);;
 >- : bool = true
@@ -681,11 +527,7 @@ 
Binary Methods

 >- : bool = false
-

This works, but there is a problem lurking here. The method -equals takes an object of the exact type -square or circle. Because of this, we can’t -define a common base class shape that also includes an -equality method:

+

This works, but there is a problem lurking here. The method equals takes an object of the exact type square or circle. Because of this, we can’t define a common base class shape that also includes an equality method:

type shape = < equals : shape -> bool; area : float >;;
 >type shape = < area : float; equals : shape -> bool >
@@ -699,16 +541,8 @@ 
Binary Methods

 >       The first object type has no method width
-

The problem is that a square expects to be compared with -a square, not an arbitrary shape; likewise for -circle. This problem is fundamental. Many languages solve -it either with narrowing (with dynamic type checking), or by method -overloading. Since OCaml has neither of these, what can we do?     

-

Since the problematic method is equality, one proposal we could -consider is to just drop it from the base type shape and -use polymorphic equality instead. However, the built-in polymorphic -equality has very poor behavior when applied to objects:

+

The problem is that a square expects to be compared with a square, not an arbitrary shape; likewise for circle. This problem is fundamental. Many languages solve it either with narrowing (with dynamic type checking), or by method overloading. Since OCaml has neither of these, what can we do?     

+

Since the problematic method is equality, one proposal we could consider is to just drop it from the base type shape and use polymorphic equality instead. However, the built-in polymorphic equality has very poor behavior when applied to objects:

Poly.(=)
   (object method area = 5 end)
@@ -716,14 +550,8 @@ 
Binary Methods

 >- : bool = false
-

The problem here is that two objects are considered equal by the -built-in polymorphic equality if and only if they are physically equal. -There are other reasons not to use the built-in polymorphic equality, -but these false negatives are a showstopper.

-

If we want to define equality for shapes in general, the remaining -solution is to use the same approach as we described for narrowing. That -is, introduce a representation type implemented using variants, -and implement the comparison based on the representation type:  

+

The problem here is that two objects are considered equal by the built-in polymorphic equality if and only if they are physically equal. There are other reasons not to use the built-in polymorphic equality, but these false negatives are a showstopper.

+

If we want to define equality for shapes in general, the remaining solution is to use the same approach as we described for narrowing. That is, introduce a representation type implemented using variants, and implement the comparison based on the representation type:  

type shape_repr =
   | Square of int
@@ -751,10 +579,7 @@ 
Binary Methods

 >  end
-

The binary method equals is now implemented in terms of -the concrete type shape_repr. When using this pattern, you -will not be able to hide the repr method, but you can hide -the type definition using the module system:

+

The binary method equals is now implemented in terms of the concrete type shape_repr. When using this pattern, you will not be able to hide the repr method, but you can hide the type definition using the module system:

module Shapes : sig
   type shape_repr
@@ -775,18 +600,9 @@ 
Binary Methods

   ...
 end
-

Note that this solution prevents us from adding new kinds of shapes -without adding new constructors to the shape_repr type, -which is quite restrictive. We can fix this, however, by using OCaml’s -rarely-used but still useful extensible variants.  

-

Extensible variants let you separate the definition of a variant type -from the definition of its constructors. The resulting type is by -definition open, in the sense that new variants can always be added. As -a result, the compiler can’t check whether pattern matching on such a -variant is exhaustive. Happily, exhaustivity is not what we need -here.

-

Here’s how we’d rewrite the above example with extensible -variants.

+

Note that this solution prevents us from adding new kinds of shapes without adding new constructors to the shape_repr type, which is quite restrictive. We can fix this, however, by using OCaml’s rarely-used but still useful extensible variants.  

+

Extensible variants let you separate the definition of a variant type from the definition of its constructors. The resulting type is by definition open, in the sense that new variants can always be added. As a result, the compiler can’t check whether pattern matching on such a variant is exhaustive. Happily, exhaustivity is not what we need here.

+

Here’s how we’d rewrite the above example with extensible variants.

type shape_repr = ..;;
 >type shape_repr = ..
@@ -814,14 +630,8 @@ 
Binary Methods

 >  end
-

One oddity of the representation type approach is that the objects -created by these classes are in one-to-one correspondence with members -of the representation type, making the objects seem somewhat -redundant.

-

But equality is an extreme instance of a binary method: it needs -access to all the information of the other object. Many other binary -methods need only partial information about the object. For instance, -consider a method that compares shapes by their sizes:

+

One oddity of the representation type approach is that the objects created by these classes are in one-to-one correspondence with members of the representation type, making the objects seem somewhat redundant.

+

But equality is an extreme instance of a binary method: it needs access to all the information of the other object. Many other binary methods need only partial information about the object. For instance, consider a method that compares shapes by their sizes:

class square w = object(self)
   method width = w
@@ -837,33 +647,13 @@ 
Binary Methods

 >  end
-

The larger method can be used on a square, -but it can also be applied to any object of type shape.

+

The larger method can be used on a square, but it can also be applied to any object of type shape.

Virtual Classes and Methods

-

A virtual class is a class where some methods or fields are -declared but not implemented. This should not be confused with the word -virtual as it is used in C++. A virtual method -in C++ uses dynamic dispatch, while regular, nonvirtual methods are -statically dispatched. In OCaml, all methods use dynamic -dispatch, but the keyword virtual means that the method or -field is not implemented. A class containing virtual methods must also -be flagged virtual and cannot be directly instantiated -(i.e., no object of this class can be created).        

-

To explore this, let’s extend our shapes examples to simple, -interactive graphics. We will use the Async concurrency library and the -Async_graphics -library, which provides an asynchronous interface to OCaml’s built-in -Graphics library. Concurrent programming with Async will be explored -later in Chapter 16, Concurrent Programming With Async; for now you can -safely ignore the details. You just need to run -opam install async_graphics to get the library installed on -your system.

-

We will give each shape a draw method that describes how -to draw the shape on the Async_graphics display:

+

A virtual class is a class where some methods or fields are declared but not implemented. This should not be confused with the word virtual as it is used in C++. A virtual method in C++ uses dynamic dispatch, while regular, nonvirtual methods are statically dispatched. In OCaml, all methods use dynamic dispatch, but the keyword virtual means that the method or field is not implemented. A class containing virtual methods must also be flagged virtual and cannot be directly instantiated (i.e., no object of this class can be created).        

+

To explore this, let’s extend our shapes examples to simple, interactive graphics. We will use the Async concurrency library and the Async_graphics library, which provides an asynchronous interface to OCaml’s built-in Graphics library. Concurrent programming with Async will be explored later in Chapter 16, Concurrent Programming With Async; for now you can safely ignore the details. You just need to run opam install async_graphics to get the library installed on your system.

+

We will give each shape a draw method that describes how to draw the shape on the Async_graphics display:

open Core
 open Async
@@ -873,9 +663,7 @@ 
Virtual Classes and Methods

Create Some Simple Shapes

-

Now let’s add classes for making squares and circles. We include an -on_click method for adding event handlers to the shapes: - 

+

Now let’s add classes for making squares and circles. We include an on_click method for adding event handlers to the shapes:  

class square w x y = object(self)
   val mutable x: int = x
@@ -900,8 +688,7 @@ 
Create Some Simple Shapes

            f ev.mouse_x ev.mouse_y)
 end
-

The square class is pretty straightforward, and the -circle class below also looks very similar:

+

The square class is pretty straightforward, and the circle class below also looks very similar:

class circle r x y = object(self)
   val mutable x: int = x
@@ -927,17 +714,8 @@ 
Create Some Simple Shapes

            f ev.mouse_x ev.mouse_y)
 end
-

These classes have a lot in common, and it would be useful to factor -out this common functionality into a superclass. We can easily move the -definitions of x and y into a superclass, but -what about on_click? Its definition depends on -contains, which has a different definition in each class. -The solution is to create a virtual class. This class will -declare a contains method but leave its definition to the -subclasses.

-

Here is the more succinct definition, starting with a virtual -shape class that implements on_click and -on_mousedown:

+

These classes have a lot in common, and it would be useful to factor out this common functionality into a superclass. We can easily move the definitions of x and y into a superclass, but what about on_click? Its definition depends on contains, which has a different definition in each class. The solution is to create a virtual class. This class will declare a contains method but leave its definition to the subclasses.

+

Here is the more succinct definition, starting with a virtual shape class that implements on_click and on_mousedown:

class virtual shape x y = object(self)
   method virtual private contains: int -> int -> bool
@@ -961,8 +739,7 @@ 
Create Some Simple Shapes

            f ev.mouse_x ev.mouse_y)
 end
-

Now we can define square and circle by -inheriting from shape:

+

Now we can define square and circle by inheriting from shape:

class square w x y = object
   inherit shape x y
@@ -991,18 +768,12 @@ 
Create Some Simple Shapes

       dx * dx + dy * dy <= radius * radius
 end
-

One way to view a virtual class is that it is like a -functor, where the “inputs” are the declared—but not defined—virtual -methods and fields. The functor application is implemented through -inheritance, when virtual methods are given concrete -implementations.

+

One way to view a virtual class is that it is like a functor, where the “inputs” are the declared—but not defined—virtual methods and fields. The functor application is implemented through inheritance, when virtual methods are given concrete implementations.

Initializers

-

You can execute expressions during the instantiation of a class by -placing them before the object expression or in the initial value of a -field:   

+

You can execute expressions during the instantiation of a class by placing them before the object expression or in the initial value of a field:   

class obj x =
   let () = Stdio.printf "Creating obj %d\n" x in
@@ -1016,15 +787,8 @@ 
Initializers

 >val o : obj = <obj>
-

However, these expressions are executed before the object has been -created and cannot refer to the methods of the object. If you need to -use an object’s methods during instantiation, you can use an -initializer. An initializer is an expression that will be executed -during instantiation but after the object has been created.

-

For example, suppose we wanted to extend our previous shapes module -with a growing_circle class for circles that expand when -clicked. We could inherit from circle and use the inherited -on_click to add a handler for click events:

+

However, these expressions are executed before the object has been created and cannot refer to the methods of the object. If you need to use an object’s methods during instantiation, you can use an initializer. An initializer is an expression that will be executed during instantiation but after the object has been created.

+

For example, suppose we wanted to extend our previous shapes module with a growing_circle class for circles that expand when clicked. We could inherit from circle and use the inherited on_click to add a handler for click events:

class growing_circle r x y = object(self)
   inherit circle r x y
@@ -1036,75 +800,34 @@ 
Initializers

Multiple Inheritance

-

When a class inherits from more than one superclass, it is using -multiple inheritance. Multiple inheritance extends the variety -of ways that classes can be combined, and it can be quite useful, -particularly with virtual classes. However, it can be tricky to use, -particularly when the inheritance hierarchy is a graph rather than a -tree, so it should be used with care.    

+

When a class inherits from more than one superclass, it is using multiple inheritance. Multiple inheritance extends the variety of ways that classes can be combined, and it can be quite useful, particularly with virtual classes. However, it can be tricky to use, particularly when the inheritance hierarchy is a graph rather than a tree, so it should be used with care.    

How Names Are Resolved

-

The main trickiness of multiple inheritance is due to naming—what -happens when a method or field with some name is defined in more than -one class?

-

If there is one thing to remember about inheritance in OCaml, it is -this: inheritance is like textual inclusion. If there is more than one -definition for a name, the last definition wins.

-

For example, consider this class, which inherits from -square and defines a new draw method that uses -draw_rect instead of fill_rect to draw the -square:

+

The main trickiness of multiple inheritance is due to naming—what happens when a method or field with some name is defined in more than one class?

+

If there is one thing to remember about inheritance in OCaml, it is this: inheritance is like textual inclusion. If there is more than one definition for a name, the last definition wins.

+

For example, consider this class, which inherits from square and defines a new draw method that uses draw_rect instead of fill_rect to draw the square:

class square_outline w x y = object
   inherit square w x y
   method draw = draw_rect x y width width
 end
-

Since the inherit declaration comes before the method -definition, the new draw method overrides the old one, and -the square is drawn using draw_rect. But, what if we had -defined square_outline as follows?

+

Since the inherit declaration comes before the method definition, the new draw method overrides the old one, and the square is drawn using draw_rect. But, what if we had defined square_outline as follows?

class square_outline w x y = object
   method draw = draw_rect x y w w
   inherit square w x y
 end
-

Here the inherit declaration comes after the method -definition, so the draw method from square -will override the other definition, and the square will be drawn using -fill_rect.

-

To reiterate, to understand what inheritance means, replace each -inherit directive with its definition, and take the last -definition of each method or field. Note that the methods and fields -added by an inheritance are those listed in its class type, so private -methods that are hidden by the type will not be included.

+

Here the inherit declaration comes after the method definition, so the draw method from square will override the other definition, and the square will be drawn using fill_rect.

+

To reiterate, to understand what inheritance means, replace each inherit directive with its definition, and take the last definition of each method or field. Note that the methods and fields added by an inheritance are those listed in its class type, so private methods that are hidden by the type will not be included.

Mixins

-

When should you use multiple inheritance? If you ask multiple people, -you’re likely to get multiple (perhaps heated) answers. Some will argue -that multiple inheritance is overly complicated; others will argue that -inheritance is problematic in general, and one should use object -composition instead. But regardless of who you talk to, you will rarely -hear that multiple inheritance is great and that you should use it -widely.   

-

In any case, if you’re programming with objects, there’s one general -pattern for multiple inheritance that is both useful and reasonably -simple: the mixin pattern. Generically, a mixin is -just a virtual class that implements a feature based on another one. If -you have a class that implements methods A, and you have a -mixin M that provides methods B from A, then -you can inherit from M—“mixing” it in—to get features -B.

-

That’s too abstract, so let’s give some examples based on our -interactive shapes. We may wish to allow a shape to be dragged by the -mouse. We can define this functionality for any object that has mutable -x and y fields and an -on_mousedown method for adding event handlers:

+

When should you use multiple inheritance? If you ask multiple people, you’re likely to get multiple (perhaps heated) answers. Some will argue that multiple inheritance is overly complicated; others will argue that inheritance is problematic in general, and one should use object composition instead. But regardless of who you talk to, you will rarely hear that multiple inheritance is great and that you should use it widely.   

+

In any case, if you’re programming with objects, there’s one general pattern for multiple inheritance that is both useful and reasonably simple: the mixin pattern. Generically, a mixin is just a virtual class that implements a feature based on another one. If you have a class that implements methods A, and you have a mixin M that provides methods B from A, then you can inherit from M—“mixing” it in—to get features B.

+

That’s too abstract, so let’s give some examples based on our interactive shapes. We may wish to allow a shape to be dragged by the mouse. We can define this functionality for any object that has mutable x and y fields and an on_mousedown method for adding event handlers:

class virtual draggable = object(self)
   method virtual on_mousedown:
@@ -1135,19 +858,14 @@ 
Mixins

               y <- ev.mouse_y + offset_y))
 end
-

This allows us to create draggable shapes using multiple -inheritance:

+

This allows us to create draggable shapes using multiple inheritance:

class small_square = object
   inherit square 20 40 40
   inherit draggable
 end
-

We can also use mixins to create animated shapes. Each animated shape -has a list of update functions to be called during animation. We create -an animated mixin to provide this update list and ensure -that the functions in it are called at regular intervals when the shape -is animated:  

+

We can also use mixins to create animated shapes. Each animated shape has a list of update functions to be called during animation. We create an animated mixin to provide this update list and ensure that the functions in it are called at regular intervals when the shape is animated:  

class virtual animated span = object(self)
   method virtual on_click:
@@ -1177,9 +895,7 @@ 
Mixins

     self#on_click (fun _x _y -> if not self#running then self#animate)
 end
-

We use initializers to add functions to this update list. For -example, this class will produce circles that move to the right for a -second when clicked:

+

We use initializers to add functions to this update list. For example, this class will produce circles that move to the right for a second when clicked:

class my_circle = object
   inherit circle 20 50 50
@@ -1220,10 +936,7 @@ 
Mixins

     updates <- update :: updates
 end
-

Since the linear and harmonic mixins are -only used for their side effects, they can be inherited multiple times -within the same object to produce a variety of different animations: - 

+

Since the linear and harmonic mixins are only used for their side effects, they can be inherited multiple times within the same object to produce a variety of different animations:  

class my_square x y = object
   inherit square 40 x y
@@ -1243,11 +956,7 @@ 
Mixins

Displaying the Animated Shapes

-

We finish our shapes module by creating a main function -to draw some shapes on the graphical display and running that function -using the Async scheduler:   

+

We finish our shapes module by creating a main function to draw some shapes on the graphical display and running that function using the Async scheduler:   

let main () =
   let shapes = [
@@ -1267,13 +976,8 @@ 
Displaying the Animated Shapes

 
 let () = never_returns (Scheduler.go_main ~main ())
-

Our main function creates a list of shapes to be -displayed and defines a repaint function that actually -draws them on the display. We then open a graphical display and ask -Async to run repaint at regular intervals.

-

Finally, build the binary by linking against the -async_graphics package, which will pull in all the other -dependencies:

+

Our main function creates a list of shapes to be displayed and defines a repaint function that actually draws them on the display. We then open a graphical display and ask Async to run repaint at regular intervals.

+

Finally, build the binary by linking against the async_graphics package, which will pull in all the other dependencies:

(executable
   (name      shapes)
@@ -1284,33 +988,20 @@ 
Displaying the Animated Shapes

 
dune build shapes.exe
-

When you run the binary, a new graphical window should appear (on -macOS, you will need to install the X11 package first, which you will be -prompted for). Try clicking on the various widgets, and gasp in awe at -the sophisticated animations that unfold as a result.

-

The graphics library described here is the one built into OCaml and -is more useful as a learning tool than anything else. There are several -third-party libraries that provide more sophisticated bindings to -various graphics subsystems:       

+

When you run the binary, a new graphical window should appear (on macOS, you will need to install the X11 package first, which you will be prompted for). Try clicking on the various widgets, and gasp in awe at the sophisticated animations that unfold as a result.

+

The graphics library described here is the one built into OCaml and is more useful as a learning tool than anything else. There are several third-party libraries that provide more sophisticated bindings to various graphics subsystems:       

Lablgtk
-
-A strongly typed interface to the GTK widget library. +
A strongly typed interface to the GTK widget library.
LablGL
-
-An interface between OCaml and OpenGL, a widely supported standard for -3D rendering. +
An interface between OCaml and OpenGL, a widely supported standard for 3D rendering.
js_of_ocaml
-
-Compiles OCaml code to JavaScript and has bindings to WebGL. This is the -emerging standard for 3D rendering in web browsers. +
Compiles OCaml code to JavaScript and has bindings to WebGL. This is the emerging standard for 3D rendering in web browsers.
-

Next: Chapter 14Maps and Hash Tables

 \ No newline at end of file +

Next: Chapter 14Maps and Hash Tables

 \ No newline at end of file diff --git a/command-line-parsing.html b/command-line-parsing.html index 68c73b54a..627228b87 100644 --- a/command-line-parsing.html +++ b/command-line-parsing.html @@ -1,88 +1,43 @@ -Command-Line Parsing - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Command-Line Parsing - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Command-Line Parsing

-

Many of the OCaml programs that you’ll write will end up as binaries -that need to be run from a command prompt. Any nontrivial command line -should support a collection of basic features:

+

Many of the OCaml programs that you’ll write will end up as binaries that need to be run from a command prompt. Any nontrivial command line should support a collection of basic features:

  • Parsing of command-line arguments

    • -

  • Generation of error messages in response to incorrect -inputs

    • +

  • Generation of error messages in response to incorrect inputs

  • Help for all the available options

  • Interactive autocompletion

-

It’s tedious and error-prone to code all of this manually for every -program you write. Core provides the Command library, which simplifies -all of this by letting you declare your command-line options in one -place and by deriving all of the above functionality from these -declarations.    

-

Command is simple to use for simple applications but also scales well -as your needs grow more complex. In particular, Command provides a -sophisticated subcommand mode that groups related commands together as -the complexity of your user interface grows. You may already be familiar -with this command-line style from the Git or Mercurial version control -systems.

+

It’s tedious and error-prone to code all of this manually for every program you write. Core provides the Command library, which simplifies all of this by letting you declare your command-line options in one place and by deriving all of the above functionality from these declarations.    

+

Command is simple to use for simple applications but also scales well as your needs grow more complex. In particular, Command provides a sophisticated subcommand mode that groups related commands together as the complexity of your user interface grows. You may already be familiar with this command-line style from the Git or Mercurial version control systems.

In this chapter, we’ll:

    -

  • Learn how to use Command to construct basic and grouped -command-line interfaces

    • -

  • Build simple equivalents to the cryptographic md5 -and shasum utilities

    • -

  • Demonstrate how to declare complex command-line interfaces in a -type-safe and elegant way  

    • +

  • Learn how to use Command to construct basic and grouped command-line interfaces

    • +

  • Build simple equivalents to the cryptographic md5 and shasum utilities

    • +

  • Demonstrate how to declare complex command-line interfaces in a type-safe and elegant way  

Basic Command-Line Parsing

-

Let’s start by working through a clone of the md5sum -command that is present on most Linux installations (the equivalent -command on macOS is simply md5). The following function -defined below reads in the contents of a file, applies the MD5 one-way -cryptographic hash function to the data, and outputs an ASCII hex -representation of the result:   

+

Let’s start by working through a clone of the md5sum command that is present on most Linux installations (the equivalent command on macOS is simply md5). The following function defined below reads in the contents of a file, applies the MD5 one-way cryptographic hash function to the data, and outputs an ASCII hex representation of the result:   

open Core
 
 let do_hash file =
   Md5.digest_file_blocking file |> Md5.to_hex |> print_endline
-

The do_hash function accepts a filename -parameter and prints the human-readable MD5 string to the console -standard output. The first step toward turning this function into a -command-line program is to create a parser for the command line -arguments. The module Command.Param provides a set of -combinators that can be combined together to define a parameter parser -for optional flags and positional arguments, including documentation, -the types they should map to, and whether to take special actions such -as pausing for interactive input if certain inputs are encountered.

+

The do_hash function accepts a filename parameter and prints the human-readable MD5 string to the console standard output. The first step toward turning this function into a command-line program is to create a parser for the command line arguments. The module Command.Param provides a set of combinators that can be combined together to define a parameter parser for optional flags and positional arguments, including documentation, the types they should map to, and whether to take special actions such as pausing for interactive input if certain inputs are encountered.

Defining an Anonymous Argument

-

Let’s build a parser for a command line UI with a single -anonymous argument, i.e., an argument that is passed in without -a flag.

+

Let’s build a parser for a command line UI with a single anonymous argument, i.e., an argument that is passed in without a flag.

let filename_param =
   let open Command.Param in
   anon ("filename" %: string)
-

Here, anon is used to signal the parsing of an anonymous -argument, and the expression ("filename" %: string) -indicates the textual name of the argument and specification that -describes the kind of value that is expected. The textual name is used -for generating help text, and the specification, which has type -Command.Arg_type.t, is used both to nail down the OCaml -type of the returned value (string, in this case) and to -guide features like input validation. The values anon, -string and %: all come from the -Command.Param module.

+

Here, anon is used to signal the parsing of an anonymous argument, and the expression ("filename" %: string) indicates the textual name of the argument and specification that describes the kind of value that is expected. The textual name is used for generating help text, and the specification, which has type Command.Arg_type.t, is used both to nail down the OCaml type of the returned value (string, in this case) and to guide features like input validation. The values anon, string and %: all come from the Command.Param module.

Defining Basic Commands

-

Once we’ve defined a specification, we need to put it to work on real -input. The simplest way is to directly create a command-line interface -with Command.basic.  

+

Once we’ve defined a specification, we need to put it to work on real input. The simplest way is to directly create a command-line interface with Command.basic.  

let command =
   Command.basic
@@ -91,26 +46,15 @@ 
Defining Basic Commands

     (Command.Param.map filename_param ~f:(fun filename () ->
          do_hash filename))
-

The summary argument is a one-line description which -goes at the top of the help screen, while the (optional) -readme argument is for providing a more detailed -description that will be provided on demand.

-

The final argument is the most interesting one, which is the -parameter parser. This will be easier to understand if we first learn a -bit more about the type signatures of the various components we’ve been -using. Let’s do that by recreating some of this code in the -toplevel.

+

The summary argument is a one-line description which goes at the top of the help screen, while the (optional) readme argument is for providing a more detailed description that will be provided on demand.

+

The final argument is the most interesting one, which is the parameter parser. This will be easier to understand if we first learn a bit more about the type signatures of the various components we’ve been using. Let’s do that by recreating some of this code in the toplevel.

let filename_param = Command.Param.(anon ("filename" %: string));;
 >val filename_param : string Command.Spec.param = <abstr>
-

The type parameter of filename_param is there to -indicate the type of the value returned by the parser; in this case, -string.

-

But Command.basic requires a parameter parser that -returns a value of type unit -> unit. We can see that by -using #show to explore the types.

+

The type parameter of filename_param is there to indicate the type of the value returned by the parser; in this case, string.

+

But Command.basic requires a parameter parser that returns a value of type unit -> unit. We can see that by using #show to explore the types.

#show Command.basic;;
 >val basic : unit Command.basic_command
@@ -121,29 +65,15 @@ 
Defining Basic Commands

 >    (unit -> 'result) Command.Spec.param -> Command.t
-

Note that the 'result parameter of the type alias -basic_command is instantiated as unit for the -type of Command.basic.

-

It makes sense that Command.basic wants a parser that -returns a function; after all, in the end, it needs a function it can -run that constitutes the execution of the program. But how do we get -such a parser, given the parser we have returns just a filename?

-

The answer is to use a map function to change the value -returned by the parser. As you can see below, the type of -Command.Param.map is very similar to the type of -List.map.

+

Note that the 'result parameter of the type alias basic_command is instantiated as unit for the type of Command.basic.

+

It makes sense that Command.basic wants a parser that returns a function; after all, in the end, it needs a function it can run that constitutes the execution of the program. But how do we get such a parser, given the parser we have returns just a filename?

+

The answer is to use a map function to change the value returned by the parser. As you can see below, the type of Command.Param.map is very similar to the type of List.map.

#show Command.Param.map;;
 >val map : 'a Command.Spec.param -> f:('a -> 'b) -> 'b Command.Spec.param
-

In our program, we used map to convert the -filename_param parser, which returns a string representing -the file name, into a parser that returns a function of type -unit -> unit containing the body of the command. It -might not be obvious that the function passed to map returns a function, -but remember that, due to currying, the invocation of map above could be -written equivalently as follows.

+

In our program, we used map to convert the filename_param parser, which returns a string representing the file name, into a parser that returns a function of type unit -> unit containing the body of the command. It might not be obvious that the function passed to map returns a function, but remember that, due to currying, the invocation of map above could be written equivalently as follows.

Command.Param.map filename_param ~f:(fun filename ->
   fun () -> do_hash filename)
@@ -152,23 +82,18 @@

Defining Basic Commands

Running Commands

-

Once we’ve defined the basic command, running it is just one function -call away.

+

Once we’ve defined the basic command, running it is just one function call away.

let () = Command_unix.run ~version:"1.0" ~build_info:"RWO" command
-

Command_unix.run takes a couple of optional arguments -that are useful to identify which version of the binary you are running -in production. You’ll need the following dune file:

+

Command_unix.run takes a couple of optional arguments that are useful to identify which version of the binary you are running in production. You’ll need the following dune file:

(executable
   (name       md5)
   (libraries  core core_unix.command_unix)
   (preprocess (pps ppx_jane)))
-

At which point we can build and execute the program using -dune exec. Let’s use this to query version information from -the binary.

+

At which point we can build and execute the program using dune exec. Let’s use this to query version information from the binary.

dune exec -- ./md5.exe -version
 >1.0
@@ -176,13 +101,8 @@ 
Running Commands

 >RWO
-

The versions that you see in the output were defined via the optional -arguments to Command_unix.run. You can leave these blank in -your own programs or get your build system to generate them directly -from your version control system. Dune provides a dune-build-info -library that automates this process for most common workflows.

-

We can invoke our binary with -help to see the -auto-generated help.

+

The versions that you see in the output were defined via the optional arguments to Command_unix.run. You can leave these blank in your own programs or get your build system to generate them directly from your version control system. Dune provides a dune-build-info library that automates this process for most common workflows.

+

We can invoke our binary with -help to see the auto-generated help.

dune exec -- ./md5.exe -help
 >Generate an MD5 hash of the input data
@@ -198,17 +118,13 @@ 
Running Commands

 >  [-help], -?                . print this help text and exit
-

If you supply the filename argument, then -do_hash is called with the argument and the MD5 output is -displayed to the standard output.

+

If you supply the filename argument, then do_hash is called with the argument and the MD5 output is displayed to the standard output.

dune exec -- ./md5.exe md5.ml
 >045215e7e7891779b261851a8458b656
-

And that’s all it takes to build our little MD5 utility! Here’s a -complete version of the example we just walked through, made slightly -more succinct by removing intermediate variables.

+

And that’s all it takes to build our little MD5 utility! Here’s a complete version of the example we just walked through, made slightly more succinct by removing intermediate variables.

open Core
 
@@ -229,10 +145,7 @@ 
Running Commands

Multi-Argument Commands

-

All the examples thus far have involved a single argument, but we can -of course create multi-argument commands as well. We can make a parser -for multiple arguments by binding together simpler parsers, using the -function Command.Param.both. Here is its type.

+

All the examples thus far have involved a single argument, but we can of course create multi-argument commands as well. We can make a parser for multiple arguments by binding together simpler parsers, using the function Command.Param.both. Here is its type.

#show Command.Param.both;;
 >val both :
@@ -240,11 +153,7 @@ 
Multi-Argument Commands

 >  'b Command.Spec.param -> ('a * 'b) Command.Spec.param
-

both allows us to take two parameter parsers and combine -them into a single parser that returns the two arguments as a pair. In -the following, we rewrite our md5 program so it takes two -anonymous arguments: the first is an integer saying how many characters -of the hash to print out, and the second is the filename.

+

both allows us to take two parameter parsers and combine them into a single parser that returns the two arguments as a pair. In the following, we rewrite our md5 program so it takes two anonymous arguments: the first is an integer saying how many characters of the hash to print out, and the second is the filename.

open Core
 
@@ -268,16 +177,13 @@ 
Multi-Argument Commands

 
 let () = Command_unix.run ~version:"1.0" ~build_info:"RWO" command
-

Building and running this command, we can see that it now indeed -expects two arguments.

+

Building and running this command, we can see that it now indeed expects two arguments.

dune exec -- ./md5.exe 5 md5.ml
 >bc879
-

This works well enough for two parameters, but if you want longer -parameter lists, this approach gets old fast. A better way is to use -let-syntax, which was discussed in Chapter 7, Error Handling.

+

This works well enough for two parameters, but if you want longer parameter lists, this approach gets old fast. A better way is to use let-syntax, which was discussed in Chapter 7, Error Handling.

let command =
   Command.basic
@@ -289,18 +195,8 @@ 
Multi-Argument Commands

     and filename = anon ("filename" %: string) in
     fun () -> do_hash hash_length filename)
-

Here, we take advantage of let-syntax’s support for parallel let -bindings, using and to join the definitions together. This -syntax translates down to the same pattern based on both -that we showed above, but it’s easier to read and use, and scales better -to more arguments.

-

The need to open both modules is a little awkward, and the -Param module in particular you really only need on the -right-hand-side of the equals-sign. This is achieved automatically by -using the let%map_open syntax, demonstrated below. We’ll -also drop the open of Command.Let_syntax in favor of -explicitly using let%map_open.Command to mark the -let-syntax as coming from the Command module

+

Here, we take advantage of let-syntax’s support for parallel let bindings, using and to join the definitions together. This syntax translates down to the same pattern based on both that we showed above, but it’s easier to read and use, and scales better to more arguments.

+

The need to open both modules is a little awkward, and the Param module in particular you really only need on the right-hand-side of the equals-sign. This is achieved automatically by using the let%map_open syntax, demonstrated below. We’ll also drop the open of Command.Let_syntax in favor of explicitly using let%map_open.Command to mark the let-syntax as coming from the Command module

let command =
   Command.basic
@@ -310,23 +206,14 @@ 
Multi-Argument Commands

      and filename = anon ("filename" %: string) in
      fun () -> do_hash hash_length filename)
-

Let-syntax is the most common way of writing parsers for -Command, and we’ll use that idiom from here on.

-

Now that we have the basics in place, the rest of the chapter will -examine some of the more advanced features of Command.

+

Let-syntax is the most common way of writing parsers for Command, and we’ll use that idiom from here on.

+

Now that we have the basics in place, the rest of the chapter will examine some of the more advanced features of Command.

Argument Types

-

You aren’t just limited to parsing command lines of strings and ints. -There are some other argument types defined in -Command.Param, like date and -percent. But most of the time, argument types for specific -types in Core and other associated libraries are defined in -the module that defines the type in question.   

-

As an example, we can tighten up the specification of the command to -Filename.arg_type to reflect that the argument must be a -valid filename, and not just any string.

+

You aren’t just limited to parsing command lines of strings and ints. There are some other argument types defined in Command.Param, like date and percent. But most of the time, argument types for specific types in Core and other associated libraries are defined in the module that defines the type in question.   

+

As an example, we can tighten up the specification of the command to Filename.arg_type to reflect that the argument must be a valid filename, and not just any string.

let command =
   Command.basic
@@ -337,15 +224,10 @@ 
Argument Types

      in
      fun () -> do_hash file)
-

This doesn’t change the validation of the provided value, but it does -enable interactive command-line completion. We’ll explain how to enable -that later in the chapter.

+

This doesn’t change the validation of the provided value, but it does enable interactive command-line completion. We’ll explain how to enable that later in the chapter.

Defining Custom Argument Types

-

We can also define our own argument types if the predefined ones -aren’t sufficient. For instance, let’s make a regular_file -argument type that ensures that the input file isn’t a character device -or some other odd UNIX file type that can’t be fully read.  

+

We can also define our own argument types if the predefined ones aren’t sufficient. For instance, let’s make a regular_file argument type that ensures that the input file isn’t a character device or some other odd UNIX file type that can’t be fully read.  

open Core
 
@@ -371,11 +253,7 @@ 
Defining Custom Argument Types

 
 let () = Command_unix.run ~version:"1.0" ~build_info:"RWO" command
-

The regular_file function transforms a -filename string parameter into the same string but first -checks that the file exists and is a regular file type. When you build -and run this code, you will see the new error messages if you try to -open a special device such as /dev/null:

+

The regular_file function transforms a filename string parameter into the same string but first checks that the file exists and is a regular file type. When you build and run this code, you will see the new error messages if you try to open a special device such as /dev/null:

dune exec -- ./md5.exe md5.ml
 >6a6c128cdc8e75f3b174559316e49a5d
@@ -394,13 +272,7 @@ 
Defining Custom Argument Types

Optional and Default Arguments

-

A more realistic md5 binary could also read from the -standard input if a filename isn’t specified. To do this, -we need to declare the filename argument as optional, which we can do -with the maybe operator.     

+

A more realistic md5 binary could also read from the standard input if a filename isn’t specified. To do this, we need to declare the filename argument as optional, which we can do with the maybe operator.     

let command =
   Command.basic
@@ -421,11 +293,7 @@ 
Optional and Default Arguments

 >       but an expression was expected of type string
 [1]
-

This is because changing the argument type has also changed the type -of the value that is returned by the parser. It now produces a -string option instead of a string, reflecting -the optionality of the argument. We can adapt our example to use the new -information and read from standard input if no file is specified.

+

This is because changing the argument type has also changed the type of the value that is returned by the parser. It now produces a string option instead of a string, reflecting the optionality of the argument. We can adapt our example to use the new information and read from standard input if no file is specified.

open Core
 
@@ -450,23 +318,14 @@ 
Optional and Default Arguments

 
 let () = Command_unix.run ~version:"1.0" ~build_info:"RWO" command
-

The filename parameter to do_hash is now a -string option type. This is resolved into a string via -get_contents to determine whether to read the standard -input or a file, and then the rest of the command is similar to our -previous examples.

+

The filename parameter to do_hash is now a string option type. This is resolved into a string via get_contents to determine whether to read the standard input or a file, and then the rest of the command is similar to our previous examples.

cat md5.ml | dune exec -- ./md5.exe
 >068f02a29536dbcf111e7adebb085aa1
-

Another possible way to handle this would be to supply a dash as the -default filename if one isn’t specified. The -maybe_with_default function can do just this, with the -benefit of not having to change the callback parameter type.

-

The following example behaves exactly the same as the previous -example, but replaces maybe with -maybe_with_default:

+

Another possible way to handle this would be to supply a dash as the default filename if one isn’t specified. The maybe_with_default function can do just this, with the benefit of not having to change the callback parameter type.

+

The following example behaves exactly the same as the previous example, but replaces maybe with maybe_with_default:

open Core
 
@@ -491,8 +350,7 @@ 
Optional and Default Arguments

 
 let () = Command_unix.run ~version:"1.0" ~build_info:"RWO" command
-

Building and running this confirms that it has the same behavior as -before.

+

Building and running this confirms that it has the same behavior as before.

cat md5.ml | dune exec -- ./md5.exe
 >370616ab5fad3dd995c136f09d0adb29
@@ -501,9 +359,7 @@ 
Optional and Default Arguments

Sequences of Arguments

-

Another common way of parsing anonymous arguments is as a variable -length list. As an example, let’s modify our MD5 code to take a -collection of files to process on the command line.  

+

Another common way of parsing anonymous arguments is as a variable length list. As an example, let’s modify our MD5 code to take a collection of files to process on the command line.  

open Core
 
@@ -531,13 +387,7 @@ 
Sequences of Arguments

 
 let () = Command_unix.run ~version:"1.0" ~build_info:"RWO" command
-

The callback function is a little more complex now, to handle the -extra options. The files are now a -string list, and an empty list reverts to using standard -input, just as our previous maybe and -maybe_with_default examples did. If the list of files isn’t -empty, then it opens up each file and runs them through -do_hash sequentially.

+

The callback function is a little more complex now, to handle the extra options. The files are now a string list, and an empty list reverts to using standard input, just as our previous maybe and maybe_with_default examples did. If the list of files isn’t empty, then it opens up each file and runs them through do_hash sequentially.

dune exec -- ./md5.exe /etc/services ./_build/default/md5.exe
 >MD5 (/etc/services) = 6501e9c7bf20b1dc56f015e341f79833
@@ -548,16 +398,8 @@ 
Sequences of Arguments

Adding Labeled Flags

-

You aren’t limited to anonymous arguments on the command line. A -flag is a named field that can be followed by an optional -argument. These flags can appear in any order on the command line, or -multiple times, depending on how they’re declared in the specification. -  

-

Let’s add two arguments to our md5 command that mimics -the macOS version. A -s flag specifies the string to be -hashed directly on the command line and -t runs a -self-test. The complete example follows.

+

You aren’t limited to anonymous arguments on the command line. A flag is a named field that can be followed by an optional argument. These flags can appear in any order on the command line, or multiple times, depending on how they’re declared in the specification.   

+

Let’s add two arguments to our md5 command that mimics the macOS version. A -s flag specifies the string to be hashed directly on the command line and -t runs a self-test. The complete example follows.

open Core
 
@@ -594,13 +436,7 @@ 
Adding Labeled Flags

 
 let () = Command_unix.run command
-

The specification now uses the flag function to define -the two new labeled, command-line arguments. The doc string -is formatted so that the first word is the short name that appears in -the usage text, with the remainder being the full help text. Notice that -the -t flag has no argument, and so we prepend its -doc text with a blank space. The help text for the -preceding code looks like this:

+

The specification now uses the flag function to define the two new labeled, command-line arguments. The doc string is formatted so that the first word is the short name that appears in the usage text, with the remainder being the full help text. Notice that the -t flag has no argument, and so we prepend its doc text with a blank space. The help text for the preceding code looks like this:

dune exec -- ./md5.exe -help
 >Generate an MD5 hash of the input data
@@ -619,58 +455,28 @@ 
Adding Labeled Flags

 >5a118fe92ac3b6c7854c595ecf6419cb
-

The -s flag in our specification requires a -string argument and isn’t optional. The Command parser -outputs an error message if the flag isn’t supplied, as with the -anonymous arguments in earlier examples. There are a number of other -functions that you can wrap flags in to control how they are parsed: - 

+

The -s flag in our specification requires a string argument and isn’t optional. The Command parser outputs an error message if the flag isn’t supplied, as with the anonymous arguments in earlier examples. There are a number of other functions that you can wrap flags in to control how they are parsed:  

    -
  • required <arg> will return -<arg> and error if not present
    • -
  • optional <arg> with return -<arg> option
    • -
  • optional_with_default <val> <arg> will -return <arg> with default <val> if -not present
    • -
  • listed <arg> will return -<arg> list (this flag may appear multiple times)
    • -
  • no_arg will return a bool that is true if -the flag is present
    • +
  • required <arg> will return <arg> and error if not present
    • +
  • optional <arg> with return <arg> option
    • +
  • optional_with_default <val> <arg> will return <arg> with default <val> if not present
    • +
  • listed <arg> will return <arg> list (this flag may appear multiple times)
    • +
  • no_arg will return a bool that is true if the flag is present
-

The flags affect the type of the callback function in exactly the -same way as anonymous arguments do. This lets you change the -specification and ensure that all the callback functions are updated -appropriately, without runtime errors.

+

The flags affect the type of the callback function in exactly the same way as anonymous arguments do. This lets you change the specification and ensure that all the callback functions are updated appropriately, without runtime errors.

Grouping Subcommands Together

-

You can get pretty far by using flags and anonymous arguments to -assemble complex, command-line interfaces. After a while, though, too -many options can make the program very confusing for newcomers to your -application. One way to solve this is by grouping common operations -together and adding some hierarchy to the command-line interface.    

-

You’ll have run across this style already when using the opam package -manager (or, in the non-OCaml world, the Git or Mercurial commands). -opam exposes commands in this form:

+

You can get pretty far by using flags and anonymous arguments to assemble complex, command-line interfaces. After a while, though, too many options can make the program very confusing for newcomers to your application. One way to solve this is by grouping common operations together and adding some hierarchy to the command-line interface.    

+

You’ll have run across this style already when using the opam package manager (or, in the non-OCaml world, the Git or Mercurial commands). opam exposes commands in this form:

$ opam env
 $ opam remote list -k git
 $ opam install --help
 $ opam install core --verbose
-

The config, remote, and -install keywords form a logical grouping of commands that -factor out a set of flags and arguments. This lets you prevent flags -that are specific to a particular subcommand from leaking into the -general configuration space.   

-

This usually only becomes a concern when your application organically -grows features. Luckily, it’s simple to extend your application to do -this in Command: just use Command.group, which lets you -merge a collection of Command.t’s into one.  

+

The config, remote, and install keywords form a logical grouping of commands that factor out a set of flags and arguments. This lets you prevent flags that are specific to a particular subcommand from leaking into the general configuration space.   

+

This usually only becomes a concern when your application organically grows features. Luckily, it’s simple to extend your application to do this in Command: just use Command.group, which lets you merge a collection of Command.t’s into one.  

Command.group;;
 >- : summary:string ->
@@ -681,13 +487,8 @@ 
Grouping Subcommands Together

 >= <fun>
-

The group signature accepts a list of basic -Command.t values and their corresponding names. When -executed, it looks for the appropriate subcommand from the name list, -and dispatches it to the right command handler.

-

Let’s build the outline of a calendar tool that does a few operations -over dates from the command line. We first need to define a command that -adds days to an input date and prints the resulting date:

+

The group signature accepts a list of basic Command.t values and their corresponding names. When executed, it looks for the appropriate subcommand from the name list, and dispatches it to the right command handler.

+

Let’s build the outline of a calendar tool that does a few operations over dates from the command line. We first need to define a command that adds days to an input date and prints the resulting date:

open Core
 
@@ -701,8 +502,7 @@ 
Grouping Subcommands Together

 
 let () = Command_unix.run add
-

Everything in this command should be familiar to you by now, and it -works as you might expect.

+

Everything in this command should be familiar to you by now, and it works as you might expect.

dune exec -- ./cal.exe -help
 >Add [days] to the [base] date and print day
@@ -719,9 +519,7 @@ 
Grouping Subcommands Together

 >2013-02-03
-

Now, let’s also add the ability to take the difference between two -dates, but, instead of creating a new binary, we’ll group both -operations as subcommands using Command.group.

+

Now, let’s also add the ability to take the difference between two dates, but, instead of creating a new binary, we’ll group both operations as subcommands using Command.group.

open Core
 
@@ -748,9 +546,7 @@ 
Grouping Subcommands Together

 
 let () = Command_unix.run command
-

And that’s all you really need to add subcommand support! Let’s build -the example first in the usual way and inspect the help output, which -now reflects the subcommands we just added.

+

And that’s all you really need to add subcommand support! Let’s build the example first in the usual way and inspect the help output, which now reflects the subcommands we just added.

(executable
   (name       cal)
@@ -771,8 +567,7 @@ 
Grouping Subcommands Together

 >  help                       . explain a given subcommand (perhaps recursively)
-

We can invoke the two commands we just defined to verify that they -work and see the date parsing in action:

+

We can invoke the two commands we just defined to verify that they work and see the date parsing in action:

dune exec -- ./cal.exe add 2012-12-25 40
 >2013-02-03
@@ -783,9 +578,7 @@ 
Grouping Subcommands Together

Prompting for Interactive Input

-

Sometimes, if a value isn’t provided on the command line, you want to -prompt for it instead. Let’s return to the calendar tool we built -before.  

+

Sometimes, if a value isn’t provided on the command line, you want to prompt for it instead. Let’s return to the calendar tool we built before.  

open Core
 
@@ -799,11 +592,7 @@ 
Prompting for Interactive Input

 
 let () = Command_unix.run add
-

This program requires you to specify both the base date -and the number of days to add onto it. If days -isn’t supplied on the command line, an error is output. Now let’s modify -it to interactively prompt for a number of days if only the -base date is supplied.

+

This program requires you to specify both the base date and the number of days to add onto it. If days isn’t supplied on the command line, an error is output. Now let’s modify it to interactively prompt for a number of days if only the base date is supplied.

open Core
 
@@ -830,14 +619,8 @@ 
Prompting for Interactive Input

 
 let () = Command_unix.run add
-

The days anonymous argument is now an optional integer -in the spec, and when it isn’t there, we simply prompt for the value as -part of the ordinary execution of our program.

-

Sometimes, it’s convenient to pack the prompting behavior into the -parser itself. For one thing, this would allow you to easily share the -prompting behavior among multiple commands. This is easy enough to do by -adding a new function, anon_prompt, which creates a parser -that automatically prompts if the value isn’t provided.

+

The days anonymous argument is now an optional integer in the spec, and when it isn’t there, we simply prompt for the value as part of the ordinary execution of our program.

+

Sometimes, it’s convenient to pack the prompting behavior into the parser itself. For one thing, this would allow you to easily share the prompting behavior among multiple commands. This is easy enough to do by adding a new function, anon_prompt, which creates a parser that automatically prompts if the value isn’t provided.

let anon_prompt name of_string =
   let arg = Command.Arg_type.create of_string in
@@ -853,8 +636,7 @@ 
Prompting for Interactive Input

      and days = anon_prompt "days" Int.of_string in
      fun () -> add_days base days)
-

We can see the prompting behavior if we run the program without -providing the second argument.

+

We can see the prompting behavior if we run the program without providing the second argument.

echo 35 | dune exec -- ./cal.exe 2013-12-01
 >enter days: 2014-01-05
@@ -863,52 +645,20 @@ 
Prompting for Interactive Input

Command-Line Autocompletion with bash

-

Modern UNIX shells usually have a tab-completion feature to -interactively help you figure out how to build a command line. These -work by pressing the Tab key in the middle of typing a command, and -seeing the options that pop up. You’ve probably used this most often to -find the files in the current directory, but it can actually be extended -for other parts of the command, too.   

-

The precise mechanism for autocompletion varies depending on what -shell you are using, but we’ll assume you are using the most common one: -bash. This is the default interactive shell on most Linux -distributions and macOS, but you may need to switch to it on *BSD or -Windows (when using Cygwin). The rest of this section assumes that -you’re using bash.  

-

Bash autocompletion isn’t always installed by default, so check your -OS package manager to see if you have it available.

+

Modern UNIX shells usually have a tab-completion feature to interactively help you figure out how to build a command line. These work by pressing the Tab key in the middle of typing a command, and seeing the options that pop up. You’ve probably used this most often to find the files in the current directory, but it can actually be extended for other parts of the command, too.   

+

The precise mechanism for autocompletion varies depending on what shell you are using, but we’ll assume you are using the most common one: bash. This is the default interactive shell on most Linux distributions and macOS, but you may need to switch to it on *BSD or Windows (when using Cygwin). The rest of this section assumes that you’re using bash.  

+

Bash autocompletion isn’t always installed by default, so check your OS package manager to see if you have it available.

  • On Debian Linux, do apt install bash-completion
  • On macOS Homebrew, do brew install bash-completion
  • On FreeBSD, do pkg install bash-completion.
-

Once bash completion is installed and configured, check that -it works by typing the ssh command and pressing the Tab -key. This should show you the list of known hosts from your -~/.ssh/known_hosts file. If it lists some hosts that you’ve -recently connected to, you can continue on. If it lists the files in -your current directory instead, then check your OS documentation to -configure completion correctly.

-

One last bit of information you’ll need to find is the location of -the bash_completion.d directory. This is where -all the shell fragments that contain the completion logic are held. On -Linux, this is often in -/etc/bash_completion.d, and in Homebrew on -macOS, it would be -/usr/local/etc/bash_completion.d by -default.

+

Once bash completion is installed and configured, check that it works by typing the ssh command and pressing the Tab key. This should show you the list of known hosts from your ~/.ssh/known_hosts file. If it lists some hosts that you’ve recently connected to, you can continue on. If it lists the files in your current directory instead, then check your OS documentation to configure completion correctly.

+

One last bit of information you’ll need to find is the location of the bash_completion.d directory. This is where all the shell fragments that contain the completion logic are held. On Linux, this is often in /etc/bash_completion.d, and in Homebrew on macOS, it would be /usr/local/etc/bash_completion.d by default.

Generating Completion Fragments from Command

-

The Command library has a declarative description of all the possible -valid options, and it can use this information to generate a shell -script that provides completion support for that command. To generate -the fragment, just run the command with the -COMMAND_OUTPUT_INSTALLATION_BASH environment variable set -to any value.

-

For example, let’s try it on our MD5 example from earlier, assuming -that the binary is called md5 in the current directory:

+

The Command library has a declarative description of all the possible valid options, and it can use this information to generate a shell script that provides completion support for that command. To generate the fragment, just run the command with the COMMAND_OUTPUT_INSTALLATION_BASH environment variable set to any value.

+

For example, let’s try it on our MD5 example from earlier, assuming that the binary is called md5 in the current directory:

env COMMAND_OUTPUT_INSTALLATION_BASH=1 dune exec -- ./md5.exe
 >function _jsautocom_32087 {
@@ -923,76 +673,39 @@ 
Generating Completion Fragments from Command

 >complete -F _jsautocom_32087 ./md5.exe
-

Recall that we used the Arg_type.file to specify the -argument type. This also supplies the completion logic so that you can -just press Tab to complete files in your current directory.

+

Recall that we used the Arg_type.file to specify the argument type. This also supplies the completion logic so that you can just press Tab to complete files in your current directory.

Installing the Completion Fragment

-

You don’t need to worry about what the preceding output script -actually does (unless you have an unhealthy fascination with shell -scripting internals, that is). Instead, redirect the output to a file in -your current directory and source it into your current shell:

+

You don’t need to worry about what the preceding output script actually does (unless you have an unhealthy fascination with shell scripting internals, that is). Instead, redirect the output to a file in your current directory and source it into your current shell:

$ env COMMAND_OUTPUT_INSTALLATION_BASH=1 ./cal_add_sub_days.native > cal.cmd
 $ . cal.cmd
 $ ./cal_add_sub_days.native <tab>
 add      diff     help     version
-

Command completion support works for flags and grouped commands and -is very useful when building larger command-line interfaces. Don’t -forget to install the shell fragment into your global -bash_completion.d directory if you want it to -be loaded in all of your login shells.  

-
-

Installing a Generic Completion Handler

-

Sadly, bash doesn’t support installing a generic handler -for all Command-based applications. This means you have to install the -completion script for every application, but you should be able to -automate this in the build and packaging system for your -application.

-

It will help to check out how other applications install -tab-completion scripts and follow their lead, as the details are very -OS-specific.

-
+

Command completion support works for flags and grouped commands and is very useful when building larger command-line interfaces. Don’t forget to install the shell fragment into your global bash_completion.d directory if you want it to be loaded in all of your login shells.  

+
+

Installing a Generic Completion Handler

+

Sadly, bash doesn’t support installing a generic handler for all Command-based applications. This means you have to install the completion script for every application, but you should be able to automate this in the build and packaging system for your application.

+

It will help to check out how other applications install tab-completion scripts and follow their lead, as the details are very OS-specific.

+

Alternative Command-Line Parsers

-

This rounds up our tour of the Command library. This isn’t the only -way to parse command-line arguments of course; there are several -alternatives available on opam. Three of the most prominent ones follow: - 

+

This rounds up our tour of the Command library. This isn’t the only way to parse command-line arguments of course; there are several alternatives available on opam. Three of the most prominent ones follow:  

The Arg module
-
-The Arg module is from the OCaml standard library, which is -used by the compiler itself to handle its command-line interface. -Command is built on top of Arg, but you can -also use Arg directly. You can use the -Command.Spec.flags_of_args_exn function to convert -Arg specifications into ones compatible with Command, which -is a simple way of porting an Arg-based command line -interface to Command.   +
The Arg module is from the OCaml standard library, which is used by the compiler itself to handle its command-line interface. Command is built on top of Arg, but you can also use Arg directly. You can use the Command.Spec.flags_of_args_exn function to convert Arg specifications into ones compatible with Command, which is a simple way of porting an Arg-based command line interface to Command.  
ocaml-getopt
-
-ocaml-getopt provides the general command-line syntax of -GNU getopt and getopt_long. The GNU -conventions are widely used in the open source world, and this library -lets your OCaml programs obey the same rules.   +
ocaml-getopt provides the general command-line syntax of GNU getopt and getopt_long. The GNU conventions are widely used in the open source world, and this library lets your OCaml programs obey the same rules.  
Cmdliner
-
-Cmdliner is a mix between the Command and Getopt libraries. It allows -for the declarative definition of command-line interfaces but exposes a -more getopt-like interface. It also automates the -generation of UNIX man pages as part of the specification. Cmdliner is -the parser used by opam to manage its command line.   +
Cmdliner is a mix between the Command and Getopt libraries. It allows for the declarative definition of command-line interfaces but exposes a more getopt-like interface. It also automates the generation of UNIX man pages as part of the specification. Cmdliner is the parser used by opam to manage its command line.  
-

Next: Chapter 16Concurrent Programming with Async

 \ No newline at end of file +

Next: Chapter 16Concurrent Programming with Async

 \ No newline at end of file diff --git a/compiler-backend.html b/compiler-backend.html index 4addcb3fc..283a40a0e 100644 --- a/compiler-backend.html +++ b/compiler-backend.html @@ -1,41 +1,20 @@ -The Compiler Backend: Bytecode and Native code - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+The Compiler Backend: Bytecode and Native code - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

The Compiler Backend: Bytecode and Native code

-

Once OCaml has passed the type checking stage, it can stop emitting -syntax and type errors and begin the process of compiling the -well-formed modules into executable code.

+

Once OCaml has passed the type checking stage, it can stop emitting syntax and type errors and begin the process of compiling the well-formed modules into executable code.

In this chapter, we’ll cover the following topics:

    -

  • The untyped intermediate lambda code where pattern matching is -optimized

    • -

  • The bytecode ocamlc compiler and -ocamlrun interpreter

    • -

  • The native code ocamlopt code generator, and -debugging and profiling native code

    • +

  • The untyped intermediate lambda code where pattern matching is optimized

    • +

  • The bytecode ocamlc compiler and ocamlrun interpreter

    • +

  • The native code ocamlopt code generator, and debugging and profiling native code

The Untyped Lambda Form

-

The first code generation phase eliminates all the static type -information into a simpler intermediate lambda form. The lambda -form discards higher-level constructs such as modules and objects and -replaces them with simpler values such as records and function pointers. -Pattern matches are also analyzed and compiled into highly optimized -automata.  

-

The lambda form is the key stage that discards the OCaml type -information and maps the source code to the runtime memory model -described in Chapter 23, Memory Representation Of Values. This stage also -performs some optimizations, most notably converting pattern-match -statements into more optimized but low-level statements.

+

The first code generation phase eliminates all the static type information into a simpler intermediate lambda form. The lambda form discards higher-level constructs such as modules and objects and replaces them with simpler values such as records and function pointers. Pattern matches are also analyzed and compiled into highly optimized automata.  

+

The lambda form is the key stage that discards the OCaml type information and maps the source code to the runtime memory model described in Chapter 23, Memory Representation Of Values. This stage also performs some optimizations, most notably converting pattern-match statements into more optimized but low-level statements.

Pattern Matching Optimization

-

The compiler dumps the lambda form in an s-expression syntax if you -add the -dlambda directive to the command line. Let’s use -this to learn more about how the OCaml pattern-matching engine works by -building three different pattern matches and comparing their lambda -forms.  

-

Let’s start by creating a straightforward exhaustive pattern match -using four normal variants:

+

The compiler dumps the lambda form in an s-expression syntax if you add the -dlambda directive to the command line. Let’s use this to learn more about how the OCaml pattern-matching engine works by building three different pattern matches and comparing their lambda forms.  

+

Let’s start by creating a straightforward exhaustive pattern match using four normal variants:

type t = | Alice | Bob | Charlie | David
 
@@ -61,31 +40,13 @@ 
Pattern Matching Optimization

 >    (makeblock 0 test/272)))
-

It’s not important to understand every detail of this internal form, -and it is explicitly undocumented since it can change across compiler -revisions. Despite these caveats, some interesting points emerge from -reading it:

+

It’s not important to understand every detail of this internal form, and it is explicitly undocumented since it can change across compiler revisions. Despite these caveats, some interesting points emerge from reading it:

    -

  • There are no mentions of modules or types any more. Global values -are created via setglobal, and OCaml values are constructed -by makeblock. The blocks are the runtime values you should -remember from Chapter 23, Memory Representation Of Values.

    • -

  • The pattern match has turned into a switch case that jumps to the -right case depending on the header tag of v. Recall that -variants without parameters are stored in memory as integers in the -order in which they appear. The pattern-matching engine knows this and -has transformed the pattern into an efficient jump table.

    • -

  • Values are addressed by a unique name that distinguishes shadowed -values by appending a number (e.g., v/1014). The type -safety checks in the earlier phase ensure that these low-level accesses -never violate runtime memory safety, so this layer doesn’t do any -dynamic checks. Unwise use of unsafe features such as the -Obj.magic module can still easily induce crashes at this -level.

    • +

  • There are no mentions of modules or types any more. Global values are created via setglobal, and OCaml values are constructed by makeblock. The blocks are the runtime values you should remember from Chapter 23, Memory Representation Of Values.

    • +

  • The pattern match has turned into a switch case that jumps to the right case depending on the header tag of v. Recall that variants without parameters are stored in memory as integers in the order in which they appear. The pattern-matching engine knows this and has transformed the pattern into an efficient jump table.

    • +

  • Values are addressed by a unique name that distinguishes shadowed values by appending a number (e.g., v/1014). The type safety checks in the earlier phase ensure that these low-level accesses never violate runtime memory safety, so this layer doesn’t do any dynamic checks. Unwise use of unsafe features such as the Obj.magic module can still easily induce crashes at this level.

-

The compiler computes a jump table in order to handle all four cases. -If we drop the number of variants to just two, then there’s no need for -the complexity of computing this table:

+

The compiler computes a jump table in order to handle all four cases. If we drop the number of variants to just two, then there’s no need for the complexity of computing this table:

type t = | Alice | Bob
 
@@ -102,11 +63,7 @@ 
Pattern Matching Optimization

 >    (makeblock 0 test/270)))
-

The compiler emits simpler conditional jumps rather than setting up a -jump table, since it statically determines that the range of possible -variants is small enough. Finally, let’s consider code that’s -essentially the same as our first pattern match example, but with -polymorphic variants instead of normal variants:

+

The compiler emits simpler conditional jumps rather than setting up a jump table, since it statically determines that the range of possible variants is small enough. Finally, let’s consider code that’s essentially the same as our first pattern match example, but with polymorphic variants instead of normal variants:

let test v =
   match v with
@@ -115,8 +72,7 @@ 
Pattern Matching Optimization

   | `Charlie -> 102
   | `David   -> 103
-

The lambda form for this also reflects the runtime representation of -polymorphic variants:

+

The lambda form for this also reflects the runtime representation of polymorphic variants:

ocamlc -dlambda -c pattern_polymorphic.ml 2>&1
 >(setglobal Pattern_polymorphic!
@@ -128,34 +84,13 @@ 
Pattern Matching Optimization

 >    (makeblock 0 test/267)))
-

We mentioned in Chapter 6, Variants that pattern matching over polymorphic -variants is slightly less efficient, and it should be clearer why this -is the case now. Polymorphic variants have a runtime value that’s -calculated by hashing the variant name, and so the compiler can’t use a -jump table as it does for normal variants. Instead, it creates a -decision tree that compares the hash values against the input variable -in as few comparisons as possible.  

-

Pattern matching is an important part of OCaml programming. You’ll -often encounter deeply nested pattern matches over complex data -structures in real code. A good paper that describes the fundamental -algorithms implemented in OCaml is “Optimizing pattern -matching” by Fabrice Le Fessant and Luc Maranget.

-

The paper describes the backtracking algorithm used in classical -pattern matching compilation, and also several OCaml-specific -optimizations, such as the use of exhaustiveness information and control -flow optimizations via static exceptions. It’s not essential that you -understand all of this just to use pattern matching, of course, but -it’ll give you insight as to why pattern matching is such an efficient -language construct in OCaml.

+

We mentioned in Chapter 6, Variants that pattern matching over polymorphic variants is slightly less efficient, and it should be clearer why this is the case now. Polymorphic variants have a runtime value that’s calculated by hashing the variant name, and so the compiler can’t use a jump table as it does for normal variants. Instead, it creates a decision tree that compares the hash values against the input variable in as few comparisons as possible.  

+

Pattern matching is an important part of OCaml programming. You’ll often encounter deeply nested pattern matches over complex data structures in real code. A good paper that describes the fundamental algorithms implemented in OCaml is “Optimizing pattern matching” by Fabrice Le Fessant and Luc Maranget.

+

The paper describes the backtracking algorithm used in classical pattern matching compilation, and also several OCaml-specific optimizations, such as the use of exhaustiveness information and control flow optimizations via static exceptions. It’s not essential that you understand all of this just to use pattern matching, of course, but it’ll give you insight as to why pattern matching is such an efficient language construct in OCaml.

Benchmarking Pattern Matching

-

Let’s benchmark these three pattern-matching techniques to quantify -their runtime costs more accurately. The Core_bench module -runs the tests thousands of times and also calculates statistical -variance of the results. You’ll need to -opam install core_bench to get the library:  

+

Let’s benchmark these three pattern-matching techniques to quantify their runtime costs more accurately. The Core_bench module runs the tests thousands of times and also calculates statistical variance of the results. You’ll need to opam install core_bench to get the library:  

open Core
 open Core_bench
@@ -229,8 +164,7 @@ 
Benchmarking Pattern Matching

   |> Bench.make_command
   |> Command_unix.run
-

Building and executing this example will run for around 30 seconds by -default, and you’ll see the results summarized in a neat table:

+

Building and executing this example will run for around 30 seconds by default, and you’ll see the results summarized in a neat table:

dune exec -- ./bench_patterns.exe -ascii -quota 0.25
 >Estimated testing time 750ms (3 benchmarks x 250ms). Change using '-quota'.
@@ -242,66 +176,36 @@ 
Benchmarking Pattern Matching

 >  Polymorphic large pattern     9.63ns       99.97%
-

These results confirm the performance hypothesis that we obtained -earlier by inspecting the lambda code. The shortest running time comes -from the small conditional pattern match, and polymorphic variant -pattern matching is the slowest. There isn’t a hugely significant -difference in these examples, but you can use the same techniques to -peer into the innards of your own source code and narrow down any -performance hotspots.

-

The lambda form is primarily a stepping stone to the bytecode -executable format that we’ll cover next. It’s often easier to look at -the textual output from this stage than to wade through the native -assembly code from compiled executables.

+

These results confirm the performance hypothesis that we obtained earlier by inspecting the lambda code. The shortest running time comes from the small conditional pattern match, and polymorphic variant pattern matching is the slowest. There isn’t a hugely significant difference in these examples, but you can use the same techniques to peer into the innards of your own source code and narrow down any performance hotspots.

+

The lambda form is primarily a stepping stone to the bytecode executable format that we’ll cover next. It’s often easier to look at the textual output from this stage than to wade through the native assembly code from compiled executables.

Generating Portable Bytecode

-

After the lambda form has been generated, we are very close to having -executable code. The OCaml toolchain branches into two separate -compilers at this point. We’ll describe the bytecode compiler first, -which consists of two pieces:        

+

After the lambda form has been generated, we are very close to having executable code. The OCaml toolchain branches into two separate compilers at this point. We’ll describe the bytecode compiler first, which consists of two pieces:        

ocamlc
-
-Compiles files into a bytecode that is a close mapping to the lambda -form +
Compiles files into a bytecode that is a close mapping to the lambda form
ocamlrun
-
-A portable interpreter that executes the bytecode +
A portable interpreter that executes the bytecode
-

The big advantage of using bytecode is simplicity, portability, and -compilation speed. The mapping from the lambda form to bytecode is -straightforward, and this results in predictable (but slow) execution -speed.

-

The bytecode interpreter implements a stack-based virtual machine. -The OCaml stack and an associated accumulator store values that consist -of:    

+

The big advantage of using bytecode is simplicity, portability, and compilation speed. The mapping from the lambda form to bytecode is straightforward, and this results in predictable (but slow) execution speed.

+

The bytecode interpreter implements a stack-based virtual machine. The OCaml stack and an associated accumulator store values that consist of:    

long
-
-Values that correspond to an OCaml int type +
Values that correspond to an OCaml int type
block
-
-Values that contain the block header and a memory address with the data -fields that contain further OCaml values indexed by an integer +
Values that contain the block header and a memory address with the data fields that contain further OCaml values indexed by an integer
code offset
-
-Values that are relative to the starting code address +
Values that are relative to the starting code address
-

The interpreter virtual machine only has seven registers in total: - -program counter, - stack, exception and argument pointers, - -accumulator, - environment and global data.

-

You can display the bytecode instructions in textual form via --dinstr. Try this on one of our earlier pattern-matching -examples:

+

The interpreter virtual machine only has seven registers in total: - program counter, - stack, exception and argument pointers, - accumulator, - environment and global data.

+

You can display the bytecode instructions in textual form via -dinstr. Try this on one of our earlier pattern-matching examples:

ocamlc -dinstr pattern_monomorphic_small.ml 2>&1
 >   branch L2
@@ -319,93 +223,38 @@ 
Generating Portable Bytecode

 >   setglobal Pattern_monomorphic_small!
-

The preceding bytecode has been simplified from the lambda form into -a set of simple instructions that are executed serially by the -interpreter.

-

There are around 140 instructions in total, but most are just minor -variants of commonly encountered operations (e.g., function application -at a specific arity). You can find full details online. - 

-
-

Where Did the Bytecode Instruction Set Come From?

-

The bytecode interpreter is much slower than compiled native code, -but is still remarkably performant for an interpreter without a JIT -compiler. Its efficiency can be traced back to Xavier Leroy’s -ground-breaking work in 1990, “The ZINC -experiment: An Economical Implementation of the ML Language”.

-

This paper laid the theoretical basis for the implementation of an -instruction set for a strictly evaluated functional language such as -OCaml. The bytecode interpreter in modern OCaml is still based on the -ZINC model. The native code compiler uses a different model since it -uses CPU registers for function calls instead of always passing -arguments on the stack, as the bytecode interpreter does.

-

Understanding the reasoning behind the different implementations of -the bytecode interpreter and the native compiler is a very useful -exercise for any budding language hacker.

-
+

The preceding bytecode has been simplified from the lambda form into a set of simple instructions that are executed serially by the interpreter.

+

There are around 140 instructions in total, but most are just minor variants of commonly encountered operations (e.g., function application at a specific arity). You can find full details online.  

+
+

Where Did the Bytecode Instruction Set Come From?

+

The bytecode interpreter is much slower than compiled native code, but is still remarkably performant for an interpreter without a JIT compiler. Its efficiency can be traced back to Xavier Leroy’s ground-breaking work in 1990, “The ZINC experiment: An Economical Implementation of the ML Language”.

+

This paper laid the theoretical basis for the implementation of an instruction set for a strictly evaluated functional language such as OCaml. The bytecode interpreter in modern OCaml is still based on the ZINC model. The native code compiler uses a different model since it uses CPU registers for function calls instead of always passing arguments on the stack, as the bytecode interpreter does.

+

Understanding the reasoning behind the different implementations of the bytecode interpreter and the native compiler is a very useful exercise for any budding language hacker.

+

Compiling and Linking Bytecode

-

The ocamlc command compiles individual ml -files into bytecode files that have a cmo extension. The -compiled bytecode files are matched with the associated cmi -interface, which contains the type signature exported to other -compilation units.  

-

A typical OCaml library consists of multiple source files, and hence -multiple cmo files that all need to be passed as -command-line arguments to use the library from other code. The compiler -can combine these multiple files into a more convenient single archive -file by using the -a flag. Bytecode archives are denoted by -the cma extension.

-

The individual objects in the library are linked as regular -cmo files in the order specified when the library file was -built. If an object file within the library isn’t referenced elsewhere -in the program, then it isn’t included in the final binary unless the --linkall flag forces its inclusion. This behavior is -analogous to how C handles object files and archives (.o -and .a, respectively).

-

The bytecode files are then linked together with the OCaml standard -library to produce an executable program. The order in which -.cmo arguments are presented on the command line defines -the order in which compilation units are initialized at runtime. -Remember that OCaml has no single main function like C, so -this link order is more important than in C programs.

+

The ocamlc command compiles individual ml files into bytecode files that have a cmo extension. The compiled bytecode files are matched with the associated cmi interface, which contains the type signature exported to other compilation units.  

+

A typical OCaml library consists of multiple source files, and hence multiple cmo files that all need to be passed as command-line arguments to use the library from other code. The compiler can combine these multiple files into a more convenient single archive file by using the -a flag. Bytecode archives are denoted by the cma extension.

+

The individual objects in the library are linked as regular cmo files in the order specified when the library file was built. If an object file within the library isn’t referenced elsewhere in the program, then it isn’t included in the final binary unless the -linkall flag forces its inclusion. This behavior is analogous to how C handles object files and archives (.o and .a, respectively).

+

The bytecode files are then linked together with the OCaml standard library to produce an executable program. The order in which .cmo arguments are presented on the command line defines the order in which compilation units are initialized at runtime. Remember that OCaml has no single main function like C, so this link order is more important than in C programs.

Executing Bytecode

-

The bytecode runtime comprises three parts: the bytecode interpreter, -GC, and a set of C functions that implement the primitive operations. -The bytecode contains instructions to call these C functions when -required.

-

The OCaml linker produces bytecode that targets the standard OCaml -runtime by default, and so needs to know about any C functions that are -referenced from other libraries that aren’t loaded by default.

-

Information about these extra libraries can be specified while -linking a bytecode archive:

+

The bytecode runtime comprises three parts: the bytecode interpreter, GC, and a set of C functions that implement the primitive operations. The bytecode contains instructions to call these C functions when required.

+

The OCaml linker produces bytecode that targets the standard OCaml runtime by default, and so needs to know about any C functions that are referenced from other libraries that aren’t loaded by default.

+

Information about these extra libraries can be specified while linking a bytecode archive:

$ ocamlc -a -o mylib.cma a.cmo b.cmo -dllib -lmylib
-

The dllib flag embeds the arguments in the archive file. -Any subsequent packages linking this archive will also include the extra -C linking directive. This in turn lets the interpreter dynamically load -the external library symbols when it executes the bytecode.

-

You can also generate a complete standalone executable that bundles -the ocamlrun interpreter with the bytecode in a single -binary. This is known as a custom runtime mode and is built as -follows:  

+

The dllib flag embeds the arguments in the archive file. Any subsequent packages linking this archive will also include the extra C linking directive. This in turn lets the interpreter dynamically load the external library symbols when it executes the bytecode.

+

You can also generate a complete standalone executable that bundles the ocamlrun interpreter with the bytecode in a single binary. This is known as a custom runtime mode and is built as follows:  

$ ocamlc -a -o mylib.cma -custom a.cmo b.cmo -cclib -lmylib
-

The custom mode is the most similar mode to native code compilation, -as both generate standalone executables. There are quite a few other -options available for compiling bytecode (notably with shared libraries -or building custom runtimes). Full details can be found in the OCaml.

-

Dune can build a self-contained bytecode executable if you specify -the byte_complete mode in the executable rule. For example, -this dune file will generate a prog.bc.exe -target:

+

The custom mode is the most similar mode to native code compilation, as both generate standalone executables. There are quite a few other options available for compiling bytecode (notably with shared libraries or building custom runtimes). Full details can be found in the OCaml.

+

Dune can build a self-contained bytecode executable if you specify the byte_complete mode in the executable rule. For example, this dune file will generate a prog.bc.exe target:

(executable
   (name prog)
@@ -415,22 +264,9 @@ 
Executing Bytecode

Embedding OCaml Bytecode in C

-

A consequence of using the bytecode compiler is that the final link -phase must be performed by ocamlc. However, you might -sometimes want to embed your OCaml code inside an existing C -application. OCaml also supports this mode of operation via the --output-obj directive. 

-

This mode causes ocamlc to output an object file -containing the bytecode for the OCaml part of the program, as well as a -caml_startup function. All of the OCaml modules are linked -into this object file as bytecode, just as they would be for an -executable.

-

This object file can then be linked with C code using the standard C -compiler, needing only the bytecode runtime library (which is installed -as libcamlrun.a). Creating an executable just requires you -to link the runtime library with the bytecode object file. Here’s an -example to show how it all fits together.

+

A consequence of using the bytecode compiler is that the final link phase must be performed by ocamlc. However, you might sometimes want to embed your OCaml code inside an existing C application. OCaml also supports this mode of operation via the -output-obj directive. 

+

This mode causes ocamlc to output an object file containing the bytecode for the OCaml part of the program, as well as a caml_startup function. All of the OCaml modules are linked into this object file as bytecode, just as they would be for an executable.

+

This object file can then be linked with C code using the standard C compiler, needing only the bytecode runtime library (which is installed as libcamlrun.a). Creating an executable just requires you to link the runtime library with the bytecode object file. Here’s an example to show how it all fits together.

Create two OCaml source files that contain a single print line:

let () = print_endline "hello embedded world 1"
@@ -462,10 +298,7 @@

Embedding OCaml Bytecode in C

ocamlc -output-obj -o embed_out.o embed_me1.ml embed_me2.ml
-

After this point, you no longer need the OCaml compiler, as -embed_out.o has all of the OCaml code compiled and linked -into a single object file. Compile an output binary using -gcc to test this out:

+

After this point, you no longer need the OCaml compiler, as embed_out.o has all of the OCaml code compiled and linked into a single object file. Compile an output binary using gcc to test this out:

$ gcc -fPIC -Wall -I`ocamlc -where` -L`ocamlc -where` -ltermcap -lm -ldl \
   -o finalbc.native main.c embed_out.o -lcamlrun
@@ -475,99 +308,43 @@ 
Embedding OCaml Bytecode in C

 hello embedded world 2
 After calling OCaml
-

You can inspect the commands that ocamlc is invoking by -adding -verbose to the command line to help figure out the -GCC command line if you get stuck. You can even obtain the C source code -to the -output-obj result by specifying a .c -output file extension instead of the .o we used -earlier:

+

You can inspect the commands that ocamlc is invoking by adding -verbose to the command line to help figure out the GCC command line if you get stuck. You can even obtain the C source code to the -output-obj result by specifying a .c output file extension instead of the .o we used earlier:

ocamlc -output-obj -o embed_out.c embed_me1.ml embed_me2.ml
-

Embedding OCaml code like this lets you write OCaml that interfaces -with any environment that works with a C compiler. You can even cross -back from the C code into OCaml by using the Callback -module to register named entry points in the OCaml code. This is -explained in detail in the interfacing with C -section of the OCaml manual.

+

Embedding OCaml code like this lets you write OCaml that interfaces with any environment that works with a C compiler. You can even cross back from the C code into OCaml by using the Callback module to register named entry points in the OCaml code. This is explained in detail in the interfacing with C section of the OCaml manual.

Compiling Fast Native Code

-

The native code compiler is ultimately the tool that most production -OCaml code goes through. It compiles the lambda form into fast native -code executables, with cross-module inlining and additional optimization -passes that the bytecode interpreter doesn’t perform. Care is taken to -ensure compatibility with the bytecode runtime, so the same code should -run identically when compiled with either toolchain.          

-

The ocamlopt command is the frontend to the native code -compiler and has a very similar interface to ocamlc. It -also accepts ml and mli files, but compiles -them to:

+

The native code compiler is ultimately the tool that most production OCaml code goes through. It compiles the lambda form into fast native code executables, with cross-module inlining and additional optimization passes that the bytecode interpreter doesn’t perform. Care is taken to ensure compatibility with the bytecode runtime, so the same code should run identically when compiled with either toolchain.          

+

The ocamlopt command is the frontend to the native code compiler and has a very similar interface to ocamlc. It also accepts ml and mli files, but compiles them to:

  • A .o file containing native object code

    • -

  • A .cmx file containing extra information for linking -and cross-module optimization

    • -

  • A .cmi compiled interface file that is the same as -the bytecode compiler

    • +

  • A .cmx file containing extra information for linking and cross-module optimization

    • +

  • A .cmi compiled interface file that is the same as the bytecode compiler

-

When the compiler links modules together into an executable, it uses -the contents of the cmx files to perform cross-module -inlining across compilation units. This can be a significant speedup for -standard library functions that are frequently used outside of their -module.

-

Collections of .cmx and .o files can also -be linked into a .cmxa archive by passing the --a flag to the compiler. However, unlike the bytecode -version, you must keep the individual cmx files in the -compiler search path so that they are available for cross-module -inlining. If you don’t do this, the compilation will still succeed, but -you will have missed out on an important optimization and have slower -binaries.

+

When the compiler links modules together into an executable, it uses the contents of the cmx files to perform cross-module inlining across compilation units. This can be a significant speedup for standard library functions that are frequently used outside of their module.

+

Collections of .cmx and .o files can also be linked into a .cmxa archive by passing the -a flag to the compiler. However, unlike the bytecode version, you must keep the individual cmx files in the compiler search path so that they are available for cross-module inlining. If you don’t do this, the compilation will still succeed, but you will have missed out on an important optimization and have slower binaries.

Inspecting Assembly Output

-

The native code compiler generates assembly language that is then -passed to the system assembler for compiling into object files. You can -get ocamlopt to output the assembly by passing the --S flag to the compiler command line. 

-

The assembly code is highly architecture-specific, so the following -discussion assumes an Intel or AMD 64-bit platform. We’ve generated the -example code using -inline 20 and -nodynlink -since it’s best to generate assembly code with the full optimizations -that the compiler supports. Even though these optimizations make the -code a bit harder to read, it will give you a more accurate picture of -what executes on the CPU. Don’t forget that you can use the lambda code -from earlier to get a slightly higher-level picture of the code if you -get lost in the more verbose assembly.

+

The native code compiler generates assembly language that is then passed to the system assembler for compiling into object files. You can get ocamlopt to output the assembly by passing the -S flag to the compiler command line. 

+

The assembly code is highly architecture-specific, so the following discussion assumes an Intel or AMD 64-bit platform. We’ve generated the example code using -inline 20 and -nodynlink since it’s best to generate assembly code with the full optimizations that the compiler supports. Even though these optimizations make the code a bit harder to read, it will give you a more accurate picture of what executes on the CPU. Don’t forget that you can use the lambda code from earlier to get a slightly higher-level picture of the code if you get lost in the more verbose assembly.

The Impact of Polymorphic Comparison

-

We warned you in Chapter 14, Maps And Hash Tables that using polymorphic -comparison is both convenient and perilous. Let’s look at precisely what -the difference is at the assembly language level now. 

-

First let’s create a comparison function where we’ve explicitly -annotated the types, so the compiler knows that only integers are being -compared:

+

We warned you in Chapter 14, Maps And Hash Tables that using polymorphic comparison is both convenient and perilous. Let’s look at precisely what the difference is at the assembly language level now. 

+

First let’s create a comparison function where we’ve explicitly annotated the types, so the compiler knows that only integers are being compared:

let cmp (a:int) (b:int) =
   if a > b then a else b
-

Now compile this into assembly and read the resulting -compare_mono.S file.

+

Now compile this into assembly and read the resulting compare_mono.S file.

ocamlopt -S compare_mono.ml
-

This file extension may be lowercase on some platforms such as Linux. -If you’ve never seen assembly language before, then the contents may be -rather scary. While you’ll need to learn x86 assembly to fully -understand it, we’ll try to give you some basic instructions to spot -patterns in this section. The excerpt of the implementation of the -cmp function can be found below:

+

This file extension may be lowercase on some platforms such as Linux. If you’ve never seen assembly language before, then the contents may be rather scary. While you’ll need to learn x86 assembly to fully understand it, we’ll try to give you some basic instructions to spot patterns in this section. The excerpt of the implementation of the cmp function can be found below:

_camlCompare_mono__cmp_1008:
         .cfi_startproc
@@ -581,24 +358,13 @@ 
The Impact of Polymorphic Comparison

         ret
         .cfi_endproc
-

The _camlCompare_mono__cmp_1008 is an assembly label -that has been computed from the module name (Compare_mono) -and the function name (cmp_1008). The numeric suffix for -the function name comes straight from the lambda form (which you can -inspect using -dlambda, but in this case isn’t -necessary).

-

The arguments to cmp are passed in the %rbx -and %rax registers, and compared using the jle -“jump if less than or equal” instruction. This requires both the -arguments to be immediate integers to work. Now let’s see what happens -if our OCaml code omits the type annotations and is a polymorphic -comparison instead:

+

The _camlCompare_mono__cmp_1008 is an assembly label that has been computed from the module name (Compare_mono) and the function name (cmp_1008). The numeric suffix for the function name comes straight from the lambda form (which you can inspect using -dlambda, but in this case isn’t necessary).

+

The arguments to cmp are passed in the %rbx and %rax registers, and compared using the jle “jump if less than or equal” instruction. This requires both the arguments to be immediate integers to work. Now let’s see what happens if our OCaml code omits the type annotations and is a polymorphic comparison instead:

let cmp a b =
   if a > b then a else b
-

Compiling this code with -S results in a significantly -more complex assembly output for the same function:

+

Compiling this code with -S results in a significantly more complex assembly output for the same function:

_camlCompare_poly__cmp_1008:
         .cfi_startproc
@@ -630,29 +396,12 @@ 
The Impact of Polymorphic Comparison

         .cfi_adjust_cfa_offset  24
         .cfi_endproc
-

The .cfi directives are assembler hints that contain -Call Frame Information that lets the debugger provide more sensible -backtraces, and they have no effect on runtime performance. Notice that -the rest of the implementation is no longer a simple register -comparison. Instead, the arguments are pushed on the stack (the -%rsp register), and a C function call is invoked by placing -a pointer to caml_greaterthan in %rax and -jumping to caml_c_call.  

-

OCaml on x86_64 architectures caches the location of the minor heap -in the %r15 register since it’s so frequently referenced in -OCaml functions. The minor heap pointer can also be changed by the C -code that’s being called (e.g., when it allocates OCaml values), and so -%r15 is restored after returning from the -caml_greaterthan call. Finally, the return value of the -comparison is popped from the stack and returned.

+

The .cfi directives are assembler hints that contain Call Frame Information that lets the debugger provide more sensible backtraces, and they have no effect on runtime performance. Notice that the rest of the implementation is no longer a simple register comparison. Instead, the arguments are pushed on the stack (the %rsp register), and a C function call is invoked by placing a pointer to caml_greaterthan in %rax and jumping to caml_c_call.  

+

OCaml on x86_64 architectures caches the location of the minor heap in the %r15 register since it’s so frequently referenced in OCaml functions. The minor heap pointer can also be changed by the C code that’s being called (e.g., when it allocates OCaml values), and so %r15 is restored after returning from the caml_greaterthan call. Finally, the return value of the comparison is popped from the stack and returned.

Benchmarking Polymorphic Comparison

-

You don’t have to fully understand the intricacies of assembly -language to see that this polymorphic comparison is much heavier than -the simple monomorphic integer comparison from earlier. Let’s confirm -this hypothesis again by writing a quick Core_bench test -with both functions:

+

You don’t have to fully understand the intricacies of assembly language to see that this polymorphic comparison is much heavier than the simple monomorphic integer comparison from earlier. Let’s confirm this hypothesis again by writing a quick Core_bench test with both functions:

open Core
 open Core_bench
@@ -678,8 +427,7 @@ 
Benchmarking Polymorphic Comparison

   |> Bench.make_command
   |> Command_unix.run
-

Running this shows quite a significant runtime difference between the -two:

+

Running this shows quite a significant runtime difference between the two:

dune exec -- ./bench_poly_and_mono.exe -ascii -quota 1
 >Estimated testing time 2s (2 benchmarks x 1s). Change using '-quota'.
@@ -690,75 +438,33 @@ 
Benchmarking Polymorphic Comparison

 >  Monomorphic comparison     471.75ns       11.65%
-

We see that the polymorphic comparison is close to 10 times slower! -These results shouldn’t be taken too seriously, as this is a very narrow -test that, like all such microbenchmarks, isn’t representative of more -complex codebases. However, if you’re building numerical code that runs -many iterations in a tight inner loop, it’s worth manually peering at -the produced assembly code to see if you can hand-optimize it.

-
-
-

Accessing Stdlib Modules from Within Core

-

In the benchmark above comparing polymorphic and monomorphic -comparison, you may have noticed that we prepended the comparison -functions with Stdlib. This is because the Core module -explicitly redefines the > and < and -= operators to be specialized for operating over -int types, as explained in Chapter 14, Maps and Hashtables. You can always recover any of -the OCaml standard library functions by accessing them through the -Stdlib module, as we did in our benchmark.

+

We see that the polymorphic comparison is close to 10 times slower! These results shouldn’t be taken too seriously, as this is a very narrow test that, like all such microbenchmarks, isn’t representative of more complex codebases. However, if you’re building numerical code that runs many iterations in a tight inner loop, it’s worth manually peering at the produced assembly code to see if you can hand-optimize it.

+
+

Accessing Stdlib Modules from Within Core

+

In the benchmark above comparing polymorphic and monomorphic comparison, you may have noticed that we prepended the comparison functions with Stdlib. This is because the Core module explicitly redefines the > and < and = operators to be specialized for operating over int types, as explained in Chapter 14, Maps and Hashtables. You can always recover any of the OCaml standard library functions by accessing them through the Stdlib module, as we did in our benchmark.

+

Debugging Native Code Binaries

-

The native code compiler builds executables that can be debugged -using conventional system debuggers such as GNU gdb. You -need to compile your libraries with the -g option to add -the debug information to the output, just as you need to with C -compilers.   

-

Extra debugging information is inserted into the output assembly when -the library is compiled in debug mode. These include the CFI stubs you -will have noticed in the profiling output earlier -(.cfi_start_proc and .cfi_end_proc to delimit -an OCaml function call, for example).

+

The native code compiler builds executables that can be debugged using conventional system debuggers such as GNU gdb. You need to compile your libraries with the -g option to add the debug information to the output, just as you need to with C compilers.   

+

Extra debugging information is inserted into the output assembly when the library is compiled in debug mode. These include the CFI stubs you will have noticed in the profiling output earlier (.cfi_start_proc and .cfi_end_proc to delimit an OCaml function call, for example).

Understanding Name Mangling

-

So how do you refer to OCaml functions in an interactive debugger -like gdb? The first thing you need to know is how OCaml -function names compile down to symbol names in the compiled object -files, a procedure generally called name mangling.    

-

Each OCaml source file is compiled into a native object file that -must export a unique set of symbols to comply with the C binary -interface. This means that any OCaml values that may be used by another -compilation unit need to be mapped onto a symbol name. This mapping has -to account for OCaml language features such as nested modules, anonymous -functions, and variable names that shadow one another.

-

The conversion follows some straightforward rules for named variables -and functions:

+

So how do you refer to OCaml functions in an interactive debugger like gdb? The first thing you need to know is how OCaml function names compile down to symbol names in the compiled object files, a procedure generally called name mangling.    

+

Each OCaml source file is compiled into a native object file that must export a unique set of symbols to comply with the C binary interface. This means that any OCaml values that may be used by another compilation unit need to be mapped onto a symbol name. This mapping has to account for OCaml language features such as nested modules, anonymous functions, and variable names that shadow one another.

+

The conversion follows some straightforward rules for named variables and functions:

    -

  • The symbol is prefixed by caml and the local module -name, with dots replaced by underscores.

    • -

  • This is followed by a double __ suffix and the -variable name.

    • -

  • The variable name is also suffixed by a _ and a -number. This is the result of the lambda compilation, which replaces -each variable name with a unique value within the module. You can -determine this number by examining the -dlambda output from -ocamlopt.

    • +

  • The symbol is prefixed by caml and the local module name, with dots replaced by underscores.

    • +

  • This is followed by a double __ suffix and the variable name.

    • +

  • The variable name is also suffixed by a _ and a number. This is the result of the lambda compilation, which replaces each variable name with a unique value within the module. You can determine this number by examining the -dlambda output from ocamlopt.

-

Anonymous functions are hard to predict without inspecting -intermediate compiler output. If you need to debug them, it’s usually -easier to modify the source code to let-bind the anonymous function to a -variable name.

+

Anonymous functions are hard to predict without inspecting intermediate compiler output. If you need to debug them, it’s usually easier to modify the source code to let-bind the anonymous function to a variable name.

Interactive Breakpoints with the GNU Debugger

-

Let’s see name mangling in action with some interactive debugging -using GNU gdb.  

-

Let’s write a mutually recursive function that selects alternating -values from a list. This isn’t tail-recursive, so our stack size will -grow as we single-step through the execution:

+

Let’s see name mangling in action with some interactive debugging using GNU gdb.  

+

Let’s write a mutually recursive function that selects alternating values from a list. This isn’t tail-recursive, so our stack size will grow as we single-step through the execution:

open Core
 
@@ -777,8 +483,7 @@ 
Interactive Breakpoints with the GNU Debugger

   |> String.concat ~sep:","
   |> print_endline
-

Compile and run this with debugging symbols. You should see the -following output:

+

Compile and run this with debugging symbols. You should see the following output:

(executable
   (name      alternate_list)
@@ -805,17 +510,12 @@ 
Interactive Breakpoints with the GNU Debugger

 Reading symbols from /home/avsm/alternate_list.native...done.
 (gdb)
-

The gdb prompt lets you enter debug directives. Let’s -set the program to break just before the first call to -take:

+

The gdb prompt lets you enter debug directives. Let’s set the program to break just before the first call to take:

(gdb) break camlAlternate_list__take_69242
 Breakpoint 1 at 0x5658d0: file alternate_list.ml, line 5.
-

We used the C symbol name by following the name mangling rules -defined earlier. A convenient way to figure out the full name is by tab -completion. Just type in a portion of the name and press the <tab> -key to see a list of possible completions.

+

We used the C symbol name by following the name mangling rules defined earlier. A convenient way to figure out the full name is by tab completion. Just type in a portion of the name and press the <tab> key to see a list of possible completions.

Once you’ve set the breakpoint, start the program executing:

(gdb) run
@@ -826,10 +526,7 @@ 
Interactive Breakpoints with the GNU Debugger

 Breakpoint 1, camlAlternate_list__take_69242 () at alternate_list.ml:5
 4         function
-

The binary has run until the first take invocation and -stopped, waiting for further instructions. GDB has lots of features, so -let’s continue the program and check the backtrace after a couple of -recursions:

+

The binary has run until the first take invocation and stopped, waiting for further instructions. GDB has lots of features, so let’s continue the program and check the backtrace after a couple of recursions:

(gdb) cont
 Continuing.
@@ -857,60 +554,28 @@ 
Interactive Breakpoints with the GNU Debugger

 1,3,5,7,9
 [Inferior 1 (process 3546) exited normally]
-

The cont command resumes execution after a breakpoint -has paused it, bt displays a stack backtrace, and -clear deletes the breakpoint so the application can execute -until completion. GDB has a host of other features we won’t cover here, -but you can view more guidelines via Mark Shinwell’s talk on “Real-world -debugging in OCaml.”

-

One very useful feature of OCaml native code is that C and OCaml -share the same stack. This means that GDB backtraces can give you a -combined view of what’s going on in your program and runtime -library. This includes any calls to C libraries or even callbacks into -OCaml from the C layer if you’re in an environment which embeds the -OCaml runtime as a library.

+

The cont command resumes execution after a breakpoint has paused it, bt displays a stack backtrace, and clear deletes the breakpoint so the application can execute until completion. GDB has a host of other features we won’t cover here, but you can view more guidelines via Mark Shinwell’s talk on “Real-world debugging in OCaml.”

+

One very useful feature of OCaml native code is that C and OCaml share the same stack. This means that GDB backtraces can give you a combined view of what’s going on in your program and runtime library. This includes any calls to C libraries or even callbacks into OCaml from the C layer if you’re in an environment which embeds the OCaml runtime as a library.

Profiling Native Code

-

The recording and analysis of where your application spends its -execution time is known as performance profiling. OCaml native -code binaries can be profiled just like any other C binary, by using the -name mangling described earlier to map between OCaml variable names and -the profiler output.    

-

Most profiling tools benefit from having some instrumentation -included in the binary. OCaml supports two such tools:

+

The recording and analysis of where your application spends its execution time is known as performance profiling. OCaml native code binaries can be profiled just like any other C binary, by using the name mangling described earlier to map between OCaml variable names and the profiler output.    

+

Most profiling tools benefit from having some instrumentation included in the binary. OCaml supports two such tools:

    -

  • GNU gprof, to measure execution time and call -graphs

    • -

  • The Perf profiling -framework in modern versions of Linux

    • +

  • GNU gprof, to measure execution time and call graphs

    • +

  • The Perf profiling framework in modern versions of Linux

-

Note that many other tools that operate on native binaries, such as -Valgrind, will work just fine with OCaml as long as the program is -linked with the -g flag to embed debugging symbols.

+

Note that many other tools that operate on native binaries, such as Valgrind, will work just fine with OCaml as long as the program is linked with the -g flag to embed debugging symbols.

Gprof

-

gprof produces an execution profile of an OCaml program -by recording a call graph of which functions call one another, and -recording the time these calls take during the program execution. 

-

Getting precise information out of gprof requires -passing the -p flag to the native code compiler when -compiling and linking the binary. This generates extra code -that records profile information to a file called gmon.out -when the program is executed. This profile information can then be -examined using gprof.

+

gprof produces an execution profile of an OCaml program by recording a call graph of which functions call one another, and recording the time these calls take during the program execution. 

+

Getting precise information out of gprof requires passing the -p flag to the native code compiler when compiling and linking the binary. This generates extra code that records profile information to a file called gmon.out when the program is executed. This profile information can then be examined using gprof.

Perf

-

Perf is a more modern alternative to gprof that doesn’t -require you to instrument the binary. Instead, it uses hardware counters -and debug information within the binary to record information -accurately.

-

Run Perf on a compiled binary to record information first. We’ll use -our write barrier benchmark from earlier, which measures memory -allocation versus in-place modification:

+

Perf is a more modern alternative to gprof that doesn’t require you to instrument the binary. Instead, it uses hardware counters and debug information within the binary to record information accurately.

+

Run Perf on a compiled binary to record information first. We’ll use our write barrier benchmark from earlier, which measures memory allocation versus in-place modification:

$ perf record -g ./barrier_bench.native
 Estimated testing time 20s (change using -quota SECS).
@@ -940,54 +605,25 @@ 
Perf

 +  30.22%  barrier.native  barrier.native  [.] camlBarrier__test_mutable_69279
 +  20.22%  barrier.native  barrier.native  [.] caml_modify
-

This trace broadly reflects the results of the benchmark itself. The -mutable benchmark consists of the combination of the call to -test_mutable and the caml_modify write barrier -function in the runtime. This adds up to slightly over half the -execution time of the application.

-

Perf has a growing collection of other commands that let you archive -these runs and compare them against each other. You can read more on the -home page.  

-
-
-

Using the Frame Pointer to Get More Accurate Traces

-

Although Perf doesn’t require adding in explicit probes to the -binary, it does need to understand how to unwind function calls so that -the kernel can accurately record the function backtrace for every event. -Since Linux 3.9 the kernel has had support for using DWARF debug -information to parse the program stack, which is emitted when the --g flag is passed to the OCaml compiler. For even more -accurate stack parsing, we need the compiler to fall back to using the -same conventions as C for function calls. On 64-bit Intel systems, this -means that a special register known as the frame pointer is -used to record function call history. Using the frame pointer in this -fashion means a slowdown (typically around 3-5%) since it’s no longer -available for general-purpose use.

-

OCaml thus makes the frame pointer an optional feature that can be -used to improve the resolution of Perf traces. opam provides a compiler -switch that compiles OCaml with the frame pointer activated:

+

This trace broadly reflects the results of the benchmark itself. The mutable benchmark consists of the combination of the call to test_mutable and the caml_modify write barrier function in the runtime. This adds up to slightly over half the execution time of the application.

+

Perf has a growing collection of other commands that let you archive these runs and compare them against each other. You can read more on the home page.  

+
+

Using the Frame Pointer to Get More Accurate Traces

+

Although Perf doesn’t require adding in explicit probes to the binary, it does need to understand how to unwind function calls so that the kernel can accurately record the function backtrace for every event. Since Linux 3.9 the kernel has had support for using DWARF debug information to parse the program stack, which is emitted when the -g flag is passed to the OCaml compiler. For even more accurate stack parsing, we need the compiler to fall back to using the same conventions as C for function calls. On 64-bit Intel systems, this means that a special register known as the frame pointer is used to record function call history. Using the frame pointer in this fashion means a slowdown (typically around 3-5%) since it’s no longer available for general-purpose use.

+

OCaml thus makes the frame pointer an optional feature that can be used to improve the resolution of Perf traces. opam provides a compiler switch that compiles OCaml with the frame pointer activated:

opam switch create 4.13+fp ocaml-variants.4.13.1+options ocaml-option-fp
-

Using the frame pointer changes the OCaml calling convention, but -opam takes care of recompiling all your libraries with the new -interface.

+

Using the frame pointer changes the OCaml calling convention, but opam takes care of recompiling all your libraries with the new interface.

+

Embedding Native Code in C

-

The native code compiler normally links a complete executable, but -can also output a standalone native object file just as the bytecode -compiler can. This object file has no further dependencies on OCaml -except for the runtime library.    

-

The native code runtime is a different library from the bytecode one, -and is installed as libasmrun.a in the OCaml standard -library directory.

-

Try this custom linking by using the same source files from the -bytecode embedding example earlier in this chapter:

+

The native code compiler normally links a complete executable, but can also output a standalone native object file just as the bytecode compiler can. This object file has no further dependencies on OCaml except for the runtime library.    

+

The native code runtime is a different library from the bytecode one, and is installed as libasmrun.a in the OCaml standard library directory.

+

Try this custom linking by using the same source files from the bytecode embedding example earlier in this chapter:

$ ocamlopt -output-obj -o embed_native.o embed_me1.ml embed_me2.ml
 $ gcc -Wall -I `ocamlc -where` -o final.native embed_native.o main.c \
@@ -998,25 +634,12 @@ 
Embedding Native Code in C

 hello embedded world 2
 After calling OCaml
-

The embed_native.o is a standalone object file that has -no further references to OCaml code beyond the runtime library, just as -with the bytecode runtime. Do remember that the link order of the -libraries is significant in modern GNU toolchains (especially as used in -Ubuntu 11.10 and later) that resolve symbols from left to right in a -single pass. 

-
-

Activating the Debug Runtime

-

Despite your best efforts, it is easy to introduce a bug into some -components, such as C bindings, that causes heap invariants to be -violated. OCaml includes a libasmrund.a variant of the -runtime library which is compiled with extra debugging checks that -perform extra memory integrity checks during every garbage collection -cycle. Running these extra checks will abort the program nearer the -point of corruption and help isolate the bug in the C code.

-

To use the debug library, just link your program with the --runtime-variant d flag:

-
+

The embed_native.o is a standalone object file that has no further references to OCaml code beyond the runtime library, just as with the bytecode runtime. Do remember that the link order of the libraries is significant in modern GNU toolchains (especially as used in Ubuntu 11.10 and later) that resolve symbols from left to right in a single pass. 

+
+

Activating the Debug Runtime

+

Despite your best efforts, it is easy to introduce a bug into some components, such as C bindings, that causes heap invariants to be violated. OCaml includes a libasmrund.a variant of the runtime library which is compiled with extra debugging checks that perform extra memory integrity checks during every garbage collection cycle. Running these extra checks will abort the program nearer the point of corruption and help isolate the bug in the C code.

+

To use the debug library, just link your program with the -runtime-variant d flag:

+
ocamlopt -runtime-variant d -verbose -o hello.native hello.ml
 >+ as  -o 'hello.o' '/tmp/build_cd0b96_dune/camlasmd3c336.s'
@@ -1038,42 +661,25 @@ 
Activating the Debug Runtime

Summarizing the File Extensions

-

We’ve seen how the compiler uses intermediate files to store various -stages of the compilation toolchain. Here’s a cheat sheet of all them in -one place.   

+

We’ve seen how the compiler uses intermediate files to store various stages of the compilation toolchain. Here’s a cheat sheet of all them in one place.   

    -
  • .ml are source files for compilation unit module -implementations.
    • -
  • .mli are source files for compilation unit module -interfaces. If missing, generated from the .ml file.
    • -
  • .cmi are compiled module interface from a corresponding -.mli source file.
    • -
  • .cmo are compiled bytecode object file of the module -implementation.
    • -
  • .cma are a library of bytecode object files packed into -a single file.
    • -
  • .o are C source files that have been compiled into -native object files by the system cc.
    • -
  • .cmt are the typed abstract syntax tree for module -implementations.
    • -
  • .cmti are the typed abstract syntax tree for module -interfaces.
    • -
  • .annot are old-style annotation file for displaying -typed, superseded by cmt files.
    • +
  • .ml are source files for compilation unit module implementations.
    • +
  • .mli are source files for compilation unit module interfaces. If missing, generated from the .ml file.
    • +
  • .cmi are compiled module interface from a corresponding .mli source file.
    • +
  • .cmo are compiled bytecode object file of the module implementation.
    • +
  • .cma are a library of bytecode object files packed into a single file.
    • +
  • .o are C source files that have been compiled into native object files by the system cc.
    • +
  • .cmt are the typed abstract syntax tree for module implementations.
    • +
  • .cmti are the typed abstract syntax tree for module interfaces.
    • +
  • .annot are old-style annotation file for displaying typed, superseded by cmt files.

The native code compiler also generates some additional files.

    -
  • .o are compiled native object files of the module -implementation.
    • -
  • .cmx contains extra information for linking and -cross-module optimization of the object file.
    • -
  • .cmxa and .a are libraries of -cmx and o units, stored in the -cmxa and a files respectively. These files are -always needed together.
    • -
  • .S or .s are the assembly language output -if -S is specified.
    • +
  • .o are compiled native object files of the module implementation.
    • +
  • .cmx contains extra information for linking and cross-module optimization of the object file.
    • +
  • .cmxa and .a are libraries of cmx and o units, stored in the cmxa and a files respectively. These files are always needed together.
    • +
  • .S or .s are the assembly language output if -S is specified.
-
\ No newline at end of file +
\ No newline at end of file diff --git a/compiler-frontend.html b/compiler-frontend.html index fbf21ffca..49262071d 100644 --- a/compiler-frontend.html +++ b/compiler-frontend.html @@ -1,166 +1,92 @@ -The Compiler Frontend: Parsing and Type Checking - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+The Compiler Frontend: Parsing and Type Checking - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

The Compiler Frontend: Parsing and Type Checking

-

Compiling source code into executable programs involves a fairly -complex set of libraries, linkers, and assemblers. While Dune mostly -hides this complexity from you, it’s still useful to understand how -these pieces work so that you can debug performance problems, or come up -with solutions for unusual situations that aren’t well handled by -existing tools.

-

OCaml has a strong emphasis on static type safety and rejects source -code that doesn’t meet its requirements as early as possible. The -compiler does this by running the source code through a series of checks -and transformations. Each stage performs its job (e.g., type checking, -optimization, or code generation) and discards some information from the -previous stage. The final native code output is low-level assembly code -that doesn’t know anything about the OCaml modules or objects that the -compiler started with.

+

Compiling source code into executable programs involves a fairly complex set of libraries, linkers, and assemblers. While Dune mostly hides this complexity from you, it’s still useful to understand how these pieces work so that you can debug performance problems, or come up with solutions for unusual situations that aren’t well handled by existing tools.

+

OCaml has a strong emphasis on static type safety and rejects source code that doesn’t meet its requirements as early as possible. The compiler does this by running the source code through a series of checks and transformations. Each stage performs its job (e.g., type checking, optimization, or code generation) and discards some information from the previous stage. The final native code output is low-level assembly code that doesn’t know anything about the OCaml modules or objects that the compiler started with.

In this chapter, we’ll cover the following topics:

    -
  • An overview of the compiler codebase and the compilation pipeline, -and what each stage represents
    • +
  • An overview of the compiler codebase and the compilation pipeline, and what each stage represents
  • Parsing, which goes from raw text to the abstract syntax tree
  • PPX’s, which further transform the AST
  • Type-checking, including module resolution
-

The details of the remainder of the compilation process, which gets -all the way to executable code comes next, in Chapter 26, The Compiler Backend Byte Code And Native Code.

+

The details of the remainder of the compilation process, which gets all the way to executable code comes next, in Chapter 26, The Compiler Backend Byte Code And Native Code.

An Overview of the Toolchain

-

The OCaml tools accept textual source code as input, using the -filename extensions .ml and .mli for modules -and signatures, respectively. We explained the basics of the build -process in Chapter 4, Files Modules And Programs, so we’ll assume you’ve -built a few OCaml programs already by this point. 

-

Each source file represents a compilation unit that is built -separately. The compiler generates intermediate files with different -filename extensions to use as it advances through the compilation -stages. The linker takes a collection of compiled units and produces a -standalone executable or library archive that can be reused by other -applications. 

+

The OCaml tools accept textual source code as input, using the filename extensions .ml and .mli for modules and signatures, respectively. We explained the basics of the build process in Chapter 4, Files Modules And Programs, so we’ll assume you’ve built a few OCaml programs already by this point. 

+

Each source file represents a compilation unit that is built separately. The compiler generates intermediate files with different filename extensions to use as it advances through the compilation stages. The linker takes a collection of compiled units and produces a standalone executable or library archive that can be reused by other applications. 

The overall compilation pipeline looks like this:  



-

Notice that the pipeline branches toward the end. OCaml has multiple -compiler backends that reuse the early stages of compilation but produce -very different final outputs. The bytecode can be run by a -portable interpreter and can even be transformed into JavaScript (via js_of_ocaml) or C source code -(via OCamlCC). The -native code compiler generates specialized executable binaries -suitable for high-performance applications.  

+

Notice that the pipeline branches toward the end. OCaml has multiple compiler backends that reuse the early stages of compilation but produce very different final outputs. The bytecode can be run by a portable interpreter and can even be transformed into JavaScript (via js_of_ocaml) or C source code (via OCamlCC). The native code compiler generates specialized executable binaries suitable for high-performance applications.  

Obtaining the Compiler Source Code

-

Although it’s not necessary to understand the examples, you may find -it useful to have a copy of the OCaml source tree checked out while you -read through this chapter. The source code is available from multiple -places:

+

Although it’s not necessary to understand the examples, you may find it useful to have a copy of the OCaml source tree checked out while you read through this chapter. The source code is available from multiple places:

    -

  • Stable releases as zip and -tar archives from the OCaml download -site

    • -

  • A Git repository with all the history and development branches -included, browsable online at GitHub

    • +

  • Stable releases as zip and tar archives from the OCaml download site

    • +

  • A Git repository with all the history and development branches included, browsable online at GitHub

-

The source tree is split up into subdirectories. The core compiler -consists of:

+

The source tree is split up into subdirectories. The core compiler consists of:

asmcomp/
-
-Native-code compiler that converts OCaml into high performance native -code executables. +
Native-code compiler that converts OCaml into high performance native code executables.
bytecomp/
-
-Bytecode compiler that converts OCaml into an interpreted executable -format. +
Bytecode compiler that converts OCaml into an interpreted executable format.
driver/
-
-Command-line interfaces for the compiler tools. +
Command-line interfaces for the compiler tools.
file_formats/
-
-Serializer and deserializers for on-disk files used by the compiler -driver. +
Serializer and deserializers for on-disk files used by the compiler driver.
lambda/
-
-The lambda conversion pass. +
The lambda conversion pass.
middle_end/
-
-The clambda, closure and flambda passes. +
The clambda, closure and flambda passes.
parsing/
-
-The OCaml lexer, parser, and libraries for manipulating them. +
The OCaml lexer, parser, and libraries for manipulating them.
runtime/
-
-The runtime library with the garbage collector. +
The runtime library with the garbage collector.
typing/
-
-The static type checking implementation and type definitions. +
The static type checking implementation and type definitions.
-

A number of tools and scripts are also built alongside the core -compiler:

+

A number of tools and scripts are also built alongside the core compiler:

debugger/
-
-The interactive bytecode debugger. +
The interactive bytecode debugger.
toplevel/
-
-Interactive top-level console. +
Interactive top-level console.
stdlib/
-
-The compiler standard library, including the Pervasives -module. +
The compiler standard library, including the Pervasives module.
otherlibs/
-
-Optional libraries such as the Unix and graphics modules. +
Optional libraries such as the Unix and graphics modules.
tools/
-
-Command-line utilities such as ocamldep that are installed -with the compiler. +
Command-line utilities such as ocamldep that are installed with the compiler.
testsuite/
-
-Regression tests for the core compiler. +
Regression tests for the core compiler.
-

We’ll go through each of the compilation stages now and explain how -they will be useful to you during day-to-day OCaml development.

+

We’ll go through each of the compilation stages now and explain how they will be useful to you during day-to-day OCaml development.

Parsing Source Code

-

When a source file is passed to the OCaml compiler, its first task is -to parse the text into a more structured abstract syntax tree (AST). The -parsing logic is implemented in OCaml itself using the techniques -described earlier in Chapter 19, Parsing With Ocamllex And Menhir. The lexer and -parser rules can be found in the parsing directory in the -source distribution.    

+

When a source file is passed to the OCaml compiler, its first task is to parse the text into a more structured abstract syntax tree (AST). The parsing logic is implemented in OCaml itself using the techniques described earlier in Chapter 19, Parsing With Ocamllex And Menhir. The lexer and parser rules can be found in the parsing directory in the source distribution.    

Syntax Errors

-

The OCaml parser’s goal is to output a well-formed AST data structure -to the next phase of compilation, and so it fails on any source code -that doesn’t match basic syntactic requirements. The compiler emits a -syntax error in this situation, with a pointer to the filename -and line and character number that’s as close to the error as -possible.  

-

Here’s an example syntax error that we obtain by performing a module -assignment as a statement instead of as a let binding:

+

The OCaml parser’s goal is to output a well-formed AST data structure to the next phase of compilation, and so it fails on any source code that doesn’t match basic syntactic requirements. The compiler emits a syntax error in this situation, with a pointer to the filename and line and character number that’s as close to the error as possible.  

+

Here’s an example syntax error that we obtain by performing a module assignment as a statement instead of as a let binding:

let () =
   module MyString = String;
@@ -175,34 +101,19 @@ 
Syntax Errors

 >Error: Syntax error
 [2]
-

The correct version of this source code creates the -MyString module correctly via a local open, and compiles -successfully:

+

The correct version of this source code creates the MyString module correctly via a local open, and compiles successfully:

let () =
   let module MyString = String in
   ()
-

The syntax error points to the line and character number of the first -token that couldn’t be parsed. In the broken example, the -module keyword isn’t a valid token at that point in -parsing, so the error location information is correct.

+

The syntax error points to the line and character number of the first token that couldn’t be parsed. In the broken example, the module keyword isn’t a valid token at that point in parsing, so the error location information is correct.

Generating Documentation from Interfaces

-

Whitespace and source code comments are removed during parsing and -aren’t significant in determining the semantics of the program. However, -other tools in the OCaml distribution can interpret comments for their -own ends.    

-

OCaml uses specially formatted comments in the source code to -generate documentation bundles. These comments are combined with the -function definitions and signatures, and output as structured -documentation in a variety of formats. Tools such as odoc -and ocamldoc can generate HTML pages, LaTeX and PDF -documents, UNIX manual pages, and even module dependency graphs that can -be viewed using Graphviz.

-

Here’s a sample of some source code that’s been annotated with -docstring comments:

+

Whitespace and source code comments are removed during parsing and aren’t significant in determining the semantics of the program. However, other tools in the OCaml distribution can interpret comments for their own ends.    

+

OCaml uses specially formatted comments in the source code to generate documentation bundles. These comments are combined with the function definitions and signatures, and output as structured documentation in a variety of formats. Tools such as odoc and ocamldoc can generate HTML pages, LaTeX and PDF documents, UNIX manual pages, and even module dependency graphs that can be viewed using Graphviz.

+

Here’s a sample of some source code that’s been annotated with docstring comments:

(** The first special comment of the file is the comment associated
    with the whole module. *)
@@ -225,70 +136,32 @@ 
Generating Documentation from Interfaces

   | `New_york   -> Rain 20
   | `California -> Sun
-

The docstrings are distinguished by beginning with the double -asterisk. There are formatting conventions for the contents of the -comment to mark metadata. For instance, the @tag fields -mark specific properties such as the author of that section of code.

-

There are two main tools used to manipulate docstring comments: the -ocamldoc tool that is supplied with the compiler, and the -odoc tool that is developed outside the compiler but is -intended to be the long-term replacement. Try compiling the HTML -documentation and UNIX man pages by running ocamldoc over -the source file:

+

The docstrings are distinguished by beginning with the double asterisk. There are formatting conventions for the contents of the comment to mark metadata. For instance, the @tag fields mark specific properties such as the author of that section of code.

+

There are two main tools used to manipulate docstring comments: the ocamldoc tool that is supplied with the compiler, and the odoc tool that is developed outside the compiler but is intended to be the long-term replacement. Try compiling the HTML documentation and UNIX man pages by running ocamldoc over the source file:

$ mkdir -p html man/man3
 $ ocamldoc -html -d html doc.ml
 $ ocamldoc -man -d man/man3 doc.ml
 $ man -M man Doc
-

You should now have HTML files inside the html/ -directory and also be able to view the UNIX manual pages held in -man/man3. There are quite a few comment formats and options -to control the output for the various backends. Refer to the OCaml manual for the -complete list.    

-

You can also use odoc to generate complete snapshots of -your project via integration with dune, as described earlier in Chapter 21, OCaml Platform.

+

You should now have HTML files inside the html/ directory and also be able to view the UNIX manual pages held in man/man3. There are quite a few comment formats and options to control the output for the various backends. Refer to the OCaml manual for the complete list.    

+

You can also use odoc to generate complete snapshots of your project via integration with dune, as described earlier in Chapter 21, OCaml Platform.

Preprocessing with ppx

-

One powerful feature in OCaml is a facility to extend the standard -language via extension points. These represent placeholders in -the OCaml syntax tree and are ignored by the standard compiler tooling, -beyond being delimited and stored in the abstract syntax tree alongside -the normal parsed source code. They are intended to be expanded by -external tools that select extension nodes that can interpret them. The -external tools can choose to generate further OCaml code by transforming -the input syntax tree, thus forming the basis of an extensible -preprocessor for the language.

-

There are two primary forms of extension points in OCaml: -attributes and extension nodes. Let’s first run -through some examples of what they look like, and then see how to use -them in your own code.

+

One powerful feature in OCaml is a facility to extend the standard language via extension points. These represent placeholders in the OCaml syntax tree and are ignored by the standard compiler tooling, beyond being delimited and stored in the abstract syntax tree alongside the normal parsed source code. They are intended to be expanded by external tools that select extension nodes that can interpret them. The external tools can choose to generate further OCaml code by transforming the input syntax tree, thus forming the basis of an extensible preprocessor for the language.

+

There are two primary forms of extension points in OCaml: attributes and extension nodes. Let’s first run through some examples of what they look like, and then see how to use them in your own code.

Extension Attributes

-

Attributes supply additional information that is attached to a node -in the OCaml syntax tree, and subsequently interpreted and expanded by -external tools.

-

The basic form of an attribute is the [@ ... ] syntax. -The number of @ symbols defines which part of the syntax -tree the attribute is bound to:

+

Attributes supply additional information that is attached to a node in the OCaml syntax tree, and subsequently interpreted and expanded by external tools.

+

The basic form of an attribute is the [@ ... ] syntax. The number of @ symbols defines which part of the syntax tree the attribute is bound to:

    -
  • a single [@ binds using a postfix notation to algebraic -categories such as expressions or individual constructors in type -definitions.
    • -
  • a double [@@ binds to blocks of code, such as module -definitions, type declarations or class fields.
    • -
  • a triple [@@@ appears as a standalone entry in a module -implementation or signature, and are not tied to any specific source -code node.
    • +
  • a single [@ binds using a postfix notation to algebraic categories such as expressions or individual constructors in type definitions.
    • +
  • a double [@@ binds to blocks of code, such as module definitions, type declarations or class fields.
    • +
  • a triple [@@@ appears as a standalone entry in a module implementation or signature, and are not tied to any specific source code node.
-

The OCaml compiler has some useful builtin attributes that we can use -to illustrate their use without requiring any external tools. Let’s -first look at the use of the standalone attribute -@@@warning to toggle an OCaml compiler warning.

+

The OCaml compiler has some useful builtin attributes that we can use to illustrate their use without requiring any external tools. Let’s first look at the use of the standalone attribute @@@warning to toggle an OCaml compiler warning.

module Abc = struct
 
@@ -303,15 +176,8 @@ 
Extension Attributes

 >module Abc : sig val a : unit val b : unit end
-

The warning in our example is taken from the compiler manual page. -This warning emits a message if the expression in a sequence doesn’t -have type unit. The @@@warning nodes in the -module implementation cause the compiler to change its behavior within -the scope of that structure only.

-

An annotation can also be more narrowly attached to a block of code. -For example, a module implementation can be annotated with -@@deprecated to indicate that it should not be used in new -code:

+

The warning in our example is taken from the compiler manual page. This warning emits a message if the expression in a sequence doesn’t have type unit. The @@@warning nodes in the module implementation cause the compiler to change its behavior within the scope of that structure only.

+

An annotation can also be more narrowly attached to a block of code. For example, a module implementation can be annotated with @@deprecated to indicate that it should not be used in new code:

module Planets = struct
   let earth = true
@@ -325,11 +191,7 @@ 
Extension Attributes

 >module Planets2016 : sig val earth : bool val pluto : bool end
-

In this example, the @@deprecated annotation is only -attached to the Planets module, and the human-readable -argument string redirects developers to the newer code. Now if we try to -use the value that has been marked as deprecated, the compiler will -issue a warning.

+

In this example, the @@deprecated annotation is only attached to the Planets module, and the human-readable argument string redirects developers to the newer code. Now if we try to use the value that has been marked as deprecated, the compiler will issue a warning.

let is_pluto_a_planet = Planets.pluto;;
 >Line 1, characters 25-38:
@@ -340,11 +202,7 @@ 
Extension Attributes

 >val is_pluto_a_planet : bool = false
-

Finally, an attribute can also be attached to an individual -expression. In the next example, the -@warn_on_literal_pattern attribute indicates that the -argument to the type constructor should not be pattern matched upon with -a constant literal.

+

Finally, an attribute can also be attached to an individual expression. In the next example, the @warn_on_literal_pattern attribute indicates that the argument to the type constructor should not be pattern matched upon with a constant literal.

type program_result =
 | Error of string [@warn_on_literal_pattern]
@@ -364,117 +222,57 @@ 
Extension Attributes

Commonly Used Extension Attributes

-

We have already used extension points in Chapter 20, Data Serialization With S Expressions to generate -boilerplate code for handling s-expressions. These are introduced by a -third-party library using the (preprocess) directive in a -dune file, for example:

+

We have already used extension points in Chapter 20, Data Serialization With S Expressions to generate boilerplate code for handling s-expressions. These are introduced by a third-party library using the (preprocess) directive in a dune file, for example:

(library
  (name hello_world)
  (libraries core)
  (preprocess (pps ppx_jane))
-

This allows you to take advantage of a community of syntax -augmentation. There are also a number of builtin attributes in the core -OCaml compiler. Some are performance oriented and give directives to the -compiler, whereas others will activate usage warnings. The full list is -available in the attributes section -of the OCaml manual.

+

This allows you to take advantage of a community of syntax augmentation. There are also a number of builtin attributes in the core OCaml compiler. Some are performance oriented and give directives to the compiler, whereas others will activate usage warnings. The full list is available in the attributes section of the OCaml manual.

Extension Nodes

-

While extension points are useful for annotating existing source -code, we also need a mechanism to store generic placeholders within the -OCaml AST for code generation. OCaml provides this facility via the -extension node syntax.

-

The general syntax for an extension node is [%id expr], -where id is an identifier for a particular extension node -rewriter and expr is the payload for the rewriter to parse. -An infix form is also available when the payload is of the same kind of -syntax. For example let%foo bar = 1 is equivalent to -[%foo let bar = 1].

-

We’ve already seen extension nodes in use via the Core syntax -extensions earlier in the book, where they act as syntactic sugar for -error handling (let%bind), for command-line parsing -(let%map) or inline testing (let%expect_test). -Extension nodes are introduced via dune rules in the same fashion as -extension attributes, via the (preprocess) attribute.

+

While extension points are useful for annotating existing source code, we also need a mechanism to store generic placeholders within the OCaml AST for code generation. OCaml provides this facility via the extension node syntax.

+

The general syntax for an extension node is [%id expr], where id is an identifier for a particular extension node rewriter and expr is the payload for the rewriter to parse. An infix form is also available when the payload is of the same kind of syntax. For example let%foo bar = 1 is equivalent to [%foo let bar = 1].

+

We’ve already seen extension nodes in use via the Core syntax extensions earlier in the book, where they act as syntactic sugar for error handling (let%bind), for command-line parsing (let%map) or inline testing (let%expect_test). Extension nodes are introduced via dune rules in the same fashion as extension attributes, via the (preprocess) attribute.

Static Type Checking

-

After obtaining a valid abstract syntax tree, the compiler has to -verify that the code obeys the rules of the OCaml type system. Code that -is syntactically correct but misuses values is rejected with an -explanation of the problem.

-

Although type checking is done in a single pass in OCaml, it actually -consists of three distinct steps that happen simultaneously:      

+

After obtaining a valid abstract syntax tree, the compiler has to verify that the code obeys the rules of the OCaml type system. Code that is syntactically correct but misuses values is rejected with an explanation of the problem.

+

Although type checking is done in a single pass in OCaml, it actually consists of three distinct steps that happen simultaneously:      

automatic type inference
-
-An algorithm that calculates types for a module without requiring manual -type annotations +
An algorithm that calculates types for a module without requiring manual type annotations
module system
-
-Combines software components with explicit knowledge of their type -signatures +
Combines software components with explicit knowledge of their type signatures
explicit subtyping
-
-Checks for objects and polymorphic variants +
Checks for objects and polymorphic variants
-

Automatic type inference lets you write succinct code for a -particular task and have the compiler ensure that your use of variables -is locally consistent.

-

Type inference doesn’t scale to very large codebases that depend on -separate compilation of files. A small change in one module may ripple -through thousands of other files and libraries and require all of them -to be recompiled. The module system solves this by providing the -facility to combine and manipulate explicit type signatures for modules -within a large project, and also to reuse them via functors and -first-class modules.  

-

Subtyping in OCaml objects is always an explicit operation (via the -:> operator). This means that it doesn’t complicate the -core type inference engine and can be tested as a separate concern.

+

Automatic type inference lets you write succinct code for a particular task and have the compiler ensure that your use of variables is locally consistent.

+

Type inference doesn’t scale to very large codebases that depend on separate compilation of files. A small change in one module may ripple through thousands of other files and libraries and require all of them to be recompiled. The module system solves this by providing the facility to combine and manipulate explicit type signatures for modules within a large project, and also to reuse them via functors and first-class modules.  

+

Subtyping in OCaml objects is always an explicit operation (via the :> operator). This means that it doesn’t complicate the core type inference engine and can be tested as a separate concern.

Displaying Inferred Types from the Compiler

-

We’ve already seen how you can explore type inference directly from -the toplevel. It’s also possible to generate type signatures for an -entire file by asking the compiler to do the work for you. Create a file -with a single type definition and value:

+

We’ve already seen how you can explore type inference directly from the toplevel. It’s also possible to generate type signatures for an entire file by asking the compiler to do the work for you. Create a file with a single type definition and value:

type t = Foo | Bar
 let v = Foo
-

Now run the compiler with the -i flag to infer the type -signature for that file. This runs the type checker but doesn’t compile -the code any further after displaying the interface to the standard -output:

+

Now run the compiler with the -i flag to infer the type signature for that file. This runs the type checker but doesn’t compile the code any further after displaying the interface to the standard output:

ocamlc -i typedef.ml
 >type t = Foo | Bar
 >val v : t
-

The output is the default signature for the module that represents -the input file. It’s often useful to redirect this output to an -mli file to give you a starting signature to edit the -external interface without having to type it all in by hand.

-

The compiler stores a compiled version of the interface as a -cmi file. This interface is either obtained from compiling -an mli signature file for a module, or by the inferred type -if there is only an ml implementation present.

-

The compiler makes sure that your ml and -mli files have compatible signatures. The type checker -throws an immediate error if this isn’t the case. For example, if you -have this as your ml file:

+

The output is the default signature for the module that represents the input file. It’s often useful to redirect this output to an mli file to give you a starting signature to edit the external interface without having to type it all in by hand.

+

The compiler stores a compiled version of the interface as a cmi file. This interface is either obtained from compiling an mli signature file for a module, or by the inferred type if there is only an ml implementation present.

+

The compiler makes sure that your ml and mli files have compatible signatures. The type checker throws an immediate error if this isn’t the case. For example, if you have this as your ml file:

type t = Foo
@@ -499,84 +297,26 @@

Displaying Inferred Types from the Compiler

> Actual declaration [2]
-
-

Which Comes First: The ml or the mli?

-

There are two schools of thought on which order OCaml code should be -written in. It’s very easy to begin writing code by starting with an -ml file and using the type inference to guide you as you -build up your functions. The mli file can then be generated -as described, and the exported functions documented.     

-

If you’re writing code that spans multiple files, it’s sometimes -easier to start by writing all the mli signatures and -checking that they type-check against one another. Once the signatures -are in place, you can write the implementations with the confidence that -they’ll all glue together correctly, with no cyclic dependencies among -the modules.

-

As with any such stylistic debate, you should experiment with which -system works best for you. Everyone agrees on one thing though: no -matter in what order you write them, production code should always -explicitly define an mli file for every ml -file in the project. It’s also perfectly fine to have an -mli file without a corresponding ml file if -you’re only declaring signatures (such as module types).

-

Signature files provide a place to write succinct documentation and -to abstract internal details that shouldn’t be exported. Maintaining -separate signature files also speeds up incremental compilation in -larger code bases, since recompiling a mli signature is -much faster than a full compilation of the implementation to native -code.

-
+
+

Which Comes First: The ml or the mli?

+

There are two schools of thought on which order OCaml code should be written in. It’s very easy to begin writing code by starting with an ml file and using the type inference to guide you as you build up your functions. The mli file can then be generated as described, and the exported functions documented.     

+

If you’re writing code that spans multiple files, it’s sometimes easier to start by writing all the mli signatures and checking that they type-check against one another. Once the signatures are in place, you can write the implementations with the confidence that they’ll all glue together correctly, with no cyclic dependencies among the modules.

+

As with any such stylistic debate, you should experiment with which system works best for you. Everyone agrees on one thing though: no matter in what order you write them, production code should always explicitly define an mli file for every ml file in the project. It’s also perfectly fine to have an mli file without a corresponding ml file if you’re only declaring signatures (such as module types).

+

Signature files provide a place to write succinct documentation and to abstract internal details that shouldn’t be exported. Maintaining separate signature files also speeds up incremental compilation in larger code bases, since recompiling a mli signature is much faster than a full compilation of the implementation to native code.

+

Type Inference

-

Type inference is the process of determining the appropriate types -for expressions based on their use. It’s a feature that’s partially -present in many other languages such as Haskell and Scala, but OCaml -embeds it as a fundamental feature throughout the core language.   

-

OCaml type inference is based on the Hindley-Milner algorithm, which -is notable for its ability to infer the most general type for an -expression without requiring any explicit type annotations. The -algorithm can deduce multiple types for an expression and has the notion -of a principal type that is the most general choice from the -possible inferences. Manual type annotations can specialize the type -explicitly, but the automatic inference selects the most general type -unless told otherwise.

-

OCaml does have some language extensions that strain the limits of -principal type inference, but by and large, most programs you write will -never require annotations (although they sometimes help the -compiler produce better error messages).

+

Type inference is the process of determining the appropriate types for expressions based on their use. It’s a feature that’s partially present in many other languages such as Haskell and Scala, but OCaml embeds it as a fundamental feature throughout the core language.   

+

OCaml type inference is based on the Hindley-Milner algorithm, which is notable for its ability to infer the most general type for an expression without requiring any explicit type annotations. The algorithm can deduce multiple types for an expression and has the notion of a principal type that is the most general choice from the possible inferences. Manual type annotations can specialize the type explicitly, but the automatic inference selects the most general type unless told otherwise.

+

OCaml does have some language extensions that strain the limits of principal type inference, but by and large, most programs you write will never require annotations (although they sometimes help the compiler produce better error messages).

Adding Type Annotations to Find Errors

-

It’s often said that the hardest part of writing OCaml code is -getting past the type checker—but once the code does compile, it works -correctly the first time! This is an exaggeration of course, but it can -certainly feel true when moving from a dynamically typed language. The -OCaml static type system protects you from certain classes of bugs such -as memory errors and abstraction violations by rejecting your program at -compilation time rather than by generating an error at runtime. Learning -how to navigate the type checker’s compile-time feedback is key to -building robust libraries and applications that take full advantage of -these static checks.    

-

There are a couple of tricks to make it easier to quickly locate type -errors in your code. The first is to introduce manual type annotations -to narrow down the source of your error more accurately. These -annotations shouldn’t actually change your types and can be removed once -your code is correct. However, they act as anchors to locate errors -while you’re still writing your code.

-

Manual type annotations are particularly useful if you use lots of -polymorphic variants or objects. Type inference with row polymorphism -can generate some very large signatures, and errors tend to propagate -more widely than if you are using more explicitly typed variants or -classes.  

+

It’s often said that the hardest part of writing OCaml code is getting past the type checker—but once the code does compile, it works correctly the first time! This is an exaggeration of course, but it can certainly feel true when moving from a dynamically typed language. The OCaml static type system protects you from certain classes of bugs such as memory errors and abstraction violations by rejecting your program at compilation time rather than by generating an error at runtime. Learning how to navigate the type checker’s compile-time feedback is key to building robust libraries and applications that take full advantage of these static checks.    

+

There are a couple of tricks to make it easier to quickly locate type errors in your code. The first is to introduce manual type annotations to narrow down the source of your error more accurately. These annotations shouldn’t actually change your types and can be removed once your code is correct. However, they act as anchors to locate errors while you’re still writing your code.

+

Manual type annotations are particularly useful if you use lots of polymorphic variants or objects. Type inference with row polymorphism can generate some very large signatures, and errors tend to propagate more widely than if you are using more explicitly typed variants or classes.  

-

For instance, consider this broken example that expresses some simple -algebraic operations over integers:

+

For instance, consider this broken example that expresses some simple algebraic operations over integers:

let rec algebra =
   function
@@ -597,9 +337,7 @@ 
Adding Type Annotations to Find Errors

         ))
     ))
-

There’s a single character typo in the code so that it uses -Nu instead of Num. The resulting type error is -impressive:

+

There’s a single character typo in the code so that it uses Nu instead of Num. The resulting type error is impressive:

ocamlc -c broken_poly.ml
 >File "broken_poly.ml", lines 9-18, characters 10-6:
@@ -630,17 +368,9 @@ 
Adding Type Annotations to Find Errors

 >       The second variant type does not allow tag(s) `Nu
 [2]
-

The type error is perfectly accurate, but rather verbose and with a -line number that doesn’t point to the exact location of the incorrect -variant name. The best the compiler can do is to point you in the -general direction of the algebra function application.

-

This is because the type checker doesn’t have enough information to -match the inferred type of the algebra definition to its -application a few lines down. It calculates types for both expressions -separately, and when they don’t match up, outputs the difference as best -it can.

-

Let’s see what happens with an explicit type annotation to help the -compiler out:

+

The type error is perfectly accurate, but rather verbose and with a line number that doesn’t point to the exact location of the incorrect variant name. The best the compiler can do is to point you in the general direction of the algebra function application.

+

This is because the type checker doesn’t have enough information to match the inferred type of the algebra definition to its application a few lines down. It calculates types for both expressions separately, and when they don’t match up, outputs the difference as best it can.

+

Let’s see what happens with an explicit type annotation to help the compiler out:

type t = [
   | `Add of t * t
@@ -668,10 +398,7 @@ 
Adding Type Annotations to Find Errors

         ))
     ))
-

This code contains exactly the same error as before, but we’ve added -a closed type definition of the polymorphic variants, and a type -annotation to the algebra definition. The compiler error we -get is much more useful now:

+

This code contains exactly the same error as before, but we’ve added a closed type definition of the polymorphic variants, and a type annotation to the algebra definition. The compiler error we get is much more useful now:

ocamlc -i broken_poly_with_annot.ml
 >File "broken_poly_with_annot.ml", line 22, characters 14-21:
@@ -682,34 +409,20 @@ 
Adding Type Annotations to Find Errors

 >       The second variant type does not allow tag(s) `Nu
 [2]
-

This error points directly to the correct line number that contains -the typo. Once you fix the problem, you can remove the manual -annotations if you prefer more succinct code. You can also leave the -annotations there, of course, to help with future refactoring and -debugging.

+

This error points directly to the correct line number that contains the typo. Once you fix the problem, you can remove the manual annotations if you prefer more succinct code. You can also leave the annotations there, of course, to help with future refactoring and debugging.

Enforcing Principal Typing

-

The compiler also has a stricter principal type checking -mode that is activated via the -principal flag. This warns -about risky uses of type information to ensure that the type inference -has one principal result. A type is considered risky if the success or -failure of type inference depends on the order in which subexpressions -are typed.   

+

The compiler also has a stricter principal type checking mode that is activated via the -principal flag. This warns about risky uses of type information to ensure that the type inference has one principal result. A type is considered risky if the success or failure of type inference depends on the order in which subexpressions are typed.   

The principality check only affects a few language features:

  • Polymorphic methods for objects

    • -

  • Permuting the order of labeled arguments in a function from their -type definition

    • +

  • Permuting the order of labeled arguments in a function from their type definition

  • Discarding optional labeled arguments

    • -

  • Generalized algebraic data types (GADTs) present from OCaml 4.0 -onward

    • -

  • Automatic disambiguation of record field and constructor names -(since OCaml 4.1)

    • +

  • Generalized algebraic data types (GADTs) present from OCaml 4.0 onward

    • +

  • Automatic disambiguation of record field and constructor names (since OCaml 4.1)

-

Here’s an example of principality warnings when used with record -disambiguation.

+

Here’s an example of principality warnings when used with record disambiguation.

type s = { foo: int; bar: unit }
 type t = { foo: int }
@@ -718,8 +431,7 @@ 
Enforcing Principal Typing

   x.bar;
   x.foo
-

Inferring the signature with -principal will show you a -new warning:

+

Inferring the signature with -principal will show you a new warning:

ocamlc -i -principal non_principal.ml
 >File "non_principal.ml", line 6, characters 4-7:
@@ -731,14 +443,8 @@ 
Enforcing Principal Typing

 >val f : s -> int
-

This example isn’t principal, since the inferred type for -x.foo is guided by the inferred type of x.bar, -whereas principal typing requires that each subexpression’s type can be -calculated independently. If the x.bar use is removed from -the definition of f, its argument would be of type -t and not type s.

-

You can fix this either by permuting the order of the type -declarations, or by adding an explicit type annotation:

+

This example isn’t principal, since the inferred type for x.foo is guided by the inferred type of x.bar, whereas principal typing requires that each subexpression’s type can be calculated independently. If the x.bar use is removed from the definition of f, its argument would be of type t and not type s.

+

You can fix this either by permuting the order of the type declarations, or by adding an explicit type annotation:

type s = { foo: int; bar: unit }
 type t = { foo: int }
@@ -747,9 +453,7 @@ 
Enforcing Principal Typing

   x.bar;
   x.foo
-

There is now no ambiguity about the inferred types, since we’ve -explicitly given the argument a type, and the order of inference of the -subexpressions no longer matters.

+

There is now no ambiguity about the inferred types, since we’ve explicitly given the argument a type, and the order of inference of the subexpressions no longer matters.

ocamlc -i -principal principal.ml
 >type s = { foo : int; bar : unit; }
@@ -757,8 +461,7 @@ 
Enforcing Principal Typing

 >val f : s -> int
-

The dune equivalent is to add the flag --principal to your build description.

+

The dune equivalent is to add the flag -principal to your build description.

(executable
   (name principal)
@@ -770,9 +473,7 @@ 
Enforcing Principal Typing

   (flags :standard -principal)
   (modules non_principal))
-

The :standard directive will include all the default -flags, and then -principal will be appended after those in -the compiler build flags.

+

The :standard directive will include all the default flags, and then -principal will be appended after those in the compiler build flags.

dune build principal.exe
 dune build non_principal.exe
@@ -782,44 +483,18 @@ 
Enforcing Principal Typing

 >Error (warning 18 [not-principal]): this type-based field disambiguation is not principal.
 [1]
-

Ideally, all code should systematically use -principal. -It reduces variance in type inference and enforces the notion of a -single known type. However, there are drawbacks to this mode: type -inference is slower, and the cmi files become larger. This -is generally only a problem if you extensively use objects, which -usually have larger type signatures to cover all their methods.

-

If compiling in principal mode works, it is guaranteed that the -program will pass type checking in non-principal mode, too. Bear in mind -that the cmi files generated in principal mode differ from -the default mode. Try to ensure that you compile your whole project with -it activated. Getting the files mixed up won’t let you violate type -safety, but it can result in the type checker failing unexpectedly very -occasionally. In this case, just recompile with a clean source tree.

+

Ideally, all code should systematically use -principal. It reduces variance in type inference and enforces the notion of a single known type. However, there are drawbacks to this mode: type inference is slower, and the cmi files become larger. This is generally only a problem if you extensively use objects, which usually have larger type signatures to cover all their methods.

+

If compiling in principal mode works, it is guaranteed that the program will pass type checking in non-principal mode, too. Bear in mind that the cmi files generated in principal mode differ from the default mode. Try to ensure that you compile your whole project with it activated. Getting the files mixed up won’t let you violate type safety, but it can result in the type checker failing unexpectedly very occasionally. In this case, just recompile with a clean source tree.

Modules and Separate Compilation

-

The OCaml module system enables smaller components to be reused -effectively in large projects while still retaining all the benefits of -static type safety. We covered the basics of using modules earlier in Chapter 4, Files Modules And Programs. The module language -that operates over these signatures also extends to functors and -first-class modules, described in Chapter 10, Functors and Chapter 11, First Class Modules, respectively.  

-

This section discusses how the compiler implements them in more -detail. Modules are essential for larger projects that consist of many -source files (also known as compilation units). It’s -impractical to recompile every single source file when changing just one -or two files, and the module system minimizes such recompilation while -still encouraging code reuse.  

+

The OCaml module system enables smaller components to be reused effectively in large projects while still retaining all the benefits of static type safety. We covered the basics of using modules earlier in Chapter 4, Files Modules And Programs. The module language that operates over these signatures also extends to functors and first-class modules, described in Chapter 10, Functors and Chapter 11, First Class Modules, respectively.  

+

This section discusses how the compiler implements them in more detail. Modules are essential for larger projects that consist of many source files (also known as compilation units). It’s impractical to recompile every single source file when changing just one or two files, and the module system minimizes such recompilation while still encouraging code reuse.  

The Mapping Between Files and Modules

-

Individual compilation units provide a convenient way to break up a -big module hierarchy into a collection of files. The relationship -between files and modules can be explained directly in terms of the -module system.  

-

Create a file called alice.ml with the following -contents:

+

Individual compilation units provide a convenient way to break up a big module hierarchy into a collection of files. The relationship between files and modules can be explained directly in terms of the module system.  

+

Create a file called alice.ml with the following contents:

let friends = [ Bob.name ]
@@ -827,8 +502,7 @@

The Mapping Between Files and Modules

val friends : Bob.t list
-

These two files produce essentially the same result as the following -code.

+

These two files produce essentially the same result as the following code.

module Alice : sig
   val friends : Bob.t list
@@ -839,47 +513,14 @@ 
The Mapping Between Files and Modules

Defining a Module Search Path

-

In the preceding example, Alice also has a reference to -another module Bob. For the overall type of -Alice to be valid, the compiler also needs to check that -the Bob module contains at least a Bob.name -value and defines a Bob.t type.  

-

The type checker resolves such module references into concrete -structures and signatures in order to unify types across module -boundaries. It does this by searching a list of directories for a -compiled interface file matching that module’s name. For example, it -will look for alice.cmi and bob.cmi on the -search path and use the first ones it encounters as the interfaces for -Alice and Bob.

-

The module search path is set by adding -I flags to the -compiler command line with the directory containing the cmi -files as the argument. Manually specifying these flags gets complex when -you have lots of libraries, and is the reason why tools like -dune and ocamlfind exist. They both automate -the process of turning third-party package names and build descriptions -into command-line flags that are passed to the compiler command -line.

-

By default, only the current directory and the OCaml standard library -will be searched for cmi files. The Stdlib -module from the standard library will also be opened by default in every -compilation unit. The standard library location is obtained by running -ocamlc -where and can be overridden by setting the -CAMLLIB environment variable. Needless to say, don’t -override the default path unless you have a good reason to (such as -setting up a cross-compilation environment).    

-
-
-

Inspecting Compilation Units with ocamlobjinfo

-

For separate compilation to be sound, we need to ensure that all the -cmi files used to type-check a module are the same across -compilation runs. If they vary, this raises the possibility of two -modules checking different type signatures for a common module with the -same name. This in turn lets the program completely violate the static -type system and can lead to memory corruption and crashes.

-

OCaml guards against this by recording a MD5 checksum in every -cmi. Let’s examine our earlier typedef.ml more -closely:

+

In the preceding example, Alice also has a reference to another module Bob. For the overall type of Alice to be valid, the compiler also needs to check that the Bob module contains at least a Bob.name value and defines a Bob.t type.  

+

The type checker resolves such module references into concrete structures and signatures in order to unify types across module boundaries. It does this by searching a list of directories for a compiled interface file matching that module’s name. For example, it will look for alice.cmi and bob.cmi on the search path and use the first ones it encounters as the interfaces for Alice and Bob.

+

The module search path is set by adding -I flags to the compiler command line with the directory containing the cmi files as the argument. Manually specifying these flags gets complex when you have lots of libraries, and is the reason why tools like dune and ocamlfind exist. They both automate the process of turning third-party package names and build descriptions into command-line flags that are passed to the compiler command line.

+

By default, only the current directory and the OCaml standard library will be searched for cmi files. The Stdlib module from the standard library will also be opened by default in every compilation unit. The standard library location is obtained by running ocamlc -where and can be overridden by setting the CAMLLIB environment variable. Needless to say, don’t override the default path unless you have a good reason to (such as setting up a cross-compilation environment).    

+
+

Inspecting Compilation Units with ocamlobjinfo

+

For separate compilation to be sound, we need to ensure that all the cmi files used to type-check a module are the same across compilation runs. If they vary, this raises the possibility of two modules checking different type signatures for a common module with the same name. This in turn lets the program completely violate the static type system and can lead to memory corruption and crashes.

+

OCaml guards against this by recording a MD5 checksum in every cmi. Let’s examine our earlier typedef.ml more closely:

ocamlc -c typedef.ml
 ocamlobjinfo typedef.cmi
@@ -891,20 +532,8 @@ 
Inspecting Compilation Units with ocamlobjinfo

 >    79ae8c0eb753af6b441fe05456c7970b    CamlinternalFormatBasics
-

ocamlobjinfo examines the compiled interface and -displays what other compilation units it depends on. In this case, we -don’t use any external modules other than Pervasives. Every -module depends on Pervasives by default, unless you use the --nopervasives flag (this is an advanced use case, and you -shouldn’t normally need it).

-

The long alphanumeric identifier beside each module name is a hash -calculated from all the types and values exported from that compilation -unit. It’s used during type-checking and linking to ensure that all of -the compilation units have been compiled consistently against one -another. A difference in the hashes means that a compilation unit with -the same module name may have conflicting type signatures in different -modules. The compiler will reject such programs with an error similar to -this:

+

ocamlobjinfo examines the compiled interface and displays what other compilation units it depends on. In this case, we don’t use any external modules other than Pervasives. Every module depends on Pervasives by default, unless you use the -nopervasives flag (this is an advanced use case, and you shouldn’t normally need it).

+

The long alphanumeric identifier beside each module name is a hash calculated from all the types and values exported from that compilation unit. It’s used during type-checking and linking to ensure that all of the compilation units have been compiled consistently against one another. A difference in the hashes means that a compilation unit with the same module name may have conflicting type signatures in different modules. The compiler will reject such programs with an error similar to this:

$ ocamlc -c foo.ml
 File "foo.ml", line 1, characters 0-1:
@@ -912,34 +541,22 @@ 
Inspecting Compilation Units with ocamlobjinfo

        and /usr/lib/ocaml/map.cmi make inconsistent assumptions
        over interface Map
-

This hash check is very conservative, but ensures that separate -compilation remains type-safe all the way up to the final link phase. -Your build system should ensure that you never see the preceding error -messages, but if you do run into it, just clean out your intermediate -files and recompile from scratch.

+

This hash check is very conservative, but ensures that separate compilation remains type-safe all the way up to the final link phase. Your build system should ensure that you never see the preceding error messages, but if you do run into it, just clean out your intermediate files and recompile from scratch.

+

Wrapping Libraries with Module Aliases

-

The module-to-file mapping described so far rigidly enforces a 1:1 -mapping between a top-level module and a file. It’s often convenient to -split larger modules into separate files to make editing easier, but -still compile them all into a single OCaml module.  

-

Dune provides a very convenient way of doing this for libraries via -automatically generating a toplevel module alias file that -places all the files in a given library as submodules within the -toplevel module for that library. This is known as wrapping the -library, and works as follows.

-

Let’s define a simple library with two files a.ml and -b.ml that each define a single value.

+

The module-to-file mapping described so far rigidly enforces a 1:1 mapping between a top-level module and a file. It’s often convenient to split larger modules into separate files to make editing easier, but still compile them all into a single OCaml module.  

+

Dune provides a very convenient way of doing this for libraries via automatically generating a toplevel module alias file that places all the files in a given library as submodules within the toplevel module for that library. This is known as wrapping the library, and works as follows.

+

Let’s define a simple library with two files a.ml and b.ml that each define a single value.

let v = "hello"
let w = 42
-

The dune file defines a library called hello that -includes these two modules.

+

The dune file defines a library called hello that includes these two modules.

(library
   (name hello)
@@ -949,8 +566,7 @@ 
Wrapping Libraries with Module Aliases

   (libraries hello)
   (modules test))
-

If we now build this library, we can look at how dune assembles the -modules into a Hello library.

+

If we now build this library, we can look at how dune assembles the modules into a Hello library.

dune build
 cat _build/default/hello.ml-gen
@@ -962,23 +578,12 @@ 
Wrapping Libraries with Module Aliases

 >module B = Hello__B
-

Dune has generated a hello.ml file which forms the -toplevel module exposed by the library. It has also renamed the -individual modules into internal mangled names such as -Hello__A, and assigned those internal modules as aliases -within the generated hello.ml file. This then allows a user -of this library to access the values as Hello.A. For -example, our test executable contains this:

+

Dune has generated a hello.ml file which forms the toplevel module exposed by the library. It has also renamed the individual modules into internal mangled names such as Hello__A, and assigned those internal modules as aliases within the generated hello.ml file. This then allows a user of this library to access the values as Hello.A. For example, our test executable contains this:

let v = Hello.A.v
 let w = Hello.B.w
-

One nice aspect about this module alias scheme is that a single -toplevel module provides a central place to write documentation about -how to use all the submodules exposed by the library. We can manually -add a hello.ml and hello.mli to our library -that does exactly this. First add the hello module to the -dune file:

+

One nice aspect about this module alias scheme is that a single toplevel module provides a central place to write documentation about how to use all the submodules exposed by the library. We can manually add a hello.ml and hello.mli to our library that does exactly this. First add the hello module to the dune file:

(library
   (name hello)
@@ -988,14 +593,12 @@ 
Wrapping Libraries with Module Aliases

   (libraries hello)
   (modules test))
-

Then the hello.ml file contains the module aliases (and -any other code you might want to add to the toplevel module).

+

Then the hello.ml file contains the module aliases (and any other code you might want to add to the toplevel module).

module A = A
 module B = B
-

Finally, the hello.mli interface file can reference all -the submodules and include documentation strings:

+

Finally, the hello.mli interface file can reference all the submodules and include documentation strings:

(** Documentation for module A *)
 module A : sig
@@ -1009,31 +612,19 @@ 
Wrapping Libraries with Module Aliases

   val w : int
 end
-

If you want to disable this behavior of dune and deliberately include -multiple toplevel modules, you can add (wrapped false) to -your libraries stanza. However, this is discouraged in general due to -the increased likelihood of linking clashes when you have a lot of -library dependencies, since every module that is linked into an -executable must have a unique name in OCaml.

+

If you want to disable this behavior of dune and deliberately include multiple toplevel modules, you can add (wrapped false) to your libraries stanza. However, this is discouraged in general due to the increased likelihood of linking clashes when you have a lot of library dependencies, since every module that is linked into an executable must have a unique name in OCaml.

Shorter Module Paths in Type Errors

-

Core uses the OCaml module system quite extensively to provide a -complete replacement standard library. It collects these modules into a -single Std module, which provides a single module that -needs to be opened to import the replacement modules and functions. - 

-

There’s one downside to this approach: type errors suddenly get much -more verbose. We can see this if you run the vanilla OCaml toplevel (not -utop).

+

Core uses the OCaml module system quite extensively to provide a complete replacement standard library. It collects these modules into a single Std module, which provides a single module that needs to be opened to import the replacement modules and functions.  

+

There’s one downside to this approach: type errors suddenly get much more verbose. We can see this if you run the vanilla OCaml toplevel (not utop).

$ ocaml
 # List.map print_endline "";;
 Error: This expression has type string but an expression was expected of type
          string list
-

This type error without Core has a straightforward type -error. When we switch to Core, though, it gets more verbose:

+

This type error without Core has a straightforward type error. When we switch to Core, though, it gets more verbose:

$ ocaml
 # open Core;;
@@ -1041,15 +632,8 @@ 
Shorter Module Paths in Type Errors

 Error: This expression has type string but an expression was expected of type
          'a Core.List.t = 'a list
-

The default List module in OCaml is overridden by -Core.List. The compiler does its best to show the type -equivalence, but at the cost of a more verbose error message.

-

The compiler can remedy this via a so-called short paths heuristic. -This causes the compiler to search all the type aliases for the shortest -module path and use that as the preferred output type. The option is -activated by passing -short-paths to the compiler, and -works on the toplevel, too. 

+

The default List module in OCaml is overridden by Core.List. The compiler does its best to show the type equivalence, but at the cost of a more verbose error message.

+

The compiler can remedy this via a so-called short paths heuristic. This causes the compiler to search all the type aliases for the shortest module path and use that as the preferred output type. The option is activated by passing -short-paths to the compiler, and works on the toplevel, too. 

$ ocaml -short-paths
 # open Core;;
@@ -1057,48 +641,24 @@ 
Shorter Module Paths in Type Errors

 Error: This expression has type string but an expression was expected of type
          'a list
-

The utop enhanced toplevel activates short paths by -default, which is why we have not had to do this before in our -interactive examples. However, the compiler doesn’t default to the short -path heuristic, since there are some situations where the type aliasing -information is useful to know, and it would be lost in the error if the -shortest module path is always picked.

-

You’ll need to choose for yourself if you prefer short paths or the -default behavior in your own projects, and pass the --short-paths flag to the compiler if you need

+

The utop enhanced toplevel activates short paths by default, which is why we have not had to do this before in our interactive examples. However, the compiler doesn’t default to the short path heuristic, since there are some situations where the type aliasing information is useful to know, and it would be lost in the error if the shortest module path is always picked.

+

You’ll need to choose for yourself if you prefer short paths or the default behavior in your own projects, and pass the -short-paths flag to the compiler if you need

The Typed Syntax Tree

-

When the type checking process has successfully completed, it is -combined with the AST to form a typed abstract syntax tree. -This contains precise location information for every token in the input -file, and decorates each token with concrete type information.       

-

The compiler can output this as compiled cmt and -cmti files that contain the typed AST for the -implementation and signatures of a compilation unit. This is activated -by passing the -bin-annot flag to the compiler.

-

The cmt files are particularly useful for IDE tools to -match up OCaml source code at a specific location to the inferred or -external types. For example, the merlin and -ocaml-lsp-server opam packages both use this information to -provide you with tooltips and docstrings within your editor, as -described earlier in Chapter 21, OCaml Platform.

+

When the type checking process has successfully completed, it is combined with the AST to form a typed abstract syntax tree. This contains precise location information for every token in the input file, and decorates each token with concrete type information.       

+

The compiler can output this as compiled cmt and cmti files that contain the typed AST for the implementation and signatures of a compilation unit. This is activated by passing the -bin-annot flag to the compiler.

+

The cmt files are particularly useful for IDE tools to match up OCaml source code at a specific location to the inferred or external types. For example, the merlin and ocaml-lsp-server opam packages both use this information to provide you with tooltips and docstrings within your editor, as described earlier in Chapter 21, OCaml Platform.

Examining the Typed Syntax Tree Directly

-

The compiler has a couple of advanced flags that can dump the raw -output of the internal AST representation. You can’t depend on these -flags to give the same output across compiler revisions, but they are a -useful learning tool. 

+

The compiler has a couple of advanced flags that can dump the raw output of the internal AST representation. You can’t depend on these flags to give the same output across compiler revisions, but they are a useful learning tool. 

We’ll use our toy typedef.ml again:

type t = Foo | Bar
 let v = Foo
-

Let’s first look at the untyped syntax tree that’s generated from the -parsing phase:

+

Let’s first look at the untyped syntax tree that’s generated from the parsing phase:

ocamlc -dparsetree typedef.ml 2>&1
 >[
@@ -1139,16 +699,9 @@ 
Examining the Typed Syntax Tree Directly

 >]
-

This is rather a lot of output for a simple two-line program, but it -shows just how much structure the OCaml parser generates even from a -small source file.

-

Each portion of the AST is decorated with the precise location -information (including the filename and character location of the -token). This code hasn’t been type checked yet, so the raw tokens are -all included.

-

The typed AST that is normally output as a compiled cmt -file can be displayed in a more developer-readable form via the --dtypedtree option:

+

This is rather a lot of output for a simple two-line program, but it shows just how much structure the OCaml parser generates even from a small source file.

+

Each portion of the AST is decorated with the precise location information (including the filename and character location of the token). This code hasn’t been type checked yet, so the raw tokens are all included.

+

The typed AST that is normally output as a compiled cmt file can be displayed in a more developer-readable form via the -dtypedtree option:

ocamlc -dtypedtree typedef.ml 2>&1
 >[
@@ -1189,24 +742,10 @@ 
Examining the Typed Syntax Tree Directly

 >]
-

The typed AST is more explicit than the untyped syntax tree. For -instance, the type declaration has been given a unique name -(t/1008), as has the v value -(v/1011).

-

You’ll rarely need to look at this raw output from the compiler -unless you’re building IDE tools, or are hacking on extensions to the -core compiler itself. However, it’s useful to know that this -intermediate form exists before we delve further into the code -generation process next, in Chapter 26, The Compiler Backend Byte Code And Native Code.

-

There are several new integrated tools emerging that combine these -typed AST files with common editors such as Emacs or Vim. The best of -these is Merlin, which -adds value and module autocompletion, displays inferred types and can -build and display errors directly from within your editor. There are -instructions available on its homepage for configuring Merlin with your -favorite editor, or its bigger sibling ocaml-lsp-server is -described earlier in Chapter 21, OCaml Platform.

+

The typed AST is more explicit than the untyped syntax tree. For instance, the type declaration has been given a unique name (t/1008), as has the v value (v/1011).

+

You’ll rarely need to look at this raw output from the compiler unless you’re building IDE tools, or are hacking on extensions to the core compiler itself. However, it’s useful to know that this intermediate form exists before we delve further into the code generation process next, in Chapter 26, The Compiler Backend Byte Code And Native Code.

+

There are several new integrated tools emerging that combine these typed AST files with common editors such as Emacs or Vim. The best of these is Merlin, which adds value and module autocompletion, displays inferred types and can build and display errors directly from within your editor. There are instructions available on its homepage for configuring Merlin with your favorite editor, or its bigger sibling ocaml-lsp-server is described earlier in Chapter 21, OCaml Platform.

-

Next: Chapter 26The Compiler Backend: Bytecode and Native code

 \ No newline at end of file +

Next: Chapter 26The Compiler Backend: Bytecode and Native code

 \ No newline at end of file diff --git a/concurrent-programming.html b/concurrent-programming.html index 6357e33da..eec82c4c7 100644 --- a/concurrent-programming.html +++ b/concurrent-programming.html @@ -1,48 +1,14 @@ -Concurrent Programming with Async - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Concurrent Programming with Async - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Concurrent Programming with Async

-

The logic of building programs that interact with the outside world -is often dominated by waiting; waiting for the click of a mouse, or for -data to be fetched from disk, or for space to be available on an -outgoing network buffer. Even mildly sophisticated interactive -applications are typically concurrent, needing to wait for -multiple different events at the same time, responding immediately to -whatever happens first.    

-

One approach to concurrency is to use preemptive system threads, -which is the dominant approach in languages like Java or C#. In this -model, each task that may require simultaneous waiting is given an -operating system thread of its own so it can block without stopping the -entire program.  

-

Another approach is to have a single-threaded program, where that -single thread runs an event loop whose job is to react to -external events like timeouts or mouse clicks by invoking a callback -function that has been registered for that purpose. This approach shows -up in languages like JavaScript that have single-threaded runtimes, as -well as in many GUI toolkits.   

-

Each of these mechanisms has its own trade-offs. System threads -require significant memory and other resources per thread. Also, the -operating system can arbitrarily interleave the execution of system -threads, requiring the programmer to carefully protect shared resources -with locks and condition variables, which is exceedingly -error-prone.

-

Single-threaded event-driven systems, on the other hand, execute a -single task at a time and do not require the same kind of complex -synchronization that preemptive threads do. However, the inverted -control structure of an event-driven program often means that your own -control flow has to be threaded awkwardly through the system’s event -loop, leading to a maze of event callbacks.

-

This chapter covers the Async library, which offers a hybrid model -that aims to provide the best of both worlds, avoiding the performance -compromises and synchronization woes of preemptive threads without the -confusing inversion of control that usually comes with event-driven -systems.  

+

The logic of building programs that interact with the outside world is often dominated by waiting; waiting for the click of a mouse, or for data to be fetched from disk, or for space to be available on an outgoing network buffer. Even mildly sophisticated interactive applications are typically concurrent, needing to wait for multiple different events at the same time, responding immediately to whatever happens first.    

+

One approach to concurrency is to use preemptive system threads, which is the dominant approach in languages like Java or C#. In this model, each task that may require simultaneous waiting is given an operating system thread of its own so it can block without stopping the entire program.  

+

Another approach is to have a single-threaded program, where that single thread runs an event loop whose job is to react to external events like timeouts or mouse clicks by invoking a callback function that has been registered for that purpose. This approach shows up in languages like JavaScript that have single-threaded runtimes, as well as in many GUI toolkits.   

+

Each of these mechanisms has its own trade-offs. System threads require significant memory and other resources per thread. Also, the operating system can arbitrarily interleave the execution of system threads, requiring the programmer to carefully protect shared resources with locks and condition variables, which is exceedingly error-prone.

+

Single-threaded event-driven systems, on the other hand, execute a single task at a time and do not require the same kind of complex synchronization that preemptive threads do. However, the inverted control structure of an event-driven program often means that your own control flow has to be threaded awkwardly through the system’s event loop, leading to a maze of event callbacks.

+

This chapter covers the Async library, which offers a hybrid model that aims to provide the best of both worlds, avoiding the performance compromises and synchronization woes of preemptive threads without the confusing inversion of control that usually comes with event-driven systems.  

Async Basics

-

Recall how I/O is typically done in Core. Here’s a simple -example.

+

Recall how I/O is typically done in Core. Here’s a simple example.

open Core;;
 #show In_channel.read_all;;
@@ -53,17 +19,8 @@ 
Async Basics

 >- : string = "This is only a test."
-

From the type of In_channel.read_all, you can see that -it must be a blocking operation. In particular, the fact that it returns -a concrete string means it can’t return until the read has completed. -The blocking nature of the call means that no progress can be made on -anything else until the call is complete.  

-

In Async, well-behaved functions never block. Instead, they return a -value of type Deferred.t that acts as a placeholder that -will eventually be filled in with the result. As an example, consider -the signature of the Async equivalent of -In_channel.read_all.  

+

From the type of In_channel.read_all, you can see that it must be a blocking operation. In particular, the fact that it returns a concrete string means it can’t return until the read has completed. The blocking nature of the call means that no progress can be made on anything else until the call is complete.  

+

In Async, well-behaved functions never block. Instead, they return a value of type Deferred.t that acts as a placeholder that will eventually be filled in with the result. As an example, consider the signature of the Async equivalent of In_channel.read_all.  

#require "async";;
 open Async;;
@@ -71,14 +28,8 @@ 
Async Basics

 >val file_contents : string -> string Deferred.t
-

We first load the Async package in the toplevel using -#require, and then open the module. Async, like Core, is -designed to be an extension to your basic programming environment, and -is intended to be opened.

-

A deferred is essentially a handle to a value that may be computed in -the future. As such, if we call Reader.file_contents, the -resulting deferred will initially be empty, as you can see by calling -Deferred.peek.  

+

We first load the Async package in the toplevel using #require, and then open the module. Async, like Core, is designed to be an extension to your basic programming environment, and is intended to be opened.

+

A deferred is essentially a handle to a value that may be computed in the future. As such, if we call Reader.file_contents, the resulting deferred will initially be empty, as you can see by calling Deferred.peek.  

let contents = Reader.file_contents "test.txt";;
 >val contents : string Deferred.t = <abstr>
@@ -86,46 +37,27 @@ 
Async Basics

 >- : string option = None
-

The value in contents isn’t yet determined partly -because nothing running could do the necessary I/O. When using Async, -processing of I/O and other events is handled by the Async scheduler. -When writing a standalone program, you need to start the scheduler -explicitly, but utop knows about Async and can start the -scheduler automatically. More than that, utop knows about -deferred values, and when you type in an expression of type -Deferred.t, it will make sure the scheduler is running and -block until the deferred is determined. Thus, we can write:

+

The value in contents isn’t yet determined partly because nothing running could do the necessary I/O. When using Async, processing of I/O and other events is handled by the Async scheduler. When writing a standalone program, you need to start the scheduler explicitly, but utop knows about Async and can start the scheduler automatically. More than that, utop knows about deferred values, and when you type in an expression of type Deferred.t, it will make sure the scheduler is running and block until the deferred is determined. Thus, we can write:

contents;;
 >- : string = "This is only a test."
-

Slightly confusingly, the type shown here is not the type of -contents, which is string Deferred.t, but -rather string, the type of the value contained within that -deferred.

-

If we peek again, we’ll see that the value of contents -has been filled in.

+

Slightly confusingly, the type shown here is not the type of contents, which is string Deferred.t, but rather string, the type of the value contained within that deferred.

+

If we peek again, we’ll see that the value of contents has been filled in.

Deferred.peek contents;;
 >- : string option = Some "This is only a test."
-

In order to do real work with deferreds, we need a way of waiting for -a deferred computation to finish, which we do using -Deferred.bind. Here’s the type-signature of -bind.

+

In order to do real work with deferreds, we need a way of waiting for a deferred computation to finish, which we do using Deferred.bind. Here’s the type-signature of bind.

#show Deferred.bind;;
 >val bind : 'a Deferred.t -> f:('a -> 'b Deferred.t) -> 'b Deferred.t
-

bind is effectively a way of sequencing concurrent -computations. In particular, Deferred.bind d ~f causes -f to be called after the value of d has been -determined.  

-

Here’s a simple use of bind for a function that replaces -a file with an uppercase version of its contents.

+

bind is effectively a way of sequencing concurrent computations. In particular, Deferred.bind d ~f causes f to be called after the value of d has been determined.  

+

Here’s a simple use of bind for a function that replaces a file with an uppercase version of its contents.

let uppercase_file filename =
   Deferred.bind (Reader.file_contents filename)
@@ -138,14 +70,8 @@ 
Async Basics

 >- : string = "THIS IS ONLY A TEST."
-

Again, bind is acting as a sequencing operator, causing -the file to be saved via the call to Writer.save only after -the contents of the file were first read via -Reader.file_contents.

-

Writing out Deferred.bind explicitly can be rather -verbose, and so Async includes an infix operator for it: ->>=. Using this operator, we can rewrite -uppercase_file as follows:

+

Again, bind is acting as a sequencing operator, causing the file to be saved via the call to Writer.save only after the contents of the file were first read via Reader.file_contents.

+

Writing out Deferred.bind explicitly can be rather verbose, and so Async includes an infix operator for it: >>=. Using this operator, we can rewrite uppercase_file as follows:

let uppercase_file filename =
   Reader.file_contents filename
@@ -154,14 +80,8 @@ 
Async Basics

 >val uppercase_file : string -> unit Deferred.t = <fun>
-

Here, we’ve dropped the parentheses around the function on the -right-hand side of the bind, and we didn’t add a level of indentation -for the contents of that function. This is standard practice for using -the infix bind operator.  

-

Now let’s look at another potential use of bind. In this -case, we’ll write a function that counts the number of lines in a -file:

+

Here, we’ve dropped the parentheses around the function on the right-hand side of the bind, and we didn’t add a level of indentation for the contents of that function. This is standard practice for using the infix bind operator.  

+

Now let’s look at another potential use of bind. In this case, we’ll write a function that counts the number of lines in a file:

let count_lines filename =
   Reader.file_contents filename
@@ -172,13 +92,7 @@ 
Async Basics

 >         'a Deferred.t
-

This looks reasonable enough, but as you can see, the compiler is -unhappy. The issue here is that bind expects a function -that returns a Deferred.t, but we’ve provided it with a -function that returns the result directly. What we need is -return, a function provided by Async that takes an ordinary -value and wraps it up in a deferred.  

+

This looks reasonable enough, but as you can see, the compiler is unhappy. The issue here is that bind expects a function that returns a Deferred.t, but we’ve provided it with a function that returns the result directly. What we need is return, a function provided by Async that takes an ordinary value and wraps it up in a deferred.  

#show_val return;;
 >val return : 'a -> 'a Deferred.t
@@ -188,8 +102,7 @@ 
Async Basics

 >- : int = 3
-

Using return, we can make count_lines -compile:

+

Using return, we can make count_lines compile:

let count_lines filename =
   Reader.file_contents filename
@@ -198,21 +111,14 @@ 
Async Basics

 >val count_lines : string -> int Deferred.t = <fun>
-

Together, bind and return form a design -pattern in functional programming known as a monad. You’ll run -across this signature in many applications beyond just threads. Indeed, -we already ran across monads in Chapter 7, Bind And Other Error Handling Idioms.  

-

Calling bind and return together is a -fairly common pattern, and as such there is a standard shortcut for it -called Deferred.map, which has the following signature:

+

Together, bind and return form a design pattern in functional programming known as a monad. You’ll run across this signature in many applications beyond just threads. Indeed, we already ran across monads in Chapter 7, Bind And Other Error Handling Idioms.  

+

Calling bind and return together is a fairly common pattern, and as such there is a standard shortcut for it called Deferred.map, which has the following signature:

#show Deferred.map;;
 >val map : 'a Deferred.t -> f:('a -> 'b) -> 'b Deferred.t
-

and comes with its own infix equivalent, >>|. -Using it, we can rewrite count_lines again a bit more -succinctly:

+

and comes with its own infix equivalent, >>|. Using it, we can rewrite count_lines again a bit more succinctly:

let count_lines filename =
   Reader.file_contents filename
@@ -223,21 +129,15 @@ 
Async Basics

 >- : int = 10
-

Note that count_lines returns a deferred, but -utop waits for that deferred to become determined, and -shows us the contents of the deferred instead.

+

Note that count_lines returns a deferred, but utop waits for that deferred to become determined, and shows us the contents of the deferred instead.

Using Let Syntax

-

As was discussed in Chapter 7, Error Handling, there is a special syntax, which we -call let syntax, designed for working with monads, which we can -enable by enabling ppx_let.    

+

As was discussed in Chapter 7, Error Handling, there is a special syntax, which we call let syntax, designed for working with monads, which we can enable by enabling ppx_let.    

#require "ppx_let";;
-

Here’s what the bind-using version of -count_lines looks like using that syntax.  

+

Here’s what the bind-using version of count_lines looks like using that syntax.  

let count_lines filename =
   let%bind text = Reader.file_contents filename in
@@ -245,8 +145,7 @@ 
Using Let Syntax

 >val count_lines : string -> int Deferred.t = <fun>
-

And here’s the map-based version of -count_lines.  

+

And here’s the map-based version of count_lines.  

let count_lines filename =
   let%map text = Reader.file_contents filename in
@@ -254,34 +153,14 @@ 
Using Let Syntax

 >val count_lines : string -> int Deferred.t = <fun>
-

The difference here is just syntactic, with these examples compiling -down to the same thing as the corresponding examples written using infix -operators. What’s nice about let syntax is that it highlights the -analogy between monadic bind and OCaml’s built-in let-binding, thereby -making your code more uniform and more readable.

-

Let syntax works for any monad, and you decide which monad is in use -by opening the appropriate Let_syntax module. Opening -Async also implicitly opens -Deferred.Let_syntax, but in some contexts you may want to -do that explicitly.

-

For the most part, let syntax is easier to read and work with, and -you should default to it when using Async, which is what we’ll do for -the remainder of the chapter.

+

The difference here is just syntactic, with these examples compiling down to the same thing as the corresponding examples written using infix operators. What’s nice about let syntax is that it highlights the analogy between monadic bind and OCaml’s built-in let-binding, thereby making your code more uniform and more readable.

+

Let syntax works for any monad, and you decide which monad is in use by opening the appropriate Let_syntax module. Opening Async also implicitly opens Deferred.Let_syntax, but in some contexts you may want to do that explicitly.

+

For the most part, let syntax is easier to read and work with, and you should default to it when using Async, which is what we’ll do for the remainder of the chapter.

Ivars and Upon

-

Deferreds are usually built using combinations of bind, -map and return, but sometimes you want to -construct a deferred where you can programmatically decide when it gets -filled in. This can be done using an ivar. (The term ivar dates -back to a language called Concurrent ML that was developed by John Reppy -in the early ’90s. The “i” in ivar stands for incremental.)   

-

There are three fundamental operations for working with an ivar: you -can create one, using Ivar.create; you can read off the -deferred that corresponds to the ivar in question, using -Ivar.read; and you can fill an ivar, thus causing the -corresponding deferred to become determined, using -Ivar.fill. These operations are illustrated below:

+

Deferreds are usually built using combinations of bind, map and return, but sometimes you want to construct a deferred where you can programmatically decide when it gets filled in. This can be done using an ivar. (The term ivar dates back to a language called Concurrent ML that was developed by John Reppy in the early ’90s. The “i” in ivar stands for incremental.)   

+

There are three fundamental operations for working with an ivar: you can create one, using Ivar.create; you can read off the deferred that corresponds to the ivar in question, using Ivar.read; and you can fill an ivar, thus causing the corresponding deferred to become determined, using Ivar.fill. These operations are illustrated below:

let ivar = Ivar.create ();;
 >val ivar : '_weak1 Ivar.t =
@@ -296,15 +175,8 @@ 
Ivars and Upon

 >- : string option = Some "Hello"
-

Ivars are something of a low-level feature; operators like -map, bind and return are -typically easier to use and think about. But ivars can be useful when -you want to build a synchronization pattern that isn’t already well -supported.

-

As an example, imagine we wanted a way of scheduling a sequence of -actions that would run after a fixed delay. In addition, we’d like to -guarantee that these delayed actions are executed in the same order they -were scheduled in. Here’s a signature that captures this idea:

+

Ivars are something of a low-level feature; operators like map, bind and return are typically easier to use and think about. But ivars can be useful when you want to build a synchronization pattern that isn’t already well supported.

+

As an example, imagine we wanted a way of scheduling a sequence of actions that would run after a fixed delay. In addition, we’d like to guarantee that these delayed actions are executed in the same order they were scheduled in. Here’s a signature that captures this idea:

module type Delayer_intf = sig
   type t
@@ -319,28 +191,14 @@ 
Ivars and Upon

 >  end
-

An action is handed to schedule in the form of a -deferred-returning thunk (a thunk is a function whose argument is of -type unit). A deferred is handed back to the caller of -schedule that will eventually be filled with the contents -of the deferred value returned by the thunk. To implement this, we’ll -use an operator called upon, which has the following -signature:  

+

An action is handed to schedule in the form of a deferred-returning thunk (a thunk is a function whose argument is of type unit). A deferred is handed back to the caller of schedule that will eventually be filled with the contents of the deferred value returned by the thunk. To implement this, we’ll use an operator called upon, which has the following signature:  

#show upon;;
 >val upon : 'a Deferred.t -> ('a -> unit) -> unit
-

Like bind and return, upon -schedules a callback to be executed when the deferred it is passed is -determined; but unlike those calls, it doesn’t create a new deferred for -this callback to fill.

-

Our delayer implementation is organized around a queue of thunks, -where every call to schedule adds a thunk to the queue and -also schedules a job in the future to grab a thunk off the queue and run -it. The waiting will be done using the function after, -which takes a time span and returns a deferred which becomes determined -after that time span elapses:

+

Like bind and return, upon schedules a callback to be executed when the deferred it is passed is determined; but unlike those calls, it doesn’t create a new deferred for this callback to fill.

+

Our delayer implementation is organized around a queue of thunks, where every call to schedule adds a thunk to the queue and also schedules a job in the future to grab a thunk off the queue and run it. The waiting will be done using the function after, which takes a time span and returns a deferred which becomes determined after that time span elapses:

module Delayer : Delayer_intf = struct
   type t = { delay: Time.Span.t;
@@ -362,32 +220,18 @@ 
Ivars and Upon

 >module Delayer : Delayer_intf
-

This code isn’t particularly long, but it is subtle. In particular, -note how the queue of thunks is used to ensure that the enqueued actions -are run in the order they were scheduled, even if the thunks scheduled -by upon are run out of order. This kind of subtlety is -typical of code that involves ivars and upon, and because -of this, you should stick to the simpler map/bind/return style of -working with deferreds when you can.

-
-

Understanding bind in Terms of Ivars and -upon

-

Here’s roughly what happens when you write -let d' = Deferred.bind d ~f.

+

This code isn’t particularly long, but it is subtle. In particular, note how the queue of thunks is used to ensure that the enqueued actions are run in the order they were scheduled, even if the thunks scheduled by upon are run out of order. This kind of subtlety is typical of code that involves ivars and upon, and because of this, you should stick to the simpler map/bind/return style of working with deferreds when you can.

+
+

Understanding bind in Terms of Ivars and upon

+

Here’s roughly what happens when you write let d' = Deferred.bind d ~f.

    -

  • A new ivar i is created to hold the final result of -the computation. The corresponding deferred is returned

    • -

  • A function is registered to be called when the deferred -d becomes determined.

    • -

  • That function, once run, calls f with the value that -was determined for d.

    • -

  • Another function is registered to be called when the deferred -returned by f becomes determined.

    • -

  • When that function is called, it uses it to fill i, -causing the corresponding deferred it to become determined.

    • +

  • A new ivar i is created to hold the final result of the computation. The corresponding deferred is returned

    • +

  • A function is registered to be called when the deferred d becomes determined.

    • +

  • That function, once run, calls f with the value that was determined for d.

    • +

  • Another function is registered to be called when the deferred returned by f becomes determined.

    • +

  • When that function is called, it uses it to fill i, causing the corresponding deferred it to become determined.

-

That sounds like a lot, but we can implement this relatively -concisely.

+

That sounds like a lot, but we can implement this relatively concisely.

let my_bind d ~f =
   let i = Ivar.create () in
@@ -397,28 +241,14 @@ 
Understanding bind in Terms of Ivars and
 >  <fun>
-

Async’s real implementation has more optimizations and is therefore -more complicated. But the above implementation is still a useful -first-order mental model for how bind works under the covers. And it’s -another good example of how upon and ivars can be useful -for building concurrency primitives.

-
+

Async’s real implementation has more optimizations and is therefore more complicated. But the above implementation is still a useful first-order mental model for how bind works under the covers. And it’s another good example of how upon and ivars can be useful for building concurrency primitives.

+

Example: An Echo Server

-

Now that we have the basics of Async under our belt, let’s look at a -small standalone Async program. In particular, we’ll write an echo -server, i.e., a program that accepts connections from clients -and spits back whatever is sent to it.  

-

The first step is to create a function that can copy data from an -input to an output. Here, we’ll use Async’s Reader and -Writer modules, which provide a convenient abstraction for -working with input and output channels:    

+

Now that we have the basics of Async under our belt, let’s look at a small standalone Async program. In particular, we’ll write an echo server, i.e., a program that accepts connections from clients and spits back whatever is sent to it.  

+

The first step is to create a function that can copy data from an input to an output. Here, we’ll use Async’s Reader and Writer modules, which provide a convenient abstraction for working with input and output channels:    

open Core
 open Async
@@ -433,55 +263,21 @@ 
Example: An Echo Server

     let%bind () = Writer.flushed w in
     copy_blocks buffer r w
-

Bind is used in the code to sequence the operations, with a -bind marking each place we wait.

+

Bind is used in the code to sequence the operations, with a bind marking each place we wait.

    -
  • First, we call Reader.read to get a block of -input.
    • -
  • When that’s complete and if a new block was returned, we write that -block to the writer.
    • -
  • Finally, we wait until the writer’s buffers are flushed, at which -point we recurse.
    • +
  • First, we call Reader.read to get a block of input.
    • +
  • When that’s complete and if a new block was returned, we write that block to the writer.
    • +
  • Finally, we wait until the writer’s buffers are flushed, at which point we recurse.
-

If we hit an end-of-file condition, the loop is ended. The deferred -returned by a call to copy_blocks becomes determined only -once the end-of-file condition is hit.  

-

One important aspect of how copy_blocks is written is -that it provides pushback, which is to say that if the process -can’t make progress writing, it will stop reading. If you don’t -implement pushback in your servers, then anything that prevents you from -writing (e.g., a client that is unable to keep up) will cause your -program to allocate unbounded amounts of memory, as it keeps track of -all the data it intends to write but hasn’t been able to yet.

-
-

Tail-Calls and Chains of Deferreds

-

There’s another memory problem you might be concerned about, which is -the allocation of deferreds. If you think about the execution of -copy_blocks, you’ll see it’s creating a chain of deferreds, -two per time through the loop. The length of this chain is unbounded, -and so, naively, you’d think this would take up an unbounded amount of -memory as the echo process continues.

-

Happily, this is a case that Async knows how to optimize. In -particular, the whole chain of deferreds should become determined -precisely when the final deferred in the chain is determined, in this -case, when the Eof condition is hit. Because of this, we -could safely replace all of these deferreds with a single deferred. -Async does just this, and so there’s no memory leak after all.

-

This is essentially a form of tail-call optimization, lifted to the -Deferred monad. Indeed, you can tell that the bind in question doesn’t -lead to a memory leak in more or less the same way you can tell that the -tail recursion optimization should apply, which is that the bind that -creates the deferred is in tail-position. In other words, nothing is -done to that deferred once it’s created; it’s simply returned as is. - 

-
-

copy_blocks provides the logic for handling a client -connection, but we still need to set up a server to receive such -connections and dispatch to copy_blocks. For this, we’ll -use Async’s Tcp module, which has a collection of utilities -for creating TCP clients and servers:  

+

If we hit an end-of-file condition, the loop is ended. The deferred returned by a call to copy_blocks becomes determined only once the end-of-file condition is hit.  

+

One important aspect of how copy_blocks is written is that it provides pushback, which is to say that if the process can’t make progress writing, it will stop reading. If you don’t implement pushback in your servers, then anything that prevents you from writing (e.g., a client that is unable to keep up) will cause your program to allocate unbounded amounts of memory, as it keeps track of all the data it intends to write but hasn’t been able to yet.

+
+

Tail-Calls and Chains of Deferreds

+

There’s another memory problem you might be concerned about, which is the allocation of deferreds. If you think about the execution of copy_blocks, you’ll see it’s creating a chain of deferreds, two per time through the loop. The length of this chain is unbounded, and so, naively, you’d think this would take up an unbounded amount of memory as the echo process continues.

+

Happily, this is a case that Async knows how to optimize. In particular, the whole chain of deferreds should become determined precisely when the final deferred in the chain is determined, in this case, when the Eof condition is hit. Because of this, we could safely replace all of these deferreds with a single deferred. Async does just this, and so there’s no memory leak after all.

+

This is essentially a form of tail-call optimization, lifted to the Deferred monad. Indeed, you can tell that the bind in question doesn’t lead to a memory leak in more or less the same way you can tell that the tail recursion optimization should apply, which is that the bind that creates the deferred is in tail-position. In other words, nothing is done to that deferred once it’s created; it’s simply returned as is.  

+
+

copy_blocks provides the logic for handling a client connection, but we still need to set up a server to receive such connections and dispatch to copy_blocks. For this, we’ll use Async’s Tcp module, which has a collection of utilities for creating TCP clients and servers:  

(** Starts a TCP server, which listens on the specified port, invoking
     copy_blocks every time a client connects. *)
@@ -498,38 +294,18 @@ 
Tail-Calls and Chains of Deferreds

     (host_and_port
       : (Socket.Address.Inet.t, int) Tcp.Server.t Deferred.t)
-

The result of calling Tcp.Server.create is a -Tcp.Server.t, which is a handle to the server that lets you -shut the server down. We don’t use that functionality here, so we -explicitly ignore server to suppress the unused-variables -error. We put in a type annotation around the ignored value to make the -nature of the value we’re ignoring explicit.

-

The most important argument to Tcp.Server.create is the -final one, which is the client connection handler. Notably, the -preceding code does nothing explicit to close down the client -connections when the communication is done. That’s because the server -will automatically shut down the connection once the deferred returned -by the handler becomes determined.

-

Finally, we need to initiate the server and start the Async -scheduler:

+

The result of calling Tcp.Server.create is a Tcp.Server.t, which is a handle to the server that lets you shut the server down. We don’t use that functionality here, so we explicitly ignore server to suppress the unused-variables error. We put in a type annotation around the ignored value to make the nature of the value we’re ignoring explicit.

+

The most important argument to Tcp.Server.create is the final one, which is the client connection handler. Notably, the preceding code does nothing explicit to close down the client connections when the communication is done. That’s because the server will automatically shut down the connection once the deferred returned by the handler becomes determined.

+

Finally, we need to initiate the server and start the Async scheduler:

(* Call [run], and then start the scheduler *)
 let () =
   run ();
   never_returns (Scheduler.go ())
-

One of the most common newbie errors with Async is to forget to run -the scheduler. It can be a bewildering mistake, because without the -scheduler, your program won’t do anything at all; even calls to -printf won’t reach the terminal.

-

It’s worth noting that even though we didn’t spend much explicit -effort on thinking about multiple clients, this server is able to handle -many clients concurrently connecting and reading and writing data.

-

Now that we have the echo server, we can connect to the echo server -using the netcat tool, which is invoked as nc. Note that we -use dune exec to both build and run the executable. We use -the double-dashes so that Dune’s parsing of arguments doesn’t interfere -with argument parsing for the executed program.

+

One of the most common newbie errors with Async is to forget to run the scheduler. It can be a bewildering mistake, because without the scheduler, your program won’t do anything at all; even calls to printf won’t reach the terminal.

+

It’s worth noting that even though we didn’t spend much explicit effort on thinking about multiple clients, this server is able to handle many clients concurrently connecting and reading and writing data.

+

Now that we have the echo server, we can connect to the echo server using the netcat tool, which is invoked as nc. Note that we use dune exec to both build and run the executable. We use the double-dashes so that Dune’s parsing of arguments doesn’t interfere with argument parsing for the executed program.

dune exec -- ./echo.exe &
 echo "This is an echo server" | nc 127.0.0.1 8765
@@ -539,14 +315,10 @@ 
Tail-Calls and Chains of Deferreds

 killall echo.exe
-
-

Functions that Never Return

-

The call to never_returns around the call to -Scheduler.go is a little bit surprising, but it has a -purpose: to make it clear to whoever invokes Scheduler.go -that the function never returns.    

-

By default, a function that doesn’t return will have an inferred -return type of 'a:

+
+

Functions that Never Return

+

The call to never_returns around the call to Scheduler.go is a little bit surprising, but it has a purpose: to make it clear to whoever invokes Scheduler.go that the function never returns.    

+

By default, a function that doesn’t return will have an inferred return type of 'a:

let rec loop_forever () = loop_forever ();;
 >val loop_forever : unit -> 'a = <fun>
@@ -554,48 +326,28 @@ 
Functions that Never Return

 >val always_fail : unit -> 'a = <fun>
-

This is a little odd, but it does make sense. After all, if a -function never returns, we’re free to impute any type at all to its -non-existent return value. As a result, from a typing perspective, a -function that never returns can fit into any context within your -program.

-

But that itself can be problematic, especially with a function like -Scheduler.go, where the fact that it never returns is -perhaps not entirely obvious. The point of never_returns is -to create an explicit marker so the user knows that the function in -question doesn’t return.

-

To do this, Scheduler.go is defined to have a return -value of Nothing.t.  

+

This is a little odd, but it does make sense. After all, if a function never returns, we’re free to impute any type at all to its non-existent return value. As a result, from a typing perspective, a function that never returns can fit into any context within your program.

+

But that itself can be problematic, especially with a function like Scheduler.go, where the fact that it never returns is perhaps not entirely obvious. The point of never_returns is to create an explicit marker so the user knows that the function in question doesn’t return.

+

To do this, Scheduler.go is defined to have a return value of Nothing.t.  

#show Scheduler.go;;
 >val go : ?raise_unhandled_exn:bool -> unit -> never_returns
-

never_returns is just an alias of -Nothing.t.

-

Nothing.t is uninhabited, which means there are -no values of that type. As such, a function can’t actually return a -value of type Nothing.t, so only a function that never -returns can have Nothing.t as its return type! And we can -cause a function that never returns to have a return value of -Nothing.t by just adding a type annotation.   

+

never_returns is just an alias of Nothing.t.

+

Nothing.t is uninhabited, which means there are no values of that type. As such, a function can’t actually return a value of type Nothing.t, so only a function that never returns can have Nothing.t as its return type! And we can cause a function that never returns to have a return value of Nothing.t by just adding a type annotation.   

let rec loop_forever () : Nothing.t = loop_forever ();;
 >val loop_forever : unit -> never_returns = <fun>
-

The function never_returns consumes a value of type -Nothing.t and returns an unconstrained type -'a.

+

The function never_returns consumes a value of type Nothing.t and returns an unconstrained type 'a.

#show_val never_returns;;
 >val never_returns : never_returns -> 'a
-

If you try to write a function that uses Scheduler.go, -and just assumes that it returns unit, you’ll get a helpful -type error.

+

If you try to write a function that uses Scheduler.go, and just assumes that it returns unit, you’ll get a helpful type error.

let do_stuff n =
   let x = 3 in
@@ -607,9 +359,7 @@ 
Functions that Never Return

 >       because it is in the result of a conditional with no else branch
-

We can fix this by inserting a call to never_returns, -thus making the fact that Scheduler.go doesn’t return -apparent to the reader.

+

We can fix this by inserting a call to never_returns, thus making the fact that Scheduler.go doesn’t return apparent to the reader.

let do_stuff n =
   let x = 3 in
@@ -618,19 +368,14 @@ 
Functions that Never Return

 >val do_stuff : int -> int = <fun>
-
+

Improving the Echo Server

-

Let’s try to go a little bit farther with our echo server by walking -through a few improvements. In particular, we will:

+

Let’s try to go a little bit farther with our echo server by walking through a few improvements. In particular, we will:

    -

  • Add a proper command-line interface with -Command

    • -

  • Add a flag to specify the port to listen on and a flag to make -the server echo back the capitalized version of whatever was sent to -it

    • -

  • Simplify the code using Async’s Pipe -interface

    • +

  • Add a proper command-line interface with Command

    • +

  • Add a flag to specify the port to listen on and a flag to make the server echo back the capitalized version of whatever was sent to it

    • +

  • Simplify the code using Async’s Pipe interface

The following code does all of this:

@@ -670,17 +415,8 @@

Improving the Echo Server

fun () -> run ~uppercase ~port) |> Command_unix.run
-

Note the use of Deferred.never in the run -function. As you might guess from the name, Deferred.never -returns a deferred that is never determined. In this case, that -indicates that the echo server doesn’t ever shut down.  

-

The biggest change in the preceding code is the use of Async’s -Pipe. A Pipe is an asynchronous communication -channel that’s used for connecting different parts of your program. You -can think of it as a consumer/producer queue that uses deferreds for -communicating when the pipe is ready to be read from or written to. Our -use of pipes is fairly minimal here, but they are an important part of -Async, so it’s worth discussing them in some detail.  

+

Note the use of Deferred.never in the run function. As you might guess from the name, Deferred.never returns a deferred that is never determined. In this case, that indicates that the echo server doesn’t ever shut down.  

+

The biggest change in the preceding code is the use of Async’s Pipe. A Pipe is an asynchronous communication channel that’s used for connecting different parts of your program. You can think of it as a consumer/producer queue that uses deferreds for communicating when the pipe is ready to be read from or written to. Our use of pipes is fairly minimal here, but they are an important part of Async, so it’s worth discussing them in some detail.  

Pipes are created in connected read/write pairs:

let (r,w) = Pipe.create ();;
@@ -688,23 +424,14 @@ 
Improving the Echo Server

 >val w : '_weak3 Pipe.Writer.t = <abstr>
-

r and w are really just read and write -handles to the same underlying object. Note that r and -w have weakly polymorphic types, as discussed in Chapter 8, Imperative Programming, and so can only contain -values of a single, yet-to-be-determined type.

-

If we just try and write to the writer, we’ll see that we block -indefinitely in utop. You can break out of the wait by -hitting Control-C:

+

r and w are really just read and write handles to the same underlying object. Note that r and w have weakly polymorphic types, as discussed in Chapter 8, Imperative Programming, and so can only contain values of a single, yet-to-be-determined type.

+

If we just try and write to the writer, we’ll see that we block indefinitely in utop. You can break out of the wait by hitting Control-C:

Pipe.write w "Hello World!";;
 >Interrupted.
-

That’s because a pipe has a certain amount of internal slack, a -number of slots in the pipe to which something can be written before the -write will block. By default, a pipe has zero slack, which means that -the deferred returned by a write is determined only when the value is -read out of the pipe.

+

That’s because a pipe has a certain amount of internal slack, a number of slots in the pipe to which something can be written before the write will block. By default, a pipe has zero slack, which means that the deferred returned by a write is determined only when the value is read out of the pipe.

let (r,w) = Pipe.create ();;
 >val r : '_weak4 Pipe.Reader.t = <abstr>
@@ -717,93 +444,48 @@ 
Improving the Echo Server

 >- : unit = ()
-

In the function run, we’re taking advantage of one of -the many utility functions provided for pipes in the Pipe -module. In particular, we’re using Pipe.transfer to set up -a process that takes data from a reader-pipe and moves it to a -writer-pipe. Here’s the type of Pipe.transfer:

+

In the function run, we’re taking advantage of one of the many utility functions provided for pipes in the Pipe module. In particular, we’re using Pipe.transfer to set up a process that takes data from a reader-pipe and moves it to a writer-pipe. Here’s the type of Pipe.transfer:

Pipe.transfer;;
 >- : 'a Pipe.Reader.t -> 'b Pipe.Writer.t -> f:('a -> 'b) -> unit Deferred.t =
 ><fun>
-

The two pipes being connected are generated by the -Reader.pipe and Writer.pipe call respectively. -Note that pushback is preserved throughout the process, so that if the -writer gets blocked, the writer’s pipe will stop pulling data from the -reader’s pipe, which will prevent the reader from reading in more -data.

-

Importantly, the deferred returned by Pipe.transfer -becomes determined once the reader has been closed and the last element -is transferred from the reader to the writer. Once that deferred becomes -determined, the server will shut down that client connection. So, when a -client disconnects, the rest of the shutdown happens transparently.

-

The command-line parsing for this program is based on the Command -library that we introduced in Chapter 15, Command Line Parsing. Opening Async, -shadows the Command module with an extended version that -contains the async call:

+

The two pipes being connected are generated by the Reader.pipe and Writer.pipe call respectively. Note that pushback is preserved throughout the process, so that if the writer gets blocked, the writer’s pipe will stop pulling data from the reader’s pipe, which will prevent the reader from reading in more data.

+

Importantly, the deferred returned by Pipe.transfer becomes determined once the reader has been closed and the last element is transferred from the reader to the writer. Once that deferred becomes determined, the server will shut down that client connection. So, when a client disconnects, the rest of the shutdown happens transparently.

+

The command-line parsing for this program is based on the Command library that we introduced in Chapter 15, Command Line Parsing. Opening Async, shadows the Command module with an extended version that contains the async call:

#show Command.async_spec;;
 >val async_spec :
 >  ('a, unit Deferred.t) Async.Command.basic_spec_command Command.with_options
-

This differs from the ordinary Command.basic call in -that the main function must return a Deferred.t, and that -the running of the command (using Command_unix.run) -automatically starts the Async scheduler, without requiring an explicit -call to Scheduler.go. 

+

This differs from the ordinary Command.basic call in that the main function must return a Deferred.t, and that the running of the command (using Command_unix.run) automatically starts the Async scheduler, without requiring an explicit call to Scheduler.go. 

Example: Searching Definitions with DuckDuckGo

-

DuckDuckGo is a search engine with a freely available search -interface. In this section, we’ll use Async to write a small -command-line utility for querying DuckDuckGo to extract definitions for -a collection of terms.      

-

Our code is going to rely on a number of other libraries, all of -which can be installed using opam. Refer to the installation -instructions if you need help on the installation. Here’s the list -of libraries we’ll need: 

+

DuckDuckGo is a search engine with a freely available search interface. In this section, we’ll use Async to write a small command-line utility for querying DuckDuckGo to extract definitions for a collection of terms.      

+

Our code is going to rely on a number of other libraries, all of which can be installed using opam. Refer to the installation instructions if you need help on the installation. Here’s the list of libraries we’ll need: 

textwrap
-
-A library for wrapping long lines. We’ll use this for printing out our -results. +
A library for wrapping long lines. We’ll use this for printing out our results.
uri
-
-A library for handling URIs, or “Uniform Resource Identifiers,” of which -HTTP URLs are an example. +
A library for handling URIs, or “Uniform Resource Identifiers,” of which HTTP URLs are an example.
yojson
-
-A JSON parsing library that was described in Chapter 18, Handling Json -Data. +
A JSON parsing library that was described in Chapter 18, Handling Json Data.
cohttp
-
-A library for creating HTTP clients and servers. We need Async support, -which comes with the cohttp-async package. +
A library for creating HTTP clients and servers. We need Async support, which comes with the cohttp-async package.

Now let’s dive into the implementation.

URI Handling

-

HTTP URLs, which identify endpoints across the Web, are actually part -of a more general family known as Uniform Resource Identifiers (URIs). -The full URI specification is defined in RFC3986 and is rather -complicated. Luckily, the uri library provides a strongly -typed interface that takes care of much of the hassle.    

-

We’ll need a function for generating the URIs that we’re going to use -to query the DuckDuckGo servers:

+

HTTP URLs, which identify endpoints across the Web, are actually part of a more general family known as Uniform Resource Identifiers (URIs). The full URI specification is defined in RFC3986 and is rather complicated. Luckily, the uri library provides a strongly typed interface that takes care of much of the hassle.    

+

We’ll need a function for generating the URIs that we’re going to use to query the DuckDuckGo servers:

open Core
 open Async
@@ -815,24 +497,12 @@ 
URI Handling

   in
   Uri.add_query_param base_uri ("q", [ query ])
-

A Uri.t is constructed from the -Uri.of_string function, and a query parameter -q is added with the desired search query. The library takes -care of encoding the URI correctly when outputting it in the network -protocol.

+

A Uri.t is constructed from the Uri.of_string function, and a query parameter q is added with the desired search query. The library takes care of encoding the URI correctly when outputting it in the network protocol.

Parsing JSON Strings

-

The HTTP response from DuckDuckGo is in JSON, a common (and -thankfully simple) format that is specified in RFC4627. We’ll parse the -JSON data using the Yojson library, which was introduced in Chapter 18, Handling Json -Data.    

-

We expect the response from DuckDuckGo to come across as a JSON -record, which is represented by the Assoc tag in Yojson’s -JSON variant. We expect the definition itself to come across under -either the key “Abstract” or “Definition,” and so the following code -looks under both keys, returning the first one for which a nonempty -value is defined:

+

The HTTP response from DuckDuckGo is in JSON, a common (and thankfully simple) format that is specified in RFC4627. We’ll parse the JSON data using the Yojson library, which was introduced in Chapter 18, Handling Json Data.    

+

We expect the response from DuckDuckGo to come across as a JSON record, which is represented by the Assoc tag in Yojson’s JSON variant. We expect the definition itself to come across under either the key “Abstract” or “Definition,” and so the following code looks under both keys, returning the first one for which a nonempty value is defined:

(* Extract the "Definition" or "Abstract" field from the DuckDuckGo
    results *)
@@ -852,10 +522,7 @@ 
Parsing JSON Strings

Executing an HTTP Client Query

-

Now let’s look at the code for dispatching the search queries over -HTTP, using the Cohttp library:     

+

Now let’s look at the code for dispatching the search queries over HTTP, using the Cohttp library:     

(* Execute the DuckDuckGo search *)
 let get_definition word =
@@ -863,9 +530,7 @@ 
Executing an HTTP Client Query

   let%map string = Cohttp_async.Body.to_string body in
   word, get_definition_from_json string
-

To better understand what’s going on, it’s useful to look at the type -for Cohttp_async.Client.get, which we can do in -utop:

+

To better understand what’s going on, it’s useful to look at the type for Cohttp_async.Client.get, which we can do in utop:

#require "cohttp-async";;
 #show Cohttp_async.Client.get;;
@@ -876,18 +541,9 @@ 
Executing an HTTP Client Query

 >  Uri.t -> (Cohttp.Response.t * Cohttp_async.Body.t) Deferred.t
-

The get call takes as a required argument a URI and -returns a deferred value containing a Cohttp.Response.t -(which we ignore) and a pipe reader to which the body of the request -will be streamed.

-

In this case, the HTTP body probably isn’t very large, so we call -Cohttp_async.Body.to_string to collect the data from the -connection as a single deferred string, rather than consuming the data -incrementally.

-

Running a single search isn’t that interesting from a concurrency -perspective, so let’s write code for dispatching multiple searches in -parallel. First, we need code for formatting and printing out the search -result:

+

The get call takes as a required argument a URI and returns a deferred value containing a Cohttp.Response.t (which we ignore) and a pipe reader to which the body of the request will be streamed.

+

In this case, the HTTP body probably isn’t very large, so we call Cohttp_async.Body.to_string to collect the data from the connection as a single deferred string, rather than consuming the data incrementally.

+

Running a single search isn’t that interesting from a concurrency perspective, so let’s write code for dispatching multiple searches in parallel. First, we need code for formatting and printing out the search result:

(* Print out a word/definition pair *)
 let print_result (word, definition) =
@@ -900,17 +556,8 @@ 
Executing an HTTP Client Query

     | Some def ->
       String.concat ~sep:"\n" (Wrapper.wrap (Wrapper.make 70) def))
-

We use the Wrapper module from the textwrap -package to do the line wrapping. It may not be obvious that this routine -is using Async, but it does: the version of printf that’s -called here is actually Async’s specialized printf that -goes through the Async scheduler rather than printing directly. The -original definition of printf is shadowed by this new one -when you open Async. An important side effect of this is -that if you write an Async program and forget to start the scheduler, -calls like printf won’t actually generate any output!

-

The next function dispatches the searches in parallel, waits for the -results, and then prints:

+

We use the Wrapper module from the textwrap package to do the line wrapping. It may not be obvious that this routine is using Async, but it does: the version of printf that’s called here is actually Async’s specialized printf that goes through the Async scheduler rather than printing directly. The original definition of printf is shadowed by this new one when you open Async. An important side effect of this is that if you write an Async program and forget to start the scheduler, calls like printf won’t actually generate any output!

+

The next function dispatches the searches in parallel, waits for the results, and then prints:

(* Run many searches in parallel, printing out the results after
    they're all done. *)
@@ -918,21 +565,14 @@ 
Executing an HTTP Client Query

   let%map results = Deferred.all (List.map words ~f:get_definition) in
   List.iter results ~f:print_result
-

We used List.map to call get_definition on -each word, and Deferred.all to wait for all the results. -Here’s the type of Deferred.all:

+

We used List.map to call get_definition on each word, and Deferred.all to wait for all the results. Here’s the type of Deferred.all:

Deferred.all;;
 >- : 'a Deferred.t list -> 'a list Deferred.t = <fun>
-

The list returned by Deferred.all reflects the order of -the deferreds passed to it. As such, the definitions will be printed out -in the same order that the search words are passed in, no matter what -order the queries return in. It also means that no printing occurs until -all results arrive.

-

We could rewrite this code to print out the results as they’re -received (and thus potentially out of order) as follows:

+

The list returned by Deferred.all reflects the order of the deferreds passed to it. As such, the definitions will be printed out in the same order that the search words are passed in, no matter what order the queries return in. It also means that no printing occurs until all results arrive.

+

We could rewrite this code to print out the results as they’re received (and thus potentially out of order) as follows:

(* Run many searches in parallel, printing out the results as you
    go *)
@@ -941,20 +581,13 @@ 
Executing an HTTP Client Query

     (List.map words ~f:(fun word ->
          get_definition word >>| print_result))
-

The difference is that we both dispatch the query and print out the -result in the closure passed to map, rather than wait for -all of the results to get back and then print them out together. We use -Deferred.all_unit, which takes a list of unit -deferreds and returns a single unit deferred that becomes -determined when every deferred on the input list is determined. We can -see the type of this function in utop:

+

The difference is that we both dispatch the query and print out the result in the closure passed to map, rather than wait for all of the results to get back and then print them out together. We use Deferred.all_unit, which takes a list of unit deferreds and returns a single unit deferred that becomes determined when every deferred on the input list is determined. We can see the type of this function in utop:

Deferred.all_unit;;
 >- : unit Deferred.t list -> unit Deferred.t = <fun>
-

Finally, we create a command-line interface using -Command.async:

+

Finally, we create a command-line interface using Command.async:

let () =
   Command.async
@@ -965,8 +598,7 @@ 
Executing an HTTP Client Query

      fun () -> search_and_print words)
   |> Command_unix.run
-

And that’s all we need for a simple but usable definition -searcher:

+

And that’s all we need for a simple but usable definition searcher:

dune exec -- ./search.exe "Concurrent Programming" "OCaml"
 >Concurrent Programming
@@ -995,19 +627,8 @@ 
Executing an HTTP Client Query

Exception Handling

-

When programming with external resources, errors are everywhere. -Everything from a flaky server to a network outage to exhausting of -local resources can lead to a runtime error. When programming in OCaml, -some of these errors will show up explicitly in a function’s return -type, and some of them will show up as exceptions. We covered exception -handling in OCaml in Chapter 7, Exceptions, but as we’ll see, exception handling in -a concurrent program presents some new challenges.    

-

Let’s get a better sense of how exceptions work in Async by creating -an asynchronous computation that (sometimes) fails with an exception. -The function maybe_raise blocks for half a second, and then -either throws an exception or returns unit, alternating -between the two behaviors on subsequent calls:

+

When programming with external resources, errors are everywhere. Everything from a flaky server to a network outage to exhausting of local resources can lead to a runtime error. When programming in OCaml, some of these errors will show up explicitly in a function’s return type, and some of them will show up as exceptions. We covered exception handling in OCaml in Chapter 7, Exceptions, but as we’ll see, exception handling in a concurrent program presents some new challenges.    

+

Let’s get a better sense of how exceptions work in Async by creating an asynchronous computation that (sometimes) fails with an exception. The function maybe_raise blocks for half a second, and then either throws an exception or returns unit, alternating between the two behaviors on subsequent calls:

let maybe_raise =
   let should_fail = ref false in
@@ -1023,13 +644,8 @@ 
Exception Handling

 >Exception: (monitor.ml.Error Exit ("Caught by monitor block_on_async"))
-

In utop, the exception thrown by -maybe_raise () terminates the evaluation of just that -expression, but in a standalone program, an uncaught exception would -bring down the entire process.

-

So, how could we capture and handle such an exception? You might try -to do this using OCaml’s built-in try/with expression, but -as you can see that doesn’t quite do the trick:

+

In utop, the exception thrown by maybe_raise () terminates the evaluation of just that expression, but in a standalone program, an uncaught exception would bring down the entire process.

+

So, how could we capture and handle such an exception? You might try to do this using OCaml’s built-in try/with expression, but as you can see that doesn’t quite do the trick:

let handle_error () =
   try
@@ -1043,18 +659,8 @@ 
Exception Handling

 >Exception: (monitor.ml.Error Exit ("Caught by monitor block_on_async"))
-

This didn’t work because try/with only captures -exceptions that are thrown by the code executed synchronously within it, -while maybe_raise schedules an Async job that will throw an -exception in the future, after the try/with expression has -exited.

-

We can capture this kind of asynchronous error using the -try_with function provided by Async. -try_with f takes as its argument a deferred-returning thunk -f and returns a deferred that becomes determined either as -Ok of whatever f returned, or -Error exn if f threw an exception before its -return value became determined.      

+

This didn’t work because try/with only captures exceptions that are thrown by the code executed synchronously within it, while maybe_raise schedules an Async job that will throw an exception in the future, after the try/with expression has exited.

+

We can capture this kind of asynchronous error using the try_with function provided by Async. try_with f takes as its argument a deferred-returning thunk f and returns a deferred that becomes determined either as Ok of whatever f returned, or Error exn if f threw an exception before its return value became determined.      

Here’s a trivial example of try_with in action.

let handle_error () =
@@ -1070,25 +676,9 @@ 
Exception Handling

Monitors

-

try_with is a useful tool for handling exceptions in -Async, but it’s not the whole story. All of Async’s exception-handling -mechanisms, try_with included, are built on top of Async’s -system of monitors, which are inspired by the error-handling -mechanism in Erlang of the same name. Monitors are fairly low-level and -are only occasionally used directly, but it’s nonetheless worth -understanding how they work.  

-

In Async, a monitor is a context that determines what to do when -there is an unhandled exception. Every Async job runs within the context -of some monitor, which, when the job is running, is referred to as the -current monitor. When a new Async job is scheduled, say, using -bind or map, it inherits the current monitor -of the job that spawned it.

-

Monitors are arranged in a tree—when a new monitor is created (say, -using Monitor.create), it is a child of the current -monitor. You can explicitly run jobs within a monitor using -within, which takes a thunk that returns a nondeferred -value, or within', which takes a thunk that returns a -deferred. Here’s an example:

+

try_with is a useful tool for handling exceptions in Async, but it’s not the whole story. All of Async’s exception-handling mechanisms, try_with included, are built on top of Async’s system of monitors, which are inspired by the error-handling mechanism in Erlang of the same name. Monitors are fairly low-level and are only occasionally used directly, but it’s nonetheless worth understanding how they work.  

+

In Async, a monitor is a context that determines what to do when there is an unhandled exception. Every Async job runs within the context of some monitor, which, when the job is running, is referred to as the current monitor. When a new Async job is scheduled, say, using bind or map, it inherits the current monitor of the job that spawned it.

+

Monitors are arranged in a tree—when a new monitor is created (say, using Monitor.create), it is a child of the current monitor. You can explicitly run jobs within a monitor using within, which takes a thunk that returns a nondeferred value, or within', which takes a thunk that returns a deferred. Here’s an example:

let blow_up () =
   let monitor = Monitor.create ~name:"blow up monitor" () in
@@ -1100,20 +690,8 @@ 
Monitors

 >Exception: (monitor.ml.Error Exit ("Caught by monitor blow up monitor"))
-

In addition to the ordinary stack-trace, the exception displays the -trace of monitors through which the exception traveled, starting at the -one we created, called “blow up monitor.” The other monitors you see -come from utop’s special handling of deferreds.

-

Monitors can do more than just augment the error-trace of an -exception. You can also use a monitor to explicitly handle errors -delivered to that monitor. The -Monitor.detach_and_get_error_stream call is a particularly -important one. It detaches the monitor from its parent, handing back the -stream of errors that would otherwise have been delivered to the parent -monitor. This allows one to do custom handling of errors, which may -include reraising errors to the parent. Here is a very simple example of -a function that captures and ignores errors in the processes it -spawns.

+

In addition to the ordinary stack-trace, the exception displays the trace of monitors through which the exception traveled, starting at the one we created, called “blow up monitor.” The other monitors you see come from utop’s special handling of deferreds.

+

Monitors can do more than just augment the error-trace of an exception. You can also use a monitor to explicitly handle errors delivered to that monitor. The Monitor.detach_and_get_error_stream call is a particularly important one. It detaches the monitor from its parent, handing back the stream of errors that would otherwise have been delivered to the parent monitor. This allows one to do custom handling of errors, which may include reraising errors to the parent. Here is a very simple example of a function that captures and ignores errors in the processes it spawns.

let swallow_error () =
   let monitor = Monitor.create () in
@@ -1125,15 +703,8 @@ 
Monitors

 >val swallow_error : unit -> 'a Deferred.t = <fun>
-

The deferred returned by this function is never determined, since the -computation ends with an exception rather than a return value. That -means that if we run this function in utop, we’ll never get -our prompt back.

-

We can fix this by using Deferred.any along with a -timeout to get a deferred we know will become determined eventually. -Deferred.any takes a list of deferreds, and returns a -deferred which will become determined assuming any of its arguments -becomes determined.

+

The deferred returned by this function is never determined, since the computation ends with an exception rather than a return value. That means that if we run this function in utop, we’ll never get our prompt back.

+

We can fix this by using Deferred.any along with a timeout to get a deferred we know will become determined eventually. Deferred.any takes a list of deferreds, and returns a deferred which will become determined assuming any of its arguments becomes determined.

Deferred.any [ after (Time.Span.of_sec 0.5)
              ; swallow_error () ];;
@@ -1141,13 +712,8 @@ 
Monitors

 >- : unit = ()
-

As you can see, the message “an error happened” is printed out before -the timeout expires.

-

Here’s an example of a monitor that passes some exceptions through to -the parent and handles others. Exceptions are sent to the parent using -Monitor.send_exn, with Monitor.current being -called to find the current monitor, which is the parent of the newly -created monitor.

+

As you can see, the message “an error happened” is printed out before the timeout expires.

+

Here’s an example of a monitor that passes some exceptions through to the parent and handles others. Exceptions are sent to the parent using Monitor.send_exn, with Monitor.current being called to find the current monitor, which is the parent of the newly created monitor.

exception Ignore_me;;
 >exception Ignore_me
@@ -1166,14 +732,8 @@ 
Monitors

 >val swallow_some_errors : exn -> 'a Deferred.t = <fun>
-

Note that we use Monitor.extract_exn to grab the -underlying exception that was thrown. Async wraps exceptions it catches -with extra information, including the monitor trace, so you need to grab -the underlying exception if you want to depend on the details of the -original exception thrown.

-

If we pass in an exception other than Ignore_me, like, -say, the built-in exception Not_found, then the exception -will be passed to the parent monitor and delivered as usual:

+

Note that we use Monitor.extract_exn to grab the underlying exception that was thrown. Async wraps exceptions it catches with extra information, including the monitor trace, so you need to grab the underlying exception if you want to depend on the details of the original exception thrown.

+

If we pass in an exception other than Ignore_me, like, say, the built-in exception Not_found, then the exception will be passed to the parent monitor and delivered as usual:

exception Another_exception;;
 >exception Another_exception
@@ -1183,8 +743,7 @@ 
Monitors

 >(monitor.ml.Error (Another_exception) ("Caught by monitor (id 69)")).
-

If instead we use Ignore_me, the exception will be -ignored, and the computation will finish when the timeout expires.

+

If instead we use Ignore_me, the exception will be ignored, and the computation will finish when the timeout expires.

Deferred.any [ after (Time.Span.of_sec 0.5)
              ; swallow_some_errors Ignore_me ];;
@@ -1192,28 +751,13 @@ 
Monitors

 >- : unit = ()
-

In practice, you should rarely use monitors directly, and instead use -functions like try_with and Monitor.protect -that are built on top of monitors. One example of a library that uses -monitors directly is Tcp.Server.create, which tracks both -exceptions thrown by the logic that handles the network connection and -by the callback for responding to an individual request, in either case -responding to an exception by closing the connection. It is for building -this kind of custom error handling that monitors can be helpful.

+

In practice, you should rarely use monitors directly, and instead use functions like try_with and Monitor.protect that are built on top of monitors. One example of a library that uses monitors directly is Tcp.Server.create, which tracks both exceptions thrown by the logic that handles the network connection and by the callback for responding to an individual request, in either case responding to an exception by closing the connection. It is for building this kind of custom error handling that monitors can be helpful.

Example: Handling Exceptions with DuckDuckGo

-

Let’s now go back and improve the exception handling of our -DuckDuckGo client. In particular, we’ll change it so that any query that -fails is reported without preventing other queries from completing. -  

-

The search code as it is fails rarely, so let’s make a change that -allows us to trigger failures more predictably. We’ll do this by making -it possible to distribute the requests over multiple servers. Then, -we’ll handle the errors that occur when one of those servers is -misspecified.

-

First we’ll need to change query_uri to take an argument -specifying the server to connect to:

+

Let’s now go back and improve the exception handling of our DuckDuckGo client. In particular, we’ll change it so that any query that fails is reported without preventing other queries from completing.   

+

The search code as it is fails rarely, so let’s make a change that allows us to trigger failures more predictably. We’ll do this by making it possible to distribute the requests over multiple servers. Then, we’ll handle the errors that occur when one of those servers is misspecified.

+

First we’ll need to change query_uri to take an argument specifying the server to connect to:

(* Generate a DuckDuckGo search URI from a query string *)
 let query_uri ~server query =
@@ -1223,11 +767,8 @@ 
Example: Handling Exceptions with DuckDuckGo

   in
   Uri.add_query_param base_uri ("q", [ query ])
-

In addition, we’ll make the necessary changes to get the list of -servers on the command-line, and to distribute the search queries -round-robin across the list of servers.

-

Now, let’s see what happens when we rebuild the application and run -it on two servers, one of which won’t respond to the query.

+

In addition, we’ll make the necessary changes to get the list of servers on the command-line, and to distribute the search queries round-robin across the list of servers.

+

Now, let’s see what happens when we rebuild the application and run it on two servers, one of which won’t respond to the query.

dune exec -- ./search.exe -servers localhost,api.duckduckgo.com "Concurrent Programming" "OCaml"
 >(monitor.ml.Error (Unix.Unix_error "Connection refused" connect 127.0.0.1:80)
@@ -1237,12 +778,7 @@ 
Example: Handling Exceptions with DuckDuckGo

 >  "Caught by monitor Tcp.close_sock_on_error"))
 [1]
-

As you can see, we got a “Connection refused” failure, which ends the -entire program, even though one of the two queries would have gone -through successfully on its own. We can handle the failures of -individual connections separately by using the try_with -function within each call to get_definition, as -follows:

+

As you can see, we got a “Connection refused” failure, which ends the entire program, even though one of the two queries would have gone through successfully on its own. We can handle the failures of individual connections separately by using the try_with function within each call to get_definition, as follows:

(* Execute the DuckDuckGo search *)
 let get_definition ~server word =
@@ -1257,13 +793,8 @@ 
Example: Handling Exceptions with DuckDuckGo

   | Ok (word, result) -> word, Ok result
   | Error _ -> word, Error "Unexpected failure"
-

Here, we first use try_with to capture the exception, -and then use match%map (another syntax provided by -ppx_let) to convert the error into the form we want: a pair -whose first element is the word being searched for, and the second -element is the (possibly erroneous) result.  

-

Now we just need to change the code for print_result so -that it can handle the new type:

+

Here, we first use try_with to capture the exception, and then use match%map (another syntax provided by ppx_let) to convert the error into the form we want: a pair whose first element is the word being searched for, and the second element is the (possibly erroneous) result.  

+

Now we just need to change the code for print_result so that it can handle the new type:

(* Print out a word/definition pair *)
 let print_result (word, definition) =
@@ -1277,8 +808,7 @@ 
Example: Handling Exceptions with DuckDuckGo

     | Ok (Some def) ->
       String.concat ~sep:"\n" (Wrapper.wrap (Wrapper.make 70) def))
-

Now, if we run that same query, we’ll get individualized handling of -the connection failures:

+

Now, if we run that same query, we’ll get individualized handling of the connection failures:

dune exec -- ./search.exe -servers localhost,api.duckduckgo.com "Concurrent Programming" OCaml
 >Concurrent Programming
@@ -1297,27 +827,12 @@ 
Example: Handling Exceptions with DuckDuckGo

Now, only the query that went to localhost failed.

-

Note that in this code, we’re relying on the fact that -Cohttp_async.Client.get will clean up after itself after an -exception, in particular by closing its file descriptors. If you need to -implement such functionality directly, you may want to use the -Monitor.protect call, which is analogous to the -protect call described in Chapter 7, Cleaning Up In The Presence Of Exceptions.

+

Note that in this code, we’re relying on the fact that Cohttp_async.Client.get will clean up after itself after an exception, in particular by closing its file descriptors. If you need to implement such functionality directly, you may want to use the Monitor.protect call, which is analogous to the protect call described in Chapter 7, Cleaning Up In The Presence Of Exceptions.

Timeouts, Cancellation, and Choices

-

In a concurrent program, one often needs to combine results from -multiple, distinct concurrent subcomputations going on in the same -program. We already saw this in our DuckDuckGo example, where we used -Deferred.all and Deferred.all_unit to wait for -a list of deferreds to become determined. Another useful primitive is -Deferred.both, which lets you wait until two deferreds of -different types have returned, returning both values as a tuple. Here, -we use the function sec, which is shorthand for creating a -time-span equal to a given number of seconds:     

+

In a concurrent program, one often needs to combine results from multiple, distinct concurrent subcomputations going on in the same program. We already saw this in our DuckDuckGo example, where we used Deferred.all and Deferred.all_unit to wait for a list of deferreds to become determined. Another useful primitive is Deferred.both, which lets you wait until two deferreds of different types have returned, returning both values as a tuple. Here, we use the function sec, which is shorthand for creating a time-span equal to a given number of seconds:     

let string_and_float =
   Deferred.both
@@ -1328,11 +843,7 @@ 
Timeouts, Cancellation, and Choices

 >- : string * float = ("A", 32.33)
-

Sometimes, however, we want to wait only for the first of multiple -events to occur. This happens particularly when dealing with timeouts. -In that case, we can use the call Deferred.any, which, -given a list of deferreds, returns a single deferred that will become -determined once any of the values on the list is determined.

+

Sometimes, however, we want to wait only for the first of multiple events to occur. This happens particularly when dealing with timeouts. In that case, we can use the call Deferred.any, which, given a list of deferreds, returns a single deferred that will become determined once any of the values on the list is determined.

Deferred.any
 [ (let%map () = after (sec 0.5) in "half a second")
@@ -1342,10 +853,7 @@ 
Timeouts, Cancellation, and Choices

 >- : string = "half a second"
-

Let’s use this to add timeouts to our DuckDuckGo searches. The -following code is a wrapper for get_definition that takes a -timeout (in the form of a Time.Span.t) and returns either -the definition, or, if that takes too long, an error:

+

Let’s use this to add timeouts to our DuckDuckGo searches. The following code is a wrapper for get_definition that takes a timeout (in the form of a Time.Span.t) and returns either the definition, or, if that takes too long, an error:

let get_definition_with_timeout ~server ~timeout word =
   Deferred.any
@@ -1356,19 +864,9 @@ 
Timeouts, Cancellation, and Choices

       | word, (Ok _ as x) -> word, x)
     ]
-

We use let%map above to transform the deferred values -we’re waiting for so that Deferred.any can choose between -values of the same type.

-

A problem with this code is that the HTTP query kicked off by -get_definition is not actually shut down when the timeout -fires. As such, get_definition_with_timeout can leak an -open connection. Happily, Cohttp does provide a way of shutting down a -client. You can pass a deferred under the label interrupt -to Cohttp_async.Client.get. Once interrupt is -determined, the client connection will be shut down.

-

The following code shows how you can change -get_definition and get_definition_with_timeout -to cancel the get call if the timeout expires:

+

We use let%map above to transform the deferred values we’re waiting for so that Deferred.any can choose between values of the same type.

+

A problem with this code is that the HTTP query kicked off by get_definition is not actually shut down when the timeout fires. As such, get_definition_with_timeout can leak an open connection. Happily, Cohttp does provide a way of shutting down a client. You can pass a deferred under the label interrupt to Cohttp_async.Client.get. Once interrupt is determined, the client connection will be shut down.

+

The following code shows how you can change get_definition and get_definition_with_timeout to cancel the get call if the timeout expires:

(* Execute the DuckDuckGo search *)
 let get_definition ~server ~interrupt word =
@@ -1383,9 +881,7 @@ 
Timeouts, Cancellation, and Choices

   | Ok (word, result) -> word, Ok result
   | Error _ -> word, Error "Unexpected failure"
-

Next, we’ll modify get_definition_with_timeout to create -a deferred to pass in to get_definition, which will become -determined when our timeout expires:

+

Next, we’ll modify get_definition_with_timeout to create a deferred to pass in to get_definition, which will become determined when our timeout expires:

let get_definition_with_timeout ~server ~timeout word =
   match%map
@@ -1394,17 +890,8 @@ 
Timeouts, Cancellation, and Choices

   | word, (Ok _ as x) -> word, x
   | word, Error _ -> word, Error "Unexpected failure"
-

This will cause the connection to shutdown cleanly when we time out; -but our code no longer explicitly knows whether or not the timeout has -kicked in. In particular, the error message on a timeout will now be -"Unexpected failure" rather than "Timed out", -which it was in our previous implementation.

-

We can get more precise handling of timeouts using Async’s -choose function. choose lets you pick among a -collection of different deferreds, reacting to exactly one of them. Each -deferred is paired, using the function choice, with a -function that is called if and only if that deferred is chosen. Here’s -the type signature of choice and choose:

+

This will cause the connection to shutdown cleanly when we time out; but our code no longer explicitly knows whether or not the timeout has kicked in. In particular, the error message on a timeout will now be "Unexpected failure" rather than "Timed out", which it was in our previous implementation.

+

We can get more precise handling of timeouts using Async’s choose function. choose lets you pick among a collection of different deferreds, reacting to exactly one of them. Each deferred is paired, using the function choice, with a function that is called if and only if that deferred is chosen. Here’s the type signature of choice and choose:

choice;;
 >- : 'a Deferred.t -> ('a -> 'b) -> 'b Deferred.Choice.t = <fun>
@@ -1412,13 +899,8 @@ 
Timeouts, Cancellation, and Choices

 >- : 'a Deferred.Choice.t list -> 'a Deferred.t = <fun>
-

Note that there’s no guarantee that the winning deferred will be the -one that becomes determined first. But choose does -guarantee that only one choice will be chosen, and only the -chosen choice will execute the attached function.

-

In the following example, we use choose to ensure that -the interrupt deferred becomes determined if and only if -the timeout deferred is chosen. Here’s the code:

+

Note that there’s no guarantee that the winning deferred will be the one that becomes determined first. But choose does guarantee that only one choice will be chosen, and only the chosen choice will execute the attached function.

+

In the following example, we use choose to ensure that the interrupt deferred becomes determined if and only if the timeout deferred is chosen. Here’s the code:

let get_definition_with_timeout ~server ~timeout word =
   let interrupt = Ivar.create () in
@@ -1437,8 +919,7 @@ 
Timeouts, Cancellation, and Choices

           word, result')
     ]
-

Now, if we run this with a suitably small timeout, we’ll see that one -query succeeds and the other fails reporting a timeout:

+

Now, if we run this with a suitably small timeout, we’ll see that one query succeeds and the other fails reporting a timeout:

dune exec -- ./search.exe "concurrent programming" ocaml -timeout 0.1s
 >concurrent programming
@@ -1462,62 +943,19 @@ 
Timeouts, Cancellation, and Choices

Working with System Threads

-

Although we haven’t worked with them yet, OCaml does have built-in -support for true system threads, i.e., kernel-level threads whose -interleaving is controlled by the operating system. We discussed in the -beginning of the chapter the advantages of Async’s cooperative threading -model over system threads, but even if you mostly use Async, OCaml’s -system threads are sometimes necessary, and it’s worth understanding -them.        

-

The most surprising aspect of OCaml’s system threads is that they -don’t afford you any access to physical parallelism. That’s because -OCaml’s runtime has a single runtime lock that at most one thread can be -holding at a time.

-

Given that threads don’t provide physical parallelism, why are they -useful at all?

-

The most common reason for using system threads is that there are -some operating system calls that have no nonblocking alternative, which -means that you can’t run them directly in a system like Async without -blocking your entire program. For this reason, Async maintains a thread -pool for running such calls. Most of the time, as a user of Async you -don’t need to think about this, but it is happening under the covers. - 

-

Another reason to have multiple threads is to deal with non-OCaml -libraries that have their own event loop or for another reason need -their own threads. In that case, it’s sometimes useful to run some OCaml -code on the foreign thread as part of the communication to your main -program. OCaml’s foreign function interface is discussed in more detail -in Chapter 22, Foreign Function Interface.

-
-

Multicore OCaml

-

OCaml doesn’t support truly parallel threads today, but it will soon. -The current development branch of OCaml, which is expected to be -released in 2022 as OCaml 5.0, has a long awaited multicore-capable -garbage collector, which is the result of years of research and hard -implementation work.  

-

We won’t discuss the multicore gc here in part because it’s not yet -released, and in part because there’s a lot of open questions about how -OCaml programs should take advantage of multicore in a way that’s safe, -convenient, and performant. Given all that, we just don’t know enough to -write a chapter about multicore today.

-

In any case, while multicore OCaml isn’t here yet, it’s an exciting -part of OCaml’s near-term future.

-
-

Another occasional use for system threads is to better interoperate -with compute-intensive OCaml code. In Async, if you have a long-running -computation that never calls bind or map, then -that computation will block out the Async runtime until it -completes.

-

One way of dealing with this is to explicitly break up the -calculation into smaller pieces that are separated by binds. But -sometimes this explicit yielding is impractical, since it may involve -intrusive changes to an existing codebase. Another solution is to run -the code in question in a separate thread. Async’s -In_thread module provides multiple facilities for doing -just this, In_thread.run being the simplest. We can simply -write:  

+

Although we haven’t worked with them yet, OCaml does have built-in support for true system threads, i.e., kernel-level threads whose interleaving is controlled by the operating system. We discussed in the beginning of the chapter the advantages of Async’s cooperative threading model over system threads, but even if you mostly use Async, OCaml’s system threads are sometimes necessary, and it’s worth understanding them.        

+

The most surprising aspect of OCaml’s system threads is that they don’t afford you any access to physical parallelism. That’s because OCaml’s runtime has a single runtime lock that at most one thread can be holding at a time.

+

Given that threads don’t provide physical parallelism, why are they useful at all?

+

The most common reason for using system threads is that there are some operating system calls that have no nonblocking alternative, which means that you can’t run them directly in a system like Async without blocking your entire program. For this reason, Async maintains a thread pool for running such calls. Most of the time, as a user of Async you don’t need to think about this, but it is happening under the covers.  

+

Another reason to have multiple threads is to deal with non-OCaml libraries that have their own event loop or for another reason need their own threads. In that case, it’s sometimes useful to run some OCaml code on the foreign thread as part of the communication to your main program. OCaml’s foreign function interface is discussed in more detail in Chapter 22, Foreign Function Interface.

+
+

Multicore OCaml

+

OCaml doesn’t support truly parallel threads today, but it will soon. The current development branch of OCaml, which is expected to be released in 2022 as OCaml 5.0, has a long awaited multicore-capable garbage collector, which is the result of years of research and hard implementation work.  

+

We won’t discuss the multicore gc here in part because it’s not yet released, and in part because there’s a lot of open questions about how OCaml programs should take advantage of multicore in a way that’s safe, convenient, and performant. Given all that, we just don’t know enough to write a chapter about multicore today.

+

In any case, while multicore OCaml isn’t here yet, it’s an exciting part of OCaml’s near-term future.

+
+

Another occasional use for system threads is to better interoperate with compute-intensive OCaml code. In Async, if you have a long-running computation that never calls bind or map, then that computation will block out the Async runtime until it completes.

+

One way of dealing with this is to explicitly break up the calculation into smaller pieces that are separated by binds. But sometimes this explicit yielding is impractical, since it may involve intrusive changes to an existing codebase. Another solution is to run the code in question in a separate thread. Async’s In_thread module provides multiple facilities for doing just this, In_thread.run being the simplest. We can simply write:  

let def = In_thread.run (fun () -> List.range 1 10);;
 >val def : int list Deferred.t = <abstr>
@@ -1525,16 +963,8 @@ 
Multicore OCaml

 >- : int list = [1; 2; 3; 4; 5; 6; 7; 8; 9]
-

to cause List.range 1 10 to be run on one of Async’s -worker threads. When the computation is complete, the result is placed -in the deferred, where it can be used in the ordinary way from -Async.

-

Interoperability between Async and system threads can be quite -tricky. Consider the following function for testing how responsive Async -is. The function takes a deferred-returning thunk, and it first runs -that thunk, and then uses Clock.every to wake up every 100 -milliseconds and print out a timestamp, until the returned deferred -becomes determined, at which point it prints out one last timestamp:

+

to cause List.range 1 10 to be run on one of Async’s worker threads. When the computation is complete, the result is placed in the deferred, where it can be used in the ordinary way from Async.

+

Interoperability between Async and system threads can be quite tricky. Consider the following function for testing how responsive Async is. The function takes a deferred-returning thunk, and it first runs that thunk, and then uses Clock.every to wake up every 100 milliseconds and print out a timestamp, until the returned deferred becomes determined, at which point it prints out one last timestamp:

let log_delays thunk =
   let start = Time.now () in
@@ -1552,8 +982,7 @@ 
Multicore OCaml

 >val log_delays : (unit -> unit Deferred.t) -> unit Deferred.t = <fun>
-

If we feed this function a simple timeout deferred, it works as you -might expect, waking up roughly every 100 milliseconds:

+

If we feed this function a simple timeout deferred, it works as you might expect, waking up roughly every 100 milliseconds:

log_delays (fun () -> after (sec 0.5));;
 >37.670135498046875us, 100.65722465515137ms, 201.19547843933105ms, 301.85389518737793ms, 402.58693695068359ms,
@@ -1561,8 +990,7 @@ 
Multicore OCaml

 >- : unit = ()
-

Now see what happens if, instead of waiting on a clock event, we wait -for a busy loop to finish running:

+

Now see what happens if, instead of waiting on a clock event, we wait for a busy loop to finish running:

let busy_loop () =
   let x = ref None in
@@ -1573,11 +1001,8 @@ 
Multicore OCaml

 >- : unit = ()
-

As you can see, instead of waking up 10 times a second, -log_delays is blocked out entirely while -busy_loop churns away.

-

If, on the other hand, we use In_thread.run to offload -this to a different system thread, the behavior will be different:

+

As you can see, instead of waking up 10 times a second, log_delays is blocked out entirely while busy_loop churns away.

+

If, on the other hand, we use In_thread.run to offload this to a different system thread, the behavior will be different:

log_delays (fun () -> In_thread.run busy_loop);;
 >31.709671020507812us, 107.50102996826172ms, 207.65542984008789ms, 307.95812606811523ms, 458.15873146057129ms, 608.44659805297852ms, 708.55593681335449ms, 808.81166458129883ms,
@@ -1585,19 +1010,8 @@ 
Multicore OCaml

 >- : unit = ()
-

Now log_delays does get a chance to run, but it’s no -longer at clean 100 millisecond intervals. The reason is that now that -we’re using system threads, we are at the mercy of the operating system -to decide when each thread gets scheduled. The behavior of threads is -very much dependent on the operating system and how it is -configured.

-

Another tricky aspect of dealing with OCaml threads has to do with -allocation. When compiling to native code, OCaml’s threads only get a -chance to give up the runtime lock when they interact with the -allocator, so if there’s a piece of code that doesn’t allocate at all, -then it will never allow another OCaml thread to run. Bytecode doesn’t -have this behavior, so if we run a nonallocating loop in bytecode, our -timer process will get to run:

+

Now log_delays does get a chance to run, but it’s no longer at clean 100 millisecond intervals. The reason is that now that we’re using system threads, we are at the mercy of the operating system to decide when each thread gets scheduled. The behavior of threads is very much dependent on the operating system and how it is configured.

+

Another tricky aspect of dealing with OCaml threads has to do with allocation. When compiling to native code, OCaml’s threads only get a chance to give up the runtime lock when they interact with the allocator, so if there’s a piece of code that doesn’t allocate at all, then it will never allow another OCaml thread to run. Bytecode doesn’t have this behavior, so if we run a nonallocating loop in bytecode, our timer process will get to run:

let noalloc_busy_loop () =
   for i = 0 to 100_000_000 do () done;;
@@ -1608,52 +1022,25 @@ 
Multicore OCaml

 >- : unit = ()
-

But if we compile this to a native-code executable, then the -nonallocating busy loop will block anything else from running:

+

But if we compile this to a native-code executable, then the nonallocating busy loop will block anything else from running:

dune exec -- native_code_log_delays.exe
 >197.41058349609375us,
 >Finished at: 1.2127914428710938s,
-

The takeaway from these examples is that predicting thread -interleavings is a subtle business. Staying within the bounds of Async -has its limitations, but it leads to more predictable behavior.

+

The takeaway from these examples is that predicting thread interleavings is a subtle business. Staying within the bounds of Async has its limitations, but it leads to more predictable behavior.

Thread-Safety and Locking

-

Once you start working with system threads, you’ll need to be careful -about mutable data structures. Most mutable OCaml data structures will -behave non-deterministically when accessed concurrently by multiple -threads. The issues you can run into range from runtime exceptions to -corrupted data structures. That means you should almost always use -mutexes when sharing mutable data between different systems threads. -Even data structures that seem like they should be safe but are mutable -under the covers, like lazy values, can behave in surprising ways when -accessed from multiple threads.     

-

There are two commonly available mutex packages for OCaml: the -Mutex module that’s part of the standard library, which is -just a wrapper over OS-level mutexes and Nano_mutex, a more -efficient alternative that takes advantage of some of the locking done -by the OCaml runtime to avoid needing to create an OS-level mutex much -of the time. As a result, creating a Nano_mutex.t is 20 -times faster than creating a Mutex.t, and acquiring the -mutex is about 40 percent faster.

-

Overall, combining Async and threads is quite tricky, but it’s pretty -simple if the following two conditions hold:

+

Once you start working with system threads, you’ll need to be careful about mutable data structures. Most mutable OCaml data structures will behave non-deterministically when accessed concurrently by multiple threads. The issues you can run into range from runtime exceptions to corrupted data structures. That means you should almost always use mutexes when sharing mutable data between different systems threads. Even data structures that seem like they should be safe but are mutable under the covers, like lazy values, can behave in surprising ways when accessed from multiple threads.     

+

There are two commonly available mutex packages for OCaml: the Mutex module that’s part of the standard library, which is just a wrapper over OS-level mutexes and Nano_mutex, a more efficient alternative that takes advantage of some of the locking done by the OCaml runtime to avoid needing to create an OS-level mutex much of the time. As a result, creating a Nano_mutex.t is 20 times faster than creating a Mutex.t, and acquiring the mutex is about 40 percent faster.

+

Overall, combining Async and threads is quite tricky, but it’s pretty simple if the following two conditions hold:

    -

  • There is no shared mutable state between the various threads -involved.

    • -

  • The computations executed by In_thread.run do not -make any calls to the Async library.

    • +

  • There is no shared mutable state between the various threads involved.

    • +

  • The computations executed by In_thread.run do not make any calls to the Async library.

-

That said, you can safely use threads in ways that violate these -constraints. In particular, foreign threads can acquire the Async lock -using calls from the Thread_safe module in Async, and -thereby run Async computations safely. This is a very flexible way of -connecting threads to the Async world, but it’s a complex use case that -is beyond the scope of this chapter.

+

That said, you can safely use threads in ways that violate these constraints. In particular, foreign threads can acquire the Async lock using calls from the Thread_safe module in Async, and thereby run Async computations safely. This is a very flexible way of connecting threads to the Async world, but it’s a complex use case that is beyond the scope of this chapter.

-

Next: Chapter 17Testing

 \ No newline at end of file +

Next: Chapter 17Testing

 \ No newline at end of file diff --git a/css/app.css b/css/app.css index 9815afe3a..835fab954 100644 --- a/css/app.css +++ b/css/app.css @@ -1438,7 +1438,7 @@ h6 { body:after{ z-index:1000; - content: "beta"; + content: "2nd Ed"; position: fixed; width: 130px; height: 40px; diff --git a/data-serialization.html b/data-serialization.html index e6f475b82..a3caf0d31 100644 --- a/data-serialization.html +++ b/data-serialization.html @@ -1,34 +1,19 @@ -Data Serialization with S-Expressions - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Data Serialization with S-Expressions - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Data Serialization with S-Expressions

-

S-expressions are nested parenthetical expressions whose atomic -values are strings. They were first popularized by the Lisp programming -language in the 1960s, and have remained one of the simplest and most -effective ways to encode structured data in a human-readable and -editable form.      

+

S-expressions are nested parenthetical expressions whose atomic values are strings. They were first popularized by the Lisp programming language in the 1960s, and have remained one of the simplest and most effective ways to encode structured data in a human-readable and editable form.      

An example s-expression might look like this.

(this (is an) (s expression))
-

S-expressions play a major role in Base and Core, effectively acting -as the default serialization format. Indeed, we’ve encountered -s-expressions multiple times already, including in Chapter 7, Error -Handling, Chapter 10, Functors, and Chapter 11, First Class Modules.

-

This chapter will go into s-expressions in more depth. In particular, -we’ll discuss:

+

S-expressions play a major role in Base and Core, effectively acting as the default serialization format. Indeed, we’ve encountered s-expressions multiple times already, including in Chapter 7, Error Handling, Chapter 10, Functors, and Chapter 11, First Class Modules.

+

This chapter will go into s-expressions in more depth. In particular, we’ll discuss:

    -

  • The details of the s-expression format, including how to parse it -while generating good error messages for debugging malformed -inputs

    • -

  • How to generate converters between s-expressions and arbitrary -OCaml types

    • -

  • How to use annotations to control the behavior of these generated -converters

    • -

  • How to integrate s-expressions into your interfaces, in -particular how to add s-expression converters to a module without -breaking abstraction boundaries

    • +

  • The details of the s-expression format, including how to parse it while generating good error messages for debugging malformed inputs

    • +

  • How to generate converters between s-expressions and arbitrary OCaml types

    • +

  • How to use annotations to control the behavior of these generated converters

    • +

  • How to integrate s-expressions into your interfaces, in particular how to add s-expression converters to a module without breaking abstraction boundaries

-

We’ll tie this together at the end of the chapter with a simple -s-expression formatted configuration file for a web server

+

We’ll tie this together at the end of the chapter with a simple s-expression formatted configuration file for a web server

Basic Usage

The type used to represent an s-expression is quite simple:  

@@ -39,12 +24,7 @@

Basic Usage

| List of t list end
-

An s-expression can be thought of as a tree where each node contains -a list of its children, and where the leaves of the tree are strings. -Core provides good support for s-expressions in its Sexp -module, including functions for converting s-expressions to and from -strings. Let’s rewrite our example s-expression in terms of this -type:

+

An s-expression can be thought of as a tree where each node contains a list of its children, and where the leaves of the tree are strings. Core provides good support for s-expressions in its Sexp module, including functions for converting s-expressions to and from strings. Let’s rewrite our example s-expression in terms of this type:

open Core;;
 Sexp.List [
@@ -55,10 +35,7 @@ 
Basic Usage

 >- : Sexp.t = (this (is an) (s expression))
-

This prints out nicely because Core registers a pretty printer with -the toplevel. This pretty printer is based on the functions in -Sexp for converting s-expressions to and from strings: - 

+

This prints out nicely because Core registers a pretty printer with the toplevel. This pretty printer is based on the functions in Sexp for converting s-expressions to and from strings:  

Sexp.to_string (Sexp.List [Sexp.Atom "1"; Sexp.Atom "2"]);;
 >- : string = "(1 2)"
@@ -66,12 +43,9 @@ 
Basic Usage

 >- : Sexp.t = (1 2 (3 4))
-
-

Base, Core, and Parsexp

-

In these examples, we’re using Core rather than Base because Core has -integrated support for parsing s-expressions, courtesy of the -Parsexp library. If you just use Base, you’ll find that you -don’t have Sexp.of_string at your disposal.

+
+

Base, Core, and Parsexp

+

In these examples, we’re using Core rather than Base because Core has integrated support for parsing s-expressions, courtesy of the Parsexp library. If you just use Base, you’ll find that you don’t have Sexp.of_string at your disposal.

open Base;;
 Sexp.of_string "(1 2 3)";;
@@ -83,20 +57,14 @@ 
Base, Core, and Parsexp

 >       This is not a function; it cannot be applied.
-

That’s because, in an attempt to keep Base light, the -s-expression parsing functions aren’t included. That said, you can -always use them by calling out to the corresponding functions from the -Parsexp library:

+

That’s because, in an attempt to keep Base light, the s-expression parsing functions aren’t included. That said, you can always use them by calling out to the corresponding functions from the Parsexp library:

Parsexp.Single.parse_string_exn "(1 2 3)";;
 >- : Sexp.t = (1 2 3)
-
-

In addition to providing the Sexp module, most of the -base types in Base and Core support conversion to and from -s-expressions. For example, we can use the conversion functions defined -in the respective modules for integers, strings, and exceptions:

+
+

In addition to providing the Sexp module, most of the base types in Base and Core support conversion to and from s-expressions. For example, we can use the conversion functions defined in the respective modules for integers, strings, and exceptions:

Int.sexp_of_t 3;;
 >- : Sexp.t = 3
@@ -106,50 +74,35 @@ 
Base, Core, and Parsexp

 >- : Sexp.t = (Invalid_argument foo)
-

It’s also possible to convert container types such as lists or arrays -that are polymorphic over the type of data they contain.

+

It’s also possible to convert container types such as lists or arrays that are polymorphic over the type of data they contain.

#show List.sexp_of_t;;
 >val sexp_of_t : ('a -> Sexp.t) -> 'a list -> Sexp.t
-

Notice that List.sexp_of_t is polymorphic and takes as -its first argument another conversion function to handle the elements of -the list to be converted. Base and Core use this scheme more generally -for defining sexp converters for polymorphic types. Here’s an example of -it in action.

+

Notice that List.sexp_of_t is polymorphic and takes as its first argument another conversion function to handle the elements of the list to be converted. Base and Core use this scheme more generally for defining sexp converters for polymorphic types. Here’s an example of it in action.

List.sexp_of_t Int.sexp_of_t [1; 2; 3];;
 >- : Sexp.t = (1 2 3)
-

The functions that go in the other direction, i.e., -reconstruct an OCaml value from an s-expression, use essentially the -same trick for handling polymorphic types, as shown below.

+

The functions that go in the other direction, i.e., reconstruct an OCaml value from an s-expression, use essentially the same trick for handling polymorphic types, as shown below.

List.t_of_sexp Int.t_of_sexp (Sexp.of_string "(1 2 3)");;
 >- : int list = [1; 2; 3]
-

Such a function will fail with an exception when presented with an -s-expression that doesn’t match the structure of the OCaml type in -question.

+

Such a function will fail with an exception when presented with an s-expression that doesn’t match the structure of the OCaml type in question.

List.t_of_sexp Int.t_of_sexp (Sexp.of_string "(1 2 three)");;
 >Exception:
 >(Of_sexp_error "int_of_sexp: (Failure int_of_string)" (invalid_sexp three))
-
-

More on Top-Level Printing

-

The values of the s-expressions that we created were printed properly -as s-expressions in the toplevel, instead of as the tree of -Atom and List variants that they’re actually -made of.  

-

This is due to OCaml’s facility for installing custom top-level -printers that can rewrite some values into more top-level-friendly -equivalents. They are generally installed as ocamlfind -packages ending in top:

+
+

More on Top-Level Printing

+

The values of the s-expressions that we created were printed properly as s-expressions in the toplevel, instead of as the tree of Atom and List variants that they’re actually made of.  

+

This is due to OCaml’s facility for installing custom top-level printers that can rewrite some values into more top-level-friendly equivalents. They are generally installed as ocamlfind packages ending in top:

ocamlfind list | grep top
 >astring.top         (version: 0.8.3)
@@ -161,17 +114,11 @@ 
More on Top-Level Printing

 >utop                (version: 2.1.0)
-

The core.top package (which you should have loaded by -default in your .ocamlinit file) loads in printers for the -Core extensions already, so you don’t need to do anything special to use -the s-expression printer.

-
+

The core.top package (which you should have loaded by default in your .ocamlinit file) loads in printers for the Core extensions already, so you don’t need to do anything special to use the s-expression printer.

+

S-Expression Converters for New Types

-

But what if you want a function to convert a brand new type to an -s-expression? You can of course write it yourself manually. Here’s an -example.  

+

But what if you want a function to convert a brand new type to an s-expression? You can of course write it yourself manually. Here’s an example.  

type t = { foo: int; bar: float };;
 >type t = { foo : int; bar : float; }
@@ -184,19 +131,8 @@ 
S-Expression Converters for New Types

 >- : Sexp.t = ((foo 3) (bar -5.5))
-

This is somewhat tiresome to write, and it gets more so when you -consider the parser, i.e., t_of_sexp, which is considerably -more complex. Writing this kind of parsing and printing code by hand is -mechanical and error prone, not to mention a drag.

-

Given how mechanical the code is, you could imagine writing a program -that inspects the type definition and automatically generates the -conversion code for you. As it turns out, there’s a syntax -extension called ppx_sexp_conv which does just that, -creating the required functions for every type annotated with -[@@deriving sexp]. To enable ppx_sexp_conv, -we’re going to enable ppx_jane, which is a larger -collection of useful extensions that includes -ppx_sexp_conv.    

+

This is somewhat tiresome to write, and it gets more so when you consider the parser, i.e., t_of_sexp, which is considerably more complex. Writing this kind of parsing and printing code by hand is mechanical and error prone, not to mention a drag.

+

Given how mechanical the code is, you could imagine writing a program that inspects the type definition and automatically generates the conversion code for you. As it turns out, there’s a syntax extension called ppx_sexp_conv which does just that, creating the required functions for every type annotated with [@@deriving sexp]. To enable ppx_sexp_conv, we’re going to enable ppx_jane, which is a larger collection of useful extensions that includes ppx_sexp_conv.    

#require "ppx_jane";;
@@ -211,18 +147,13 @@

S-Expression Converters for New Types

>- : t = {foo = 3; bar = 35.}
-

The syntax extension can be used outside of type declarations as -well. As discussed in Chapter 7, Error Handling, [@@deriving sexp] can -be attached to the declaration of an exception to improve the quality of -errors printed by OCaml’s top-level exception handler.

-

Here are two exception declarations, one with an annotation, and one -without:

+

The syntax extension can be used outside of type declarations as well. As discussed in Chapter 7, Error Handling, [@@deriving sexp] can be attached to the declaration of an exception to improve the quality of errors printed by OCaml’s top-level exception handler.

+

Here are two exception declarations, one with an annotation, and one without:

exception Ordinary_exn of string list;;
 exception Exn_with_sexp of string list [@@deriving sexp];;
-

And here’s the difference in what you see when you throw these -exceptions.

+

And here’s the difference in what you see when you throw these exceptions.

raise (Ordinary_exn ["1";"2";"3"]);;
 >Exception: Ordinary_exn(_).
@@ -230,8 +161,7 @@ 
S-Expression Converters for New Types

 >Exception: (//toplevel//.Exn_with_sexp (1 2 3))
-

ppx_sexp_conv also supports inline declarations that -generate converters for anonymous types.

+

ppx_sexp_conv also supports inline declarations that generate converters for anonymous types.

[%sexp_of: int * string ];;
 >- : int * string -> Sexp.t = <fun>
@@ -239,54 +169,25 @@ 
S-Expression Converters for New Types

 >- : Sexp.t = (3 foo)
-

The syntax extensions bundled with Base and Core almost all have the -same basic structure: they auto-generate code based on type definitions, -implementing functionality that you could in theory have implemented by -hand, but with far less programmer effort.

-
-

Syntax Extensions and PPX

-

OCaml doesn’t directly support deriving s-expression converters from -type definitions. Instead, it provides a mechanism called PPX -which allows you to add to the compilation pipeline code for -transforming OCaml programs at the syntactic level, via the --ppx compiler flag.

-

PPXs operate on OCaml’s abstract syntax tree, or AST, which -is a data type that represents the syntax of a well-formed OCaml -program. Annotations like [%sexp_of: int] or -[@@deriving sexp] are part of special extensions to the -syntax, called extension points, which were added to the -language to give a place to put information that would be consumed by -syntax extensions like ppx_sexp_conv.   

-

ppx_sexp_conv is part of a family of syntax extensions, -including ppx_compare, described in Chapter 14, Maps And Hash Tables, and ppx_fields, -described in Chapter 5, Records, that generate code based on type -declarations.   -     

-

Using these extensions from a dune file is as simple as -adding this directive to a (library) or -(executable) stanza to indicate that the files should be -run through a preprocessor:

+

The syntax extensions bundled with Base and Core almost all have the same basic structure: they auto-generate code based on type definitions, implementing functionality that you could in theory have implemented by hand, but with far less programmer effort.

+
+

Syntax Extensions and PPX

+

OCaml doesn’t directly support deriving s-expression converters from type definitions. Instead, it provides a mechanism called PPX which allows you to add to the compilation pipeline code for transforming OCaml programs at the syntactic level, via the -ppx compiler flag.

+

PPXs operate on OCaml’s abstract syntax tree, or AST, which is a data type that represents the syntax of a well-formed OCaml program. Annotations like [%sexp_of: int] or [@@deriving sexp] are part of special extensions to the syntax, called extension points, which were added to the language to give a place to put information that would be consumed by syntax extensions like ppx_sexp_conv.   

+

ppx_sexp_conv is part of a family of syntax extensions, including ppx_compare, described in Chapter 14, Maps And Hash Tables, and ppx_fields, described in Chapter 5, Records, that generate code based on type declarations.        

+

Using these extensions from a dune file is as simple as adding this directive to a (library) or (executable) stanza to indicate that the files should be run through a preprocessor:

(executable
   (name hello)
   (preprocess (pps ppx_sexp_conv))
 )
-
+

The Sexp Format

-

The textual representation of s-expressions is pretty -straightforward. An s-expression is written down as a nested -parenthetical expression, with whitespace-separated strings as the -atoms. Quotes are used for atoms that contain parentheses or spaces -themselves; backslash is the escape character; and semicolons are used -to introduce single-line comments. Thus, the following file, -example.scm:  

+

The textual representation of s-expressions is pretty straightforward. An s-expression is written down as a nested parenthetical expression, with whitespace-separated strings as the atoms. Quotes are used for atoms that contain parentheses or spaces themselves; backslash is the escape character; and semicolons are used to introduce single-line comments. Thus, the following file, example.scm:  

((foo 3.3) ;; This is a comment
  (bar "this is () an \" atom"))
@@ -297,21 +198,17 @@

The Sexp Format

>- : Sexp.t = ((foo 3.3) (bar "this is () an \" atom"))
-

As you can see, the comment is not part of the loaded -s-expression.

+

As you can see, the comment is not part of the loaded s-expression.

All in, the s-expression format supports three comment syntaxes:

;
-
-Comments out everything to the end of line +
Comments out everything to the end of line
#|,|#
-
-Delimiters for commenting out a block +
Delimiters for commenting out a block
#;
-
-Comments out the first complete s-expression that follows +
Comments out the first complete s-expression that follows

The following example shows all of these in action:

@@ -334,10 +231,7 @@

The Sexp Format

>- : Sexp.t = ((this is included) (this stays) (and now we're done)) -

If we introduce an error into our s-expression, by, say, creating a -file broken_example.scm which is example.scm, -without the open-paren in front of bar, we’ll get a parse -error:

+

If we introduce an error into our s-expression, by, say, creating a file broken_example.scm which is example.scm, without the open-paren in front of bar, we’ll get a parse error:

Sexp.load_sexp "example_broken.scm";;
 >Exception:
@@ -349,20 +243,9 @@ 
The Sexp Format

Preserving Invariants

-

Modules and module interfaces are an important part of how OCaml code -is structured and designed. One of the key reasons we use module -interfaces is to make it possible to enforce invariants. In particular, -by restricting how values of a given type can be created and -transformed, interfaces let you enforce various rules, including -ensuring that your data is well-formed.  

-

When you add s-expression converters (or really any deserializer) to -an API, you’re adding an alternate path for creating values, and if -you’re not careful, that alternate path can violate the carefully -maintained invariants of your code.

-

In the following, we’ll show how this problem can crop up, and how to -resolve it. Let’s consider a module Int_interval for -representing closed integer intervals, similar to the one described in -Chapter 10, Functors.

+

Modules and module interfaces are an important part of how OCaml code is structured and designed. One of the key reasons we use module interfaces is to make it possible to enforce invariants. In particular, by restricting how values of a given type can be created and transformed, interfaces let you enforce various rules, including ensuring that your data is well-formed.  

+

When you add s-expression converters (or really any deserializer) to an API, you’re adding an alternate path for creating values, and if you’re not careful, that alternate path can violate the carefully maintained invariants of your code.

+

In the following, we’ll show how this problem can crop up, and how to resolve it. Let’s consider a module Int_interval for representing closed integer intervals, similar to the one described in Chapter 10, Functors.

Here’s the signature.

type t [@@deriving sexp]
@@ -374,11 +257,7 @@ 
Preserving Invariants

 val is_empty : t -> bool
 val contains : t -> int -> bool
-

In addition to basic operations for creating and evaluating -intervals, this interface also exposes s-expression converters. Note -that the [@@deriving sexp] syntax works in a signature as -well, but in this case, it just adds the signature for the conversion -functions, not the implementation.

+

In addition to basic operations for creating and evaluating intervals, this interface also exposes s-expression converters. Note that the [@@deriving sexp] syntax works in a signature as well, but in this case, it just adds the signature for the conversion functions, not the implementation.

Here’s the implementation of Int_interval.

open Core
@@ -400,14 +279,8 @@ 
Preserving Invariants

   | Empty -> false
   | Range (low, high) -> x >= low && x <= high
-

One critical invariant here is that Range is only used -to represent non-empty intervals. A call to create with a -lower bound above the upper bound will return an Empty.

-

Now, let’s demonstrate the functionality with some tests, using the -expect test framework described in Chapter 17, Testing. First, we’ll write a test helper that -takes an interval and a list of points, and prints out the result of -checking for emptiness, and a classification of which points are inside -and outside the interval.

+

One critical invariant here is that Range is only used to represent non-empty intervals. A call to create with a lower bound above the upper bound will return an Empty.

+

Now, let’s demonstrate the functionality with some tests, using the expect test framework described in Chapter 17, Testing. First, we’ll write a test helper that takes an interval and a list of points, and prints out the result of checking for emptiness, and a classification of which points are inside and outside the interval.

let test_interval i points =
   let in_, out =
@@ -443,12 +316,8 @@ 
Preserving Invariants

     in:
     out: 1, 2, 3, 4, 5, 6, 7, 8, 9 |}]
-

Note that the result of checking is_empty lines up with -the test of what elements are contained and not contained in the -interval.

-

Now, let’s test out the s-expression converters, starting with -sexp_of_t. This test lets you see that a flipped-bounds -interval is represented by Empty.

+

Note that the result of checking is_empty lines up with the test of what elements are contained and not contained in the interval.

+

Now, let’s test out the s-expression converters, starting with sexp_of_t. This test lets you see that a flipped-bounds interval is represented by Empty.

let%expect_test "test to_sexp" =
   let t lo hi =
@@ -462,12 +331,7 @@ 
Preserving Invariants

   t 6 3;
   [%expect {| Empty |}]
-

The next thing to check is the t_of_sexp converters, and -here, we run into a problem. In particular, consider what would happen -if we create an interval from the s-expression (Range 6 3). -That’s an s-expression that shouldn’t ever be generated by the library, -since intervals should never have swapped bounds. But there’s nothing to -stop us from generating that s-expression by hand.

+

The next thing to check is the t_of_sexp converters, and here, we run into a problem. In particular, consider what would happen if we create an interval from the s-expression (Range 6 3). That’s an s-expression that shouldn’t ever be generated by the library, since intervals should never have swapped bounds. But there’s nothing to stop us from generating that s-expression by hand.

let%expect_test "test (range 6 3)" =
   let i = Int_interval.t_of_sexp (Sexp.of_string "(Range 6 3)") in
@@ -478,27 +342,15 @@ 
Preserving Invariants

     in:
     out: 1, 2, 3, 4, 5, 6, 7, 8, 9 |}]
-

You can see something bad has happened, since this interval is -detected as non-empty, but doesn’t appear to contain anything. The -problem traces back to the fact that t_of_sexp doesn’t -check the same invariant that create does. We can fix this, -by overriding the auto-generated s-expression converter with one that -checks the invariant, in this case, by calling create.

+

You can see something bad has happened, since this interval is detected as non-empty, but doesn’t appear to contain anything. The problem traces back to the fact that t_of_sexp doesn’t check the same invariant that create does. We can fix this, by overriding the auto-generated s-expression converter with one that checks the invariant, in this case, by calling create.

let t_of_sexp sexp =
   match t_of_sexp sexp with
   | Empty -> Empty
   | Range (x, y) -> create x y
-

Overriding an existing function definition with a new one is -perfectly acceptable in OCaml. Since t_of_sexp is defined -with an ordinary let rather than a let rec, -the call to the t_of_sexp goes to the derived version of -the function, rather than being a recursive call.

-

Note that, rather than fixing up the invariant, we could have instead -thrown an exception if the invariant was violated. In any case, the -approach we took means that rerunning our test produces a more -consistent and sensible result.

+

Overriding an existing function definition with a new one is perfectly acceptable in OCaml. Since t_of_sexp is defined with an ordinary let rather than a let rec, the call to the t_of_sexp goes to the derived version of the function, rather than being a recursive call.

+

Note that, rather than fixing up the invariant, we could have instead thrown an exception if the invariant was violated. In any case, the approach we took means that rerunning our test produces a more consistent and sensible result.

let%expect_test "test (range 6 3)" =
   let i = Int_interval.t_of_sexp (Sexp.of_string "(Range 6 3)") in
@@ -512,12 +364,7 @@ 
Preserving Invariants

Getting Good Error Messages

-

There are two steps to deserializing a type from an s-expression: -first, converting the bytes in a file to an s-expression; and the -second, converting that s-expression into the type in question. One -problem with this is that it can be hard to localize errors to the right -place using this scheme. Consider the following example.    

+

There are two steps to deserializing a type from an s-expression: first, converting the bytes in a file to an s-expression; and the second, converting that s-expression into the type in question. One problem with this is that it can be hard to localize errors to the right place using this scheme. Consider the following example.    

open Core
 
@@ -538,9 +385,7 @@ 
Getting Good Error Messages

  (b not-a-string)
  (c 1.0))
-

you’ll get the following error. (Note that we set the -OCAMLRUNPARAM environment variable to suppress the stack -trace here.)

+

you’ll get the following error. (Note that we set the OCAMLRUNPARAM environment variable to suppress the stack trace here.)

OCAMLRUNPARAM=b=0 dune exec -- ./read_foo.exe
 >Uncaught exception:
@@ -550,12 +395,8 @@ 
Getting Good Error Messages

 >
 [2]
-

If all you have is the error message and the string, it’s not -terribly informative. In particular, you know that the parsing errored -out on the atom “not-an-integer,” but you don’t know which one! In a -large file, this kind of bad error message can be pure misery.

-

But there’s hope! We can make a small change to the code to improve -the error message greatly:

+

If all you have is the error message and the string, it’s not terribly informative. In particular, you know that the parsing errored out on the atom “not-an-integer,” but you don’t know which one! In a large file, this kind of bad error message can be pure misery.

+

But there’s hope! We can make a small change to the code to improve the error message greatly:

open Core
 
@@ -580,34 +421,15 @@ 
Getting Good Error Messages

 >
 [2]
-

Here, example.scm:2:5 tells us that the error occurred -in the file "example.scm" on line 2, character 5. This is a -much better start for figuring out what went wrong. The ability to find -the precise location of the error depends on the sexp converter -reporting errors using the function of_sexp_error. This is -already done by converters generated by ppx_sexp_conv, but -you should make sure to do the same when you write custom -converters.

+

Here, example.scm:2:5 tells us that the error occurred in the file "example.scm" on line 2, character 5. This is a much better start for figuring out what went wrong. The ability to find the precise location of the error depends on the sexp converter reporting errors using the function of_sexp_error. This is already done by converters generated by ppx_sexp_conv, but you should make sure to do the same when you write custom converters.

Sexp-Conversion Directives

-

ppx_sexp_conv supports a collection of directives for -modifying the default behavior of the auto-generated sexp converters. -These directives allow you to customize the way in which types are -represented as s-expressions without having to write a custom converter. - 

+

ppx_sexp_conv supports a collection of directives for modifying the default behavior of the auto-generated sexp converters. These directives allow you to customize the way in which types are represented as s-expressions without having to write a custom converter.  

@sexp.opaque

-

The most commonly used directive is [@sexp_opaque], -whose purpose is to mark a given component of a type as being -unconvertible. Anything marked with the [@sexp.opaque] -attribute will be presented as the atom <opaque> by -the to-sexp converter, and will trigger an exception from the from-sexp -converter.  

-

Note that the type of a component marked as opaque doesn’t need to -have a sexp converter defined. By default, if we define a type without a -sexp converter and then try to use it as part of another type with a -sexp converter, we’ll error out:

+

The most commonly used directive is [@sexp_opaque], whose purpose is to mark a given component of a type as being unconvertible. Anything marked with the [@sexp.opaque] attribute will be presented as the atom <opaque> by the to-sexp converter, and will trigger an exception from the from-sexp converter.  

+

Note that the type of a component marked as opaque doesn’t need to have a sexp converter defined. By default, if we define a type without a sexp converter and then try to use it as part of another type with a sexp converter, we’ll error out:

type no_converter = int * int;;
 >type no_converter = int * int
@@ -616,24 +438,20 @@ 
@sexp.opaque

 >Error: Unbound value no_converter_of_sexp
-

But with [@sexp.opaque], we can embed our opaque -no_converter type within the other data structure without -an error.

+

But with [@sexp.opaque], we can embed our opaque no_converter type within the other data structure without an error.

type t =
   { a: (no_converter [@sexp.opaque]);
     b: string
   } [@@deriving sexp];;
-

And if we now convert a value of this type to an s-expression, we’ll -see the contents of field a marked as opaque:

+

And if we now convert a value of this type to an s-expression, we’ll see the contents of field a marked as opaque:

sexp_of_t { a = (3,4); b = "foo" };;
 >- : Sexp.t = ((a <opaque>) (b foo))
-

Note that t_of_sexp is still generated, but will fail at -runtime when called.

+

Note that t_of_sexp is still generated, but will fail at runtime when called.

t_of_sexp (Sexp.of_string "((a whatever) (b foo))");;
 >Exception:
@@ -641,32 +459,24 @@ 
@sexp.opaque

 >  (invalid_sexp whatever))
-

It might seem perverse to create a parser for a type containing a -[@sexp.opaque] value, but it’s not as useless as it seems. -In particular, such a converter won’t necessarily fail on all inputs. -Consider a record containing a list of opaque values:

+

It might seem perverse to create a parser for a type containing a [@sexp.opaque] value, but it’s not as useless as it seems. In particular, such a converter won’t necessarily fail on all inputs. Consider a record containing a list of opaque values:

type t =
   { a: (no_converter [@sexp.opaque]) list;
     b: string
   } [@@deriving sexp];;
-

The t_of_sexp function can still succeed, as long as -the list is empty.

+

The t_of_sexp function can still succeed, as long as the list is empty.

t_of_sexp (Sexp.of_string "((a ()) (b foo))");;
 >- : t = {a = []; b = "foo"}
-

Sometimes, though, one or other of the converters is useless, and you -want to explicitly choose what to generate. You can do that by using -[@@deriving sexp_of] or [@@deriving of_sexp] -instead of [@@deriving sexp].

+

Sometimes, though, one or other of the converters is useless, and you want to explicitly choose what to generate. You can do that by using [@@deriving sexp_of] or [@@deriving of_sexp] instead of [@@deriving sexp].

@sexp.list

-

Sometimes, sexp converters have more parentheses than one would -ideally like. Consider the following variant type.  

+

Sometimes, sexp converters have more parentheses than one would ideally like. Consider the following variant type.  

type compatible_versions =
   | Specific of string list
@@ -680,9 +490,7 @@ 
@sexp.list

 >- : Sexp.t = (Specific (3.12.0 3.12.1 3.13.0))
-

The set of parens around the list of versions is arguably excessive. -We can drop those parens using the [@sexp.list] -directive.

+

The set of parens around the list of versions is arguably excessive. We can drop those parens using the [@sexp.list] directive.

type compatible_versions =
   | Specific of string list [@sexp.list]
@@ -697,9 +505,7 @@ 
@sexp.list

@sexp.option

-

By default, optional values are represented either as () -for None. Here’s an example of a record type containing an -option:  

+

By default, optional values are represented either as () for None. Here’s an example of a record type containing an option:  

type t =
   { a: int option;
@@ -714,20 +520,14 @@ 
@sexp.option

 >- : Sexp.t = ((a (3)) (b hello))
-

This all works as you might expect, but in the context of a record, -you might want a different behavior, which is to make the field itself -optional. The [@sexp.option] directive gives you just -that.

+

This all works as you might expect, but in the context of a record, you might want a different behavior, which is to make the field itself optional. The [@sexp.option] directive gives you just that.

type t =
   { a: int option [@sexp.option];
     b: string;
   } [@@deriving sexp]
-

And here is the new syntax. Note that when the value of -a is Some, it shows up in the s-expression -unadorned, and when it’s None, the entire record field is -omitted.

+

And here is the new syntax. Note that when the value of a is Some, it shows up in the s-expression unadorned, and when it’s None, the entire record field is omitted.

sexp_of_t { a = Some 3; b = "hello" };;
 >- : Sexp.t = ((a 3) (b hello))
@@ -738,11 +538,8 @@ 
@sexp.option

Specifying Defaults

-

[@sexp.option] gives you a way of interpreting the -s-expression for a record where some of the fields are left unspecified. -The [@default] directive provides another.    

-

Consider the following type, which represents the configuration of a -very simple web server:

+

[@sexp.option] gives you a way of interpreting the s-expression for a record where some of the fields are left unspecified. The [@default] directive provides another.    

+

Consider the following type, which represents the configuration of a very simple web server:

type http_server_config = {
   web_root: string;
@@ -750,9 +547,7 @@ 
Specifying Defaults

   addr: string;
 } [@@deriving sexp];;
-

One could imagine making some of these parameters optional; in -particular, by default, we might want the web server to bind to port 80, -and to listen as localhost. We can do this as follows:

+

One could imagine making some of these parameters optional; in particular, by default, we might want the web server to bind to port 80, and to listen as localhost. We can do this as follows:

type http_server_config = {
   web_root: string;
@@ -760,9 +555,7 @@ 
Specifying Defaults

   addr: string [@default "localhost"];
 } [@@deriving sexp];;
-

Now, if we try to convert an s-expression that specifies only the -web_root, we’ll see that the other values are filled in -with the desired defaults:

+

Now, if we try to convert an s-expression that specifies only the web_root, we’ll see that the other values are filled in with the desired defaults:

let cfg =
   "((web_root /var/www/html))"
@@ -772,16 +565,13 @@ 
Specifying Defaults

 >  {web_root = "/var/www/html"; port = 80; addr = "localhost"}
-

If we convert the configuration back out to an s-expression, you’ll -notice that all of the fields are present, even though they’re not -strictly necessary:

+

If we convert the configuration back out to an s-expression, you’ll notice that all of the fields are present, even though they’re not strictly necessary:

sexp_of_http_server_config cfg;;
 >- : Sexp.t = ((web_root /var/www/html) (port 80) (addr localhost))
-

We could make the generated s-expression also drop default values, by -using the [@sexp_drop_default] directive:  

+

We could make the generated s-expression also drop default values, by using the [@sexp_drop_default] directive:  

type http_server_config = {
   web_root: string;
@@ -801,10 +591,7 @@ 
Specifying Defaults

 >- : Sexp.t = ((web_root /var/www/html))
-

As you can see, the fields that are at their default values are -omitted from the generated s-expression. On the other hand, if we -convert a config with non-default values, they will show up in the -generated s-expression.

+

As you can see, the fields that are at their default values are omitted from the generated s-expression. On the other hand, if we convert a config with non-default values, they will show up in the generated s-expression.

sexp_of_http_server_config { cfg with port = 8080 };;
 >- : Sexp.t = ((web_root /var/www/html) (port 8080))
@@ -813,27 +600,16 @@ 
Specifying Defaults

 >- : Sexp.t = ((web_root /var/www/html) (port 8080) (addr 192.168.0.1))
-

This can be very useful in designing config file formats that are -both reasonably terse and easy to generate and maintain. It can also be -useful for backwards compatibility: if you add a new field to your -config record but make that field optional, then you should still be -able to parse older versions of your config.

-

The exact attribute you use depends on the comparison functions -available over the type that you wish to drop:

+

This can be very useful in designing config file formats that are both reasonably terse and easy to generate and maintain. It can also be useful for backwards compatibility: if you add a new field to your config record but make that field optional, then you should still be able to parse older versions of your config.

+

The exact attribute you use depends on the comparison functions available over the type that you wish to drop:

    -
  • [@sexp_drop_default.compare] if the type supports -[%compare]
    • -
  • [@sexp_drop_default.equal] if the type supports -[%equal]
    • -
  • [@sexp_drop_default.sexp] if you want to compare the -sexp representations
    • -
  • [@sexp_drop_default f] and give an explicit equality -function
    • +
  • [@sexp_drop_default.compare] if the type supports [%compare]
    • +
  • [@sexp_drop_default.equal] if the type supports [%equal]
    • +
  • [@sexp_drop_default.sexp] if you want to compare the sexp representations
    • +
  • [@sexp_drop_default f] and give an explicit equality function
-

Most of the type definitions supplied with Base and Core provide the -comparison and equality operations, so those are reasonable default -attributes to use.

+

Most of the type definitions supplied with Base and Core provide the comparison and equality operations, so those are reasonable default attributes to use.

-

Next: Chapter 21The OCaml Platform

 \ No newline at end of file +

Next: Chapter 21The OCaml Platform

 \ No newline at end of file diff --git a/error-handling.html b/error-handling.html index 47c12989d..5a63b067d 100644 --- a/error-handling.html +++ b/error-handling.html @@ -1,30 +1,18 @@ -Error Handling - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Error Handling - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Error Handling

-

Nobody likes dealing with errors. It’s tedious, it’s easy to get -wrong, and it’s usually just not as fun as thinking about how your -program is going to succeed. But error handling is important, and -however much you don’t like thinking about it, having your software fail -due to poor error handling is worse.

-

Thankfully, OCaml has powerful tools for handling errors reliably and -with a minimum of pain. In this chapter we’ll discuss some of the -different approaches in OCaml to handling errors, and give some advice -on how to design interfaces that make error handling easier.

-

We’ll start by describing the two basic approaches for reporting -errors in OCaml: error-aware return types and exceptions.   

+

Nobody likes dealing with errors. It’s tedious, it’s easy to get wrong, and it’s usually just not as fun as thinking about how your program is going to succeed. But error handling is important, and however much you don’t like thinking about it, having your software fail due to poor error handling is worse.

+

Thankfully, OCaml has powerful tools for handling errors reliably and with a minimum of pain. In this chapter we’ll discuss some of the different approaches in OCaml to handling errors, and give some advice on how to design interfaces that make error handling easier.

+

We’ll start by describing the two basic approaches for reporting errors in OCaml: error-aware return types and exceptions.   

Error-Aware Return Types

-

The best way in OCaml to signal an error is to include that error in -your return value. Consider the type of the find function -in the List module:

+

The best way in OCaml to signal an error is to include that error in your return value. Consider the type of the find function in the List module:

open Base;;
 List.find;;
 >- : 'a list -> f:('a -> bool) -> 'a option = <fun>
-

The option in the return type indicates that the function may not -succeed in finding a suitable element:

+

The option in the return type indicates that the function may not succeed in finding a suitable element:

List.find [1;2;3] ~f:(fun x -> x >= 2);;
 >- : int option = Some 2
@@ -32,15 +20,8 @@ 
Error-Aware Return Types

 >- : int option = None
-

Including errors in the return values of your functions requires the -caller to handle the error explicitly, allowing the caller to make the -choice of whether to recover from the error or propagate it onward.

-

Consider the compute_bounds function below, which takes -a list and a comparison function and returns upper and lower bounds for -the list by finding the smallest and largest element on the list. -List.hd and List.last, which return -None when they encounter an empty list, are used to extract -the largest and smallest element of the list:

+

Including errors in the return values of your functions requires the caller to handle the error explicitly, allowing the caller to make the choice of whether to recover from the error or propagate it onward.

+

Consider the compute_bounds function below, which takes a list and a comparison function and returns upper and lower bounds for the list by finding the smallest and largest element on the list. List.hd and List.last, which return None when they encounter an empty list, are used to extract the largest and smallest element of the list:

let compute_bounds ~compare list =
   let sorted = List.sort ~compare list in
@@ -51,15 +32,8 @@ 
Error-Aware Return Types

 >  <fun>
-

The match expression is used to handle the error cases, -propagating a None in hd or last -into the return value of compute_bounds.

-

On the other hand, in the find_mismatches that follows, -errors encountered during the computation do not propagate to the return -value of the function. find_mismatches takes two hash -tables as arguments and searches for keys that have different data in -one table than in the other. As such, the failure to find a key in one -table isn’t a failure of any sort:

+

The match expression is used to handle the error cases, propagating a None in hd or last into the return value of compute_bounds.

+

On the other hand, in the find_mismatches that follows, errors encountered during the computation do not propagate to the return value of the function. find_mismatches takes two hash tables as arguments and searches for keys that have different data in one table than in the other. As such, the failure to find a key in one table isn’t a failure of any sort:

let find_mismatches table1 table2 =
   Hashtbl.fold table1 ~init:[] ~f:(fun ~key ~data mismatches ->
@@ -71,30 +45,18 @@ 
Error-Aware Return Types

 >  ('a, int) Hashtbl.Poly.t -> ('a, int) Hashtbl.Poly.t -> 'a list = <fun>
-

The use of options to encode errors underlines the fact that it’s not -clear whether a particular outcome, like not finding something on a -list, is an error or is just another valid outcome. This depends on the -larger context of your program, and thus is not something that a -general-purpose library can know in advance. One of the advantages of -error-aware return types is that they work well in both situations.

+

The use of options to encode errors underlines the fact that it’s not clear whether a particular outcome, like not finding something on a list, is an error or is just another valid outcome. This depends on the larger context of your program, and thus is not something that a general-purpose library can know in advance. One of the advantages of error-aware return types is that they work well in both situations.

Encoding Errors with Result

-

Options aren’t always a sufficiently expressive way to report errors. -Specifically, when you encode an error as None, there’s -nowhere to say anything about the nature of the error.

-

Result.t is meant to address this deficiency. The type -is defined as follows: 

+

Options aren’t always a sufficiently expressive way to report errors. Specifically, when you encode an error as None, there’s nowhere to say anything about the nature of the error.

+

Result.t is meant to address this deficiency. The type is defined as follows: 

module Result : sig
    type ('a,'b) t = | Ok of 'a
                     | Error of 'b
 end
-

A Result.t is essentially an option augmented with the -ability to store other information in the error case. Like -Some and None for options, the constructors -Ok and Error are available at the toplevel. As -such, we can write:

+

A Result.t is essentially an option augmented with the ability to store other information in the error case. Like Some and None for options, the constructors Ok and Error are available at the toplevel. As such, we can write:

[ Ok 3; Error "abject failure"; Ok 4 ];;
 >- : (int, string) result list = [Ok 3; Error "abject failure"; Ok 4]
@@ -104,32 +66,21 @@ 
Encoding Errors with Result

Error and Or_error

-

Result.t gives you complete freedom to choose the type -of value you use to represent errors, but it’s often useful to -standardize on an error type. Among other things, this makes it easier -to write utility functions to automate common error handling patterns. - 

-

But which type to choose? Is it better to represent errors as -strings? Some more structured representation like XML? Or something else -entirely?

-

Base’s answer to this question is the Error.t type. You -can, for example, construct one from a string.

+

Result.t gives you complete freedom to choose the type of value you use to represent errors, but it’s often useful to standardize on an error type. Among other things, this makes it easier to write utility functions to automate common error handling patterns.  

+

But which type to choose? Is it better to represent errors as strings? Some more structured representation like XML? Or something else entirely?

+

Base’s answer to this question is the Error.t type. You can, for example, construct one from a string.

Error.of_string "something went wrong";;
 >- : Error.t = something went wrong
-

An Or_error.t is simply a Result.t with -the error case specialized to the Error.t type. Here’s an -example.

+

An Or_error.t is simply a Result.t with the error case specialized to the Error.t type. Here’s an example.

Error (Error.of_string "failed!");;
 >- : ('a, Error.t) result = Error failed!
-

The Or_error module provides a bunch of useful operators -for constructing errors. For example, Or_error.try_with can -be used for catching exceptions from a computation.

+

The Or_error module provides a bunch of useful operators for constructing errors. For example, Or_error.try_with can be used for catching exceptions from a computation.

let float_of_string s =
   Or_error.try_with (fun () -> Float.of_string s);;
@@ -141,32 +92,18 @@ 
Error and Or_error

 >Base__.Result.Error (Invalid_argument "Float.of_string a.bc")
-

Perhaps the most common way to create Error.ts is using -s-expressions. An s-expression is a balanced parenthetical -expression where the leaves of the expressions are strings. Here’s a -simple example:  

+

Perhaps the most common way to create Error.ts is using s-expressions. An s-expression is a balanced parenthetical expression where the leaves of the expressions are strings. Here’s a simple example:  

(This (is an) (s expression))
-

S-expressions are supported by the Sexplib package that is -distributed with Base and is the most common serialization format used -in Base. Indeed, most types in Base come with built-in s-expression -converters.  

+

S-expressions are supported by the Sexplib package that is distributed with Base and is the most common serialization format used in Base. Indeed, most types in Base come with built-in s-expression converters.  

Error.create "Unexpected character" 'c' Char.sexp_of_t;;
 >- : Error.t = ("Unexpected character" c)
-

We’re not restricted to doing this kind of error reporting with -built-in types. As we’ll discuss in more detail in Chapter 20, Data Serialization With S-Expressions, Sexplib -comes with a syntax extension that can autogenerate sexp converters for -specific types. We can enable it in the toplevel with a -#require statement enabling ppx_jane, which is -a package that pulls in multiple different syntax extensions, including -ppx_sexp_value, the one we need here. (Because of technical -issues with the toplevel, we can’t easily enable these syntax extensions -individually.)        

+

We’re not restricted to doing this kind of error reporting with built-in types. As we’ll discuss in more detail in Chapter 20, Data Serialization With S-Expressions, Sexplib comes with a syntax extension that can autogenerate sexp converters for specific types. We can enable it in the toplevel with a #require statement enabling ppx_jane, which is a package that pulls in multiple different syntax extensions, including ppx_sexp_value, the one we need here. (Because of technical issues with the toplevel, we can’t easily enable these syntax extensions individually.)        

#require "ppx_jane";;
 Error.t_of_sexp
@@ -174,12 +111,7 @@ 
Error and Or_error

 >- : Error.t = ("List is too long" (1 2 3))
-

Error also supports operations for transforming errors. -For example, it’s often useful to augment an error with information -about the context of the error or to combine multiple errors together. -Error.tag and Error.of_list fulfill these -roles:   

+

Error also supports operations for transforming errors. For example, it’s often useful to augment an error with information about the context of the error or to combine multiple errors together. Error.tag and Error.of_list fulfill these roles:   

Error.tag
   (Error.of_list [ Error.of_string "Your tires were slashed";
@@ -189,10 +121,7 @@ 
Error and Or_error

 >("over the weekend" "Your tires were slashed" "Your windshield was smashed")
-

A very common way of generating errors is the %message -syntax extension, which provides a compact syntax for providing a string -describing the error, along with further values represented as -s-expressions. Here’s an example.

+

A very common way of generating errors is the %message syntax extension, which provides a compact syntax for providing a string describing the error, along with further values represented as s-expressions. Here’s an example.

let a = "foo" and b = ("foo",[3;4]);;
 >val a : string = "foo"
@@ -203,19 +132,11 @@ 
Error and Or_error

 >Base__.Result.Error ("Something went wrong" (a foo) (b (foo (3 4))))
-

This is the most common idiom for generating -Error.t’s.

+

This is the most common idiom for generating Error.t’s.

bind and Other Error Handling Idioms

-

As you write more error handling code in OCaml, you’ll discover that -certain patterns start to emerge. A number of these common patterns have -been codified by functions in modules like Option and -Result. One particularly useful pattern is built around the -function bind, which is both an ordinary function and an -infix operator >>=. Here’s the definition of -bind for options:  

+

As you write more error handling code in OCaml, you’ll discover that certain patterns start to emerge. A number of these common patterns have been codified by functions in modules like Option and Result. One particularly useful pattern is built around the function bind, which is both an ordinary function and an infix operator >>=. Here’s the definition of bind for options:  

let bind option ~f =
   match option with
@@ -224,13 +145,7 @@ 
bind and Other Error Handling Idioms

 >val bind : 'a option -> f:('a -> 'b option) -> 'b option = <fun>
-

As you can see, bind None f returns None -without calling f, and bind (Some x) ~f -returns f x. bind can be used as a way of -sequencing together error-producing functions so that the first one to -produce an error terminates the computation. Here’s a rewrite of -compute_bounds to use a nested series of -binds:

+

As you can see, bind None f returns None without calling f, and bind (Some x) ~f returns f x. bind can be used as a way of sequencing together error-producing functions so that the first one to produce an error terminates the computation. Here’s a rewrite of compute_bounds to use a nested series of binds:

let compute_bounds ~compare list =
   let sorted = List.sort ~compare list in
@@ -241,13 +156,7 @@ 
bind and Other Error Handling Idioms

 >  <fun>
-

The preceding code is a little bit hard to swallow, however, on a -syntactic level. We can make it easier to read and drop some of the -parentheses, by using the infix operator form of bind, -which we get access to by locally opening -Option.Monad_infix. The module is called -Monad_infix because the bind operator is part -of a subinterface called Monad, which we’ll see again in Chapter 16, Concurrent Programming With Async.

+

The preceding code is a little bit hard to swallow, however, on a syntactic level. We can make it easier to read and drop some of the parentheses, by using the infix operator form of bind, which we get access to by locally opening Option.Monad_infix. The module is called Monad_infix because the bind operator is part of a subinterface called Monad, which we’ll see again in Chapter 16, Concurrent Programming With Async.

let compute_bounds ~compare list =
   let open Option.Monad_infix in
@@ -259,17 +168,10 @@ 
bind and Other Error Handling Idioms

 >  <fun>
-

This use of bind isn’t really materially better than the -one we started with, and indeed, for small examples like this, direct -matching of options is generally better than using bind. -But for large, complex examples with many stages of error handling, the -bind idiom becomes clearer and easier to manage.

-
-

Monads and Let_syntax

-

We can make this look a little bit more ordinary by using a syntax -extension that’s designed specifically for monadic binds, called -Let_syntax. Here’s what the above example looks like using -this extension.

+

This use of bind isn’t really materially better than the one we started with, and indeed, for small examples like this, direct matching of options is generally better than using bind. But for large, complex examples with many stages of error handling, the bind idiom becomes clearer and easier to manage.

+
+

Monads and Let_syntax

+

We can make this look a little bit more ordinary by using a syntax extension that’s designed specifically for monadic binds, called Let_syntax. Here’s what the above example looks like using this extension.

#require "ppx_let";;
 let compute_bounds ~compare list =
@@ -282,22 +184,11 @@ 
Monads and Let_syntax

 >  <fun>
-

Note that we needed a #require statement to enable the -extension.

-

To understand what’s going on here, you need to know that -let%bind x = some_expr in some_other_expr is rewritten into -some_expr >>= fun x -> some_other_expr.

-

The advantage of Let_syntax is that it makes monadic -bind look more like a regular let-binding. This works nicely because you -can think of the monadic bind in this case as a special form of let -binding that has some built-in error handling semantics.

-
-

There are other useful idioms encoded in the functions in -Option. One example is Option.both, which -takes two optional values and produces a new optional pair that is -None if either of its arguments are None. -Using Option.both, we can make compute_bounds -even shorter:

+

Note that we needed a #require statement to enable the extension.

+

To understand what’s going on here, you need to know that let%bind x = some_expr in some_other_expr is rewritten into some_expr >>= fun x -> some_other_expr.

+

The advantage of Let_syntax is that it makes monadic bind look more like a regular let-binding. This works nicely because you can think of the monadic bind in this case as a special form of let binding that has some built-in error handling semantics.

+
+

There are other useful idioms encoded in the functions in Option. One example is Option.both, which takes two optional values and produces a new optional pair that is None if either of its arguments are None. Using Option.both, we can make compute_bounds even shorter:

let compute_bounds ~compare list =
   let sorted = List.sort ~compare list in
@@ -306,38 +197,25 @@ 
Monads and Let_syntax

 >  <fun>
-

These error-handling functions are valuable because they let you -express your error handling both explicitly and concisely. We’ve only -discussed these functions in the context of the Option -module, but more functionality of this kind can be found in the -Result and Or_error modules. 

+

These error-handling functions are valuable because they let you express your error handling both explicitly and concisely. We’ve only discussed these functions in the context of the Option module, but more functionality of this kind can be found in the Result and Or_error modules. 

Exceptions

-

Exceptions in OCaml are not that different from exceptions in many -other languages, like Java, C#, and Python. Exceptions are a way to -terminate a computation and report an error, while providing a mechanism -to catch and handle (and possibly recover from) exceptions that are -triggered by subcomputations.  

-

You can trigger an exception by, for example, dividing an integer by -zero:

+

Exceptions in OCaml are not that different from exceptions in many other languages, like Java, C#, and Python. Exceptions are a way to terminate a computation and report an error, while providing a mechanism to catch and handle (and possibly recover from) exceptions that are triggered by subcomputations.  

+

You can trigger an exception by, for example, dividing an integer by zero:

3 / 0;;
 >Exception: Division_by_zero.
-

And an exception can terminate a computation even if it happens -nested somewhere deep within it:

+

And an exception can terminate a computation even if it happens nested somewhere deep within it:

List.map ~f:(fun x -> 100 / x) [1;3;0;4];;
 >Exception: Division_by_zero.
-

If we put a printf in the middle of the computation, we -can see that List.map is interrupted partway through its -execution, never getting to the end of the list:

+

If we put a printf in the middle of the computation, we can see that List.map is interrupted partway through its execution, never getting to the end of the list:

List.map ~f:(fun x -> Stdio.printf "%d\n%!" x; 100 / x) [1;3;0;4];;
 >1
@@ -346,8 +224,7 @@ 
Exceptions

 >Exception: Division_by_zero.
-

In addition to built-in exceptions like Divide_by_zero, -OCaml lets you define your own:  

+

In addition to built-in exceptions like Divide_by_zero, OCaml lets you define your own:  

exception Key_not_found of string;;
 >exception Key_not_found of string
@@ -355,8 +232,7 @@ 
Exceptions

 >Exception: Key_not_found("a").
-

Exceptions are ordinary values and can be manipulated just like other -OCaml values:

+

Exceptions are ordinary values and can be manipulated just like other OCaml values:

let exceptions = [ Division_by_zero; Key_not_found "b" ];;
 >val exceptions : exn list = [Division_by_zero; Key_not_found("b")]
@@ -366,18 +242,8 @@ 
Exceptions

 >- : exn list = [Key_not_found("b")]
-

Exceptions are all of the same type, exn, which is -itself something of a special case in the OCaml type system. It is -similar to the variant types we encountered in Chapter 6, Variants, except that -it is open, meaning that it’s not fully defined in any one -place. In particular, new tags (specifically, new exceptions) can be -added to it by different parts of the program. This is in contrast to -ordinary variants, which are defined with a closed universe of available -tags. One result of this is that you can never have an exhaustive match -on an exn, since the full set of possible exceptions is not -known. 

-

The following function uses the Key_not_found exception -we defined above to signal an error:

+

Exceptions are all of the same type, exn, which is itself something of a special case in the OCaml type system. It is similar to the variant types we encountered in Chapter 6, Variants, except that it is open, meaning that it’s not fully defined in any one place. In particular, new tags (specifically, new exceptions) can be added to it by different parts of the program. This is in contrast to ordinary variants, which are defined with a closed universe of available tags. One result of this is that you can never have an exhaustive match on an exn, since the full set of possible exceptions is not known. 

+

The following function uses the Key_not_found exception we defined above to signal an error:

let rec find_exn alist key = match alist with
   | [] -> raise (Key_not_found key)
@@ -391,41 +257,24 @@ 
Exceptions

 >Exception: Key_not_found("c").
-

Note that we named the function find_exn to warn the -user that the function routinely throws exceptions, a convention that is -used heavily in Base.  

-

In the preceding example, raise throws the exception, -thus terminating the computation. The type of raise is a bit surprising -when you first see it: [raise}{.idx}

+

Note that we named the function find_exn to warn the user that the function routinely throws exceptions, a convention that is used heavily in Base.  

+

In the preceding example, raise throws the exception, thus terminating the computation. The type of raise is a bit surprising when you first see it: [raise}{.idx}

raise;;
 >- : exn -> 'a = <fun>
-

The return type of 'a makes it look like -raise manufactures a value to return that is completely -unconstrained in its type. That seems impossible, and it is. Really, -raise has a return type of 'a because it never -returns at all. This behavior isn’t restricted to functions like -raise that terminate by throwing exceptions. Here’s another -example of a function that doesn’t return a value:

+

The return type of 'a makes it look like raise manufactures a value to return that is completely unconstrained in its type. That seems impossible, and it is. Really, raise has a return type of 'a because it never returns at all. This behavior isn’t restricted to functions like raise that terminate by throwing exceptions. Here’s another example of a function that doesn’t return a value:

let rec forever () = forever ();;
 >val forever : unit -> 'a = <fun>
-

forever doesn’t return a value for a different reason: -it’s an infinite loop.

-

This all matters because it means that the return type of -raise can be whatever it needs to be to fit into the -context it is called in. Thus, the type system will let us throw an -exception anywhere in a program.

-
-

Declaring Exceptions Using [@@deriving sexp]

-

OCaml can’t always generate a useful textual representation of an -exception. For example:  

+

forever doesn’t return a value for a different reason: it’s an infinite loop.

+

This all matters because it means that the return type of raise can be whatever it needs to be to fit into the context it is called in. Thus, the type system will let us throw an exception anywhere in a program.

+
+

Declaring Exceptions Using [@@deriving sexp]

+

OCaml can’t always generate a useful textual representation of an exception. For example:  

type 'a bounds = { lower: 'a; upper: 'a };;
 >type 'a bounds = { lower : 'a; upper : 'a; }
@@ -435,9 +284,7 @@ 
Declaring Exceptions Using [@@deriving sexp]

 >- : exn = Crossed_bounds(_)
-

But if we declare the exception (and the types it depends on) using -[@@deriving sexp], we’ll get something with more -information:

+

But if we declare the exception (and the types it depends on) using [@@deriving sexp], we’ll get something with more information:

type 'a bounds = { lower: 'a; upper: 'a } [@@deriving sexp];;
 >type 'a bounds = { lower : 'a; upper : 'a; }
@@ -449,36 +296,19 @@ 
Declaring Exceptions Using [@@deriving sexp]

 >- : exn = (//toplevel//.Crossed_bounds ((lower 10) (upper 0)))
-

The period in front of Crossed_bounds is there because -the representation generated by [@@deriving sexp] includes -the full module path of the module where the exception in question is -defined. In this case, the string //toplevel// is used to -indicate that this was declared at the utop prompt, rather than in a -module.

-

This is all part of the support for s-expressions provided by the -Sexplib library and syntax extension, which is described in more detail -in Chapter 20, Data Serialization With S-Expressions.

-
+

The period in front of Crossed_bounds is there because the representation generated by [@@deriving sexp] includes the full module path of the module where the exception in question is defined. In this case, the string //toplevel// is used to indicate that this was declared at the utop prompt, rather than in a module.

+

This is all part of the support for s-expressions provided by the Sexplib library and syntax extension, which is described in more detail in Chapter 20, Data Serialization With S-Expressions.

+

Helper Functions for Throwing Exceptions

-

Base provides a number of helper functions to simplify the task of -throwing exceptions. The simplest one is failwith, which -could be defined as follows:   

+

Base provides a number of helper functions to simplify the task of throwing exceptions. The simplest one is failwith, which could be defined as follows:   

let failwith msg = raise (Failure msg);;
 >val failwith : string -> 'a = <fun>
-

There are several other useful functions for raising exceptions, -which can be found in the API documentation for the Common -and Exn modules in Base.

-

Another important way of throwing an exception is the -assert directive. assert is used for -situations where a violation of the condition in question indicates a -bug. Consider the following piece of code for zipping together two -lists: 

+

There are several other useful functions for raising exceptions, which can be found in the API documentation for the Common and Exn modules in Base.

+

Another important way of throwing an exception is the assert directive. assert is used for situations where a violation of the condition in question indicates a bug. Consider the following piece of code for zipping together two lists: 

let merge_lists xs ys ~f =
   if List.length xs <> List.length ys then None
@@ -498,13 +328,8 @@ 
Helper Functions for Throwing Exceptions

 >- : int list option = None
-

Here we use assert false, which means that the -assert, once reached, is guaranteed to trigger. In general, -one can put an arbitrary condition in the assertion.

-

In this case, the assert can never be triggered because -we have a check that makes sure that the lists are of the same length -before we call loop. If we change the code so that we drop -this test, then we can trigger the assert:

+

Here we use assert false, which means that the assert, once reached, is guaranteed to trigger. In general, one can put an arbitrary condition in the assertion.

+

In this case, the assert can never be triggered because we have a check that makes sure that the lists are of the same length before we call loop. If we change the code so that we drop this test, then we can trigger the assert:

let merge_lists xs ys ~f =
   let rec loop xs ys =
@@ -519,51 +344,25 @@ 
Helper Functions for Throwing Exceptions

 >Exception: "Assert_failure //toplevel//:6:14".
-

This shows what’s special about assert: it captures the -line number and character offset of the source location from which the -assertion was made.

+

This shows what’s special about assert: it captures the line number and character offset of the source location from which the assertion was made.

Exception Handlers

-

So far, we’ve only seen exceptions fully terminate the execution of a -computation. But sometimes, we want a program to be able to respond to -and recover from an exception. This is achieved through the use of -exception handlers.  

-

In OCaml, an exception handler is declared using a -try/with expression. Here’s the basic -syntax.

+

So far, we’ve only seen exceptions fully terminate the execution of a computation. But sometimes, we want a program to be able to respond to and recover from an exception. This is achieved through the use of exception handlers.  

+

In OCaml, an exception handler is declared using a try/with expression. Here’s the basic syntax.

try <expr> with
 | <pat1> -> <expr1>
 | <pat2> -> <expr2>
 ...
-

A try/with clause first evaluates its body, -expr. If no exception is thrown, then the result -of evaluating the body is what the entire try/with clause -evaluates to.

-

But if the evaluation of the body throws an exception, then the -exception will be fed to the pattern-match clauses following the -with. If the exception matches a pattern, then we consider -the exception caught, and the try/with clause evaluates to -the expression on the right-hand side of the matching pattern.

-

Otherwise, the original exception continues up the stack of function -calls, to be handled by the next outer exception handler. If the -exception is never caught, it terminates the program.

+

A try/with clause first evaluates its body, expr. If no exception is thrown, then the result of evaluating the body is what the entire try/with clause evaluates to.

+

But if the evaluation of the body throws an exception, then the exception will be fed to the pattern-match clauses following the with. If the exception matches a pattern, then we consider the exception caught, and the try/with clause evaluates to the expression on the right-hand side of the matching pattern.

+

Otherwise, the original exception continues up the stack of function calls, to be handled by the next outer exception handler. If the exception is never caught, it terminates the program.

Cleaning Up in the Presence of Exceptions

-

One headache with exceptions is that they can terminate your -execution at unexpected places, leaving your program in an awkward -state. Consider the following function for loading a file full of -numerical data. This code parses data that matches a simple -comma-separated file format, where each field is a floating point -number. In this example we open Stdio, to get access to -routines for reading from files.   

+

One headache with exceptions is that they can terminate your execution at unexpected places, leaving your program in an awkward state. Consider the following function for loading a file full of numerical data. This code parses data that matches a simple comma-separated file format, where each field is a floating point number. In this example we open Stdio, to get access to routines for reading from files.   

open Stdio;;
 let parse_line line =
@@ -581,18 +380,8 @@ 
Cleaning Up in the Presence of Exceptions

 >val load : string -> float list list = <fun>
-

One problem with this code is that the parsing function can throw an -exception if the file in question is malformed. Unfortunately, that -means that the In_channel.t that was opened will never be -closed, leading to a file-descriptor leak.

-

We can fix this using Base’s Exn.protect function, which -takes two arguments: a thunk f, which is the main body of -the computation to be run; and a thunk finally, which is to -be called when f exits, whether it exits normally or with -an exception. This is similar to the try/finally construct -available in many programming languages, but it is implemented in a -library, rather than being a built-in primitive. Here’s how it could be -used to fix our load function:

+

One problem with this code is that the parsing function can throw an exception if the file in question is malformed. Unfortunately, that means that the In_channel.t that was opened will never be closed, leading to a file-descriptor leak.

+

We can fix this using Base’s Exn.protect function, which takes two arguments: a thunk f, which is the main body of the computation to be run; and a thunk finally, which is to be called when f exits, whether it exits normally or with an exception. This is similar to the try/finally construct available in many programming languages, but it is implemented in a library, rather than being a built-in primitive. Here’s how it could be used to fix our load function:

let load filename =
   let inc = In_channel.create filename in
@@ -602,8 +391,7 @@ 
Cleaning Up in the Presence of Exceptions

 >val load : string -> float list list = <fun>
-

This is a common enough problem that In_channel has a -function called with_file that automates this pattern:

+

This is a common enough problem that In_channel has a function called with_file that automates this pattern:

let load filename =
   In_channel.with_file filename ~f:(fun inc ->
@@ -611,19 +399,11 @@ 
Cleaning Up in the Presence of Exceptions

 >val load : string -> float list list = <fun>
-

In_channel.with_file is built on top of -protect so that it can clean up after itself in the -presence of exceptions.

+

In_channel.with_file is built on top of protect so that it can clean up after itself in the presence of exceptions.

Catching Specific Exceptions

-

OCaml’s exception-handling system allows you to tune your -error-recovery logic to the particular error that was thrown. For -example, find_exn, which we defined earlier in the chapter, -throws Key_not_found when the element in question can’t be -found. Let’s look at an example of how you could take advantage of this. -In particular, consider the following function:   

+

OCaml’s exception-handling system allows you to tune your error-recovery logic to the particular error that was thrown. For example, find_exn, which we defined earlier in the chapter, throws Key_not_found when the element in question can’t be found. Let’s look at an example of how you could take advantage of this. In particular, consider the following function:   

let lookup_weight ~compute_weight alist key =
   try
@@ -636,28 +416,15 @@ 
Catching Specific Exceptions

 >  <fun>
-

As you can see from the type, lookup_weight takes an -association list, a key for looking up a corresponding value in that -list, and a function for computing a floating-point weight from the -looked-up value. If no value is found, then a weight of 0. -should be returned.

-

The use of exceptions in this code, however, presents some problems. -In particular, what happens if compute_weight throws an -exception? Ideally, lookup_weight should propagate that -exception on, but if the exception happens to be -Key_not_found, then that’s not what will happen:

+

As you can see from the type, lookup_weight takes an association list, a key for looking up a corresponding value in that list, and a function for computing a floating-point weight from the looked-up value. If no value is found, then a weight of 0. should be returned.

+

The use of exceptions in this code, however, presents some problems. In particular, what happens if compute_weight throws an exception? Ideally, lookup_weight should propagate that exception on, but if the exception happens to be Key_not_found, then that’s not what will happen:

lookup_weight ~compute_weight:(fun _ -> raise (Key_not_found "foo"))
 ["a",3; "b",4] "a";;
 >- : float = 0.
-

This kind of problem is hard to detect in advance because the type -system doesn’t tell you what exceptions a given function might throw. -For this reason, it’s usually best to avoid relying on the identity of -the exception to determine the nature of a failure. A better approach is -to narrow the scope of the exception handler, so that when it fires it’s -very clear what part of the code failed:

+

This kind of problem is hard to detect in advance because the type system doesn’t tell you what exceptions a given function might throw. For this reason, it’s usually best to avoid relying on the identity of the exception to determine the nature of a failure. A better approach is to narrow the scope of the exception handler, so that when it fires it’s very clear what part of the code failed:

let lookup_weight ~compute_weight alist key =
   match
@@ -671,11 +438,7 @@ 
Catching Specific Exceptions

 >  <fun>
-

This nesting of a try within a match -expression is both awkward and involves some unnecessary computation (in -particular, the allocation of the option). Happily, OCaml allows for -exceptions to be caught by match expressions directly, which lets you -write this more concisely as follows.  

+

This nesting of a try within a match expression is both awkward and involves some unnecessary computation (in particular, the allocation of the option). Happily, OCaml allows for exceptions to be caught by match expressions directly, which lets you write this more concisely as follows.  

let lookup_weight ~compute_weight alist key =
   match find_exn alist key with
@@ -686,11 +449,8 @@ 
Catching Specific Exceptions

 >  <fun>
-

Note that the exception keyword is used to mark the -exception-handling cases.

-

Best of all is to avoid exceptions entirely, which we could do by -using the exception-free function from Base, -List.Assoc.find, instead:

+

Note that the exception keyword is used to mark the exception-handling cases.

+

Best of all is to avoid exceptions entirely, which we could do by using the exception-free function from Base, List.Assoc.find, instead:

let lookup_weight ~compute_weight alist key =
   match List.Assoc.find ~equal:String.equal alist key with
@@ -704,10 +464,7 @@ 
Catching Specific Exceptions

Backtraces

-

A big part of the value of exceptions is that they provide useful -debugging information in the form of a stack backtrace. Consider the -following simple program:   -   

+

A big part of the value of exceptions is that they provide useful debugging information in the form of a stack backtrace. Consider the following simple program:      

open Base
 open Stdio
@@ -721,9 +478,7 @@ 
Backtraces

   printf "%d\n" (list_max [1;2;3]);
   printf "%d\n" (list_max [])
-

If we build and run this program, we’ll get a stack backtrace that -will provide some information about where the error occurred and the -stack of function calls that were in place at the time of the error:

+

If we build and run this program, we’ll get a stack backtrace that will provide some information about where the error occurred and the stack of function calls that were in place at the time of the error:

dune exec -- ./blow_up.exe
 >3
@@ -732,33 +487,17 @@ 
Backtraces

 >Called from Dune__exe__Blow_up in file "blow_up.ml", line 11, characters 16-29
 [2]
-

You can also capture a backtrace within your program by calling -Backtrace.Exn.most_recent, which returns the backtrace of -the most recently thrown exception. This is useful for reporting -detailed information on errors that did not cause your program to fail. - 

-

This works well if you have backtraces enabled, but that isn’t always -the case. In fact, by default, OCaml has backtraces turned off, and even -if you have them turned on at runtime, you can’t get backtraces unless -you have compiled with debugging symbols. Base reverses the default, so -if you’re linking in Base, you will have backtraces enabled by -default.

-

Even using Base and compiling with debugging symbols, you can turn -backtraces off via the OCAMLRUNPARAM environment variable, -as shown below.

+

You can also capture a backtrace within your program by calling Backtrace.Exn.most_recent, which returns the backtrace of the most recently thrown exception. This is useful for reporting detailed information on errors that did not cause your program to fail.  

+

This works well if you have backtraces enabled, but that isn’t always the case. In fact, by default, OCaml has backtraces turned off, and even if you have them turned on at runtime, you can’t get backtraces unless you have compiled with debugging symbols. Base reverses the default, so if you’re linking in Base, you will have backtraces enabled by default.

+

Even using Base and compiling with debugging symbols, you can turn backtraces off via the OCAMLRUNPARAM environment variable, as shown below.

OCAMLRUNPARAM=b=0 dune exec -- ./blow_up.exe
 >3
 >Fatal error: exception Dune__exe__Blow_up.Empty_list
 [2]
-

The resulting error message is considerably less informative. You can -also turn backtraces off in your code by calling -Backtrace.Exn.set_recording false.  

-

There is a legitimate reason to run without backtraces: speed. -OCaml’s exceptions are fairly fast, but they’re faster still if you -disable backtraces. Here’s a simple benchmark that shows the effect, -using the core_bench package:

+

The resulting error message is considerably less informative. You can also turn backtraces off in your code by calling Backtrace.Exn.set_recording false.  

+

There is a legitimate reason to run without backtraces: speed. OCaml’s exceptions are fairly fast, but they’re faster still if you disable backtraces. Here’s a simple benchmark that shows the effect, using the core_bench package:

open Core
 open Core_bench
@@ -794,17 +533,12 @@ 
Backtraces

   |> Bench.make_command
   |> Command_unix.run
-

We’re testing four cases here:     - 

+

We’re testing four cases here:      

  • a simple computation with no exception,
    • -
  • the same, but with an exception handler but no exception -thrown,
    • +
  • the same, but with an exception handler but no exception thrown,
  • the same, but where an exception is thrown,
    • -
  • and finally, the same, but where we throw an exception using -raise_notrace, which is a version of raise -which locally avoids the costs of keeping track of the backtrace.
    • +
  • and finally, the same, but where we throw an exception using raise_notrace, which is a version of raise which locally avoids the costs of keeping track of the backtrace.

Here are the results.

@@ -820,13 +554,8 @@

Backtraces

> end with exn notrace 11.69ns 23.28c
-

Note that we lose just a small number of cycles to setting up an -exception handler, which means that an unused exception handler is quite -cheap indeed. We lose a much bigger chunk, around 55 cycles, to actually -raising an exception. If we explicitly raise an exception with no -backtrace, it costs us about 25 cycles.

-

We can also disable backtraces, as we discussed, using -OCAMLRUNPARAM. That changes the results a bit.

+

Note that we lose just a small number of cycles to setting up an exception handler, which means that an unused exception handler is quite cheap indeed. We lose a much bigger chunk, around 55 cycles, to actually raising an exception. If we explicitly raise an exception with no backtrace, it costs us about 25 cycles.

+

We can also disable backtraces, as we discussed, using OCAMLRUNPARAM. That changes the results a bit.

OCAMLRUNPARAM=b=0 dune exec -- 
 ./exn_cost.exe -ascii -quota 1 -clear-columns time cycles
@@ -840,25 +569,12 @@ 
Backtraces

 >  end with exn notrace     11.48ns      22.86c
-

The only significant change here is that raising an exception in the -ordinary way becomes just a bit cheaper: 20 cycles instead of 55 cycles. -But it’s still not as fast as using raise_notrace -explicitly.

-

Differences on this scale should only matter if you’re using -exceptions routinely as part of your flow control. That’s not a common -pattern, and when you do need it, it’s better from a performance -perspective to use raise_notrace. All of which is to say, -you should almost always leave stack-traces on.

+

The only significant change here is that raising an exception in the ordinary way becomes just a bit cheaper: 20 cycles instead of 55 cycles. But it’s still not as fast as using raise_notrace explicitly.

+

Differences on this scale should only matter if you’re using exceptions routinely as part of your flow control. That’s not a common pattern, and when you do need it, it’s better from a performance perspective to use raise_notrace. All of which is to say, you should almost always leave stack-traces on.

From Exceptions to Error-Aware Types and Back Again

-

Both exceptions and error-aware types are necessary parts of -programming in OCaml. As such, you often need to move between these two -worlds. Happily, Base comes with some useful helper functions to help -you do just that. For example, given a piece of code that can throw an -exception, you can capture that exception into an option as -follows:   

+

Both exceptions and error-aware types are necessary parts of programming in OCaml. As such, you often need to move between these two worlds. Happily, Base comes with some useful helper functions to help you do just that. For example, given a piece of code that can throw an exception, you can capture that exception into an option as follows:   

let find alist key =
 Option.try_with (fun () -> find_exn alist key);;
@@ -869,8 +585,7 @@ 
From Exceptions to Error-Aware Types and Back Again

 >- : int option = Base.Option.Some 2
-

Result and Or_error have similar -try_with functions. So, we could write:

+

Result and Or_error have similar try_with functions. So, we could write:

let find alist key =
 Or_error.try_with (fun () -> find_exn alist key);;
@@ -891,34 +606,12 @@ 
From Exceptions to Error-Aware Types and Back Again

Choosing an Error-Handling Strategy

-

Given that OCaml supports both exceptions and error-aware return -types, how do you choose between them? The key is to think about the -trade-off between concision and explicitness. 

-

Exceptions are more concise because they allow you to defer the job -of error handling to some larger scope, and because they don’t clutter -up your types. But this concision comes at a cost: exceptions are all -too easy to ignore. Error-aware return types, on the other hand, are -fully manifest in your type definitions, making the errors that your -code might generate explicit and impossible to ignore. 

-

The right trade-off depends on your application. If you’re writing a -rough-and-ready program where getting it done quickly is key and failure -is not that expensive, then using exceptions extensively may be the way -to go. If, on the other hand, you’re writing production software whose -failure is costly, then you should probably lean in the direction of -using error-aware return types.

-

To be clear, it doesn’t make sense to avoid exceptions entirely. The -maxim of “use exceptions for exceptional conditions” applies. If an -error occurs sufficiently rarely, then throwing an exception is often -the right behavior.

-

Also, for errors that are omnipresent, error-aware return types may -be overkill. A good example is out-of-memory errors, which can occur -anywhere, and so you’d need to use error-aware return types everywhere -to capture those. Having every operation marked as one that might fail -is no more explicit than having none of them marked.

-

In short, for errors that are a foreseeable and ordinary part of the -execution of your production code and that are not omnipresent, -error-aware return types are typically the right solution.

+

Given that OCaml supports both exceptions and error-aware return types, how do you choose between them? The key is to think about the trade-off between concision and explicitness. 

+

Exceptions are more concise because they allow you to defer the job of error handling to some larger scope, and because they don’t clutter up your types. But this concision comes at a cost: exceptions are all too easy to ignore. Error-aware return types, on the other hand, are fully manifest in your type definitions, making the errors that your code might generate explicit and impossible to ignore. 

+

The right trade-off depends on your application. If you’re writing a rough-and-ready program where getting it done quickly is key and failure is not that expensive, then using exceptions extensively may be the way to go. If, on the other hand, you’re writing production software whose failure is costly, then you should probably lean in the direction of using error-aware return types.

+

To be clear, it doesn’t make sense to avoid exceptions entirely. The maxim of “use exceptions for exceptional conditions” applies. If an error occurs sufficiently rarely, then throwing an exception is often the right behavior.

+

Also, for errors that are omnipresent, error-aware return types may be overkill. A good example is out-of-memory errors, which can occur anywhere, and so you’d need to use error-aware return types everywhere to capture those. Having every operation marked as one that might fail is no more explicit than having none of them marked.

+

In short, for errors that are a foreseeable and ordinary part of the execution of your production code and that are not omnipresent, error-aware return types are typically the right solution.

-

Next: Chapter 08Imperative Programming

 \ No newline at end of file +

Next: Chapter 08Imperative Programming

 \ No newline at end of file diff --git a/faqs.html b/faqs.html index 252a13e3d..ca7a85bb1 100644 --- a/faqs.html +++ b/faqs.html @@ -1,4 +1,4 @@ -Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
\ No newline at end of file diff --git a/files-modules-and-programs.html b/files-modules-and-programs.html index a51e2c916..cbbddcd3a 100644 --- a/files-modules-and-programs.html +++ b/files-modules-and-programs.html @@ -1,30 +1,11 @@ -Files, Modules, and Programs - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Files, Modules, and Programs - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Files, Modules, and Programs

-

We’ve so far experienced OCaml largely through the toplevel. As you -move from exercises to real-world programs, you’ll need to leave the -toplevel behind and start building programs from files. Files are more -than just a convenient way to store and manage your code; in OCaml, they -also correspond to modules, which act as boundaries that divide your -program into conceptual units.

-

In this chapter, we’ll show you how to build an OCaml program from a -collection of files, as well as the basics of working with modules and -module signatures.

+

We’ve so far experienced OCaml largely through the toplevel. As you move from exercises to real-world programs, you’ll need to leave the toplevel behind and start building programs from files. Files are more than just a convenient way to store and manage your code; in OCaml, they also correspond to modules, which act as boundaries that divide your program into conceptual units.

+

In this chapter, we’ll show you how to build an OCaml program from a collection of files, as well as the basics of working with modules and module signatures.

Single-File Programs

-

We’ll start with an example: a utility that reads lines from -stdin, computes a frequency count of the lines, and prints -out the ten most frequent lines. We’ll start with a simple -implementation, which we’ll save as the file freq.ml.   

-

This implementation will use two functions from the -List.Assoc module, which provides utility functions for -interacting with association lists, i.e., lists of -key/value pairs. In particular, we use the function -List.Assoc.find, which looks up a key in an association -list; and List.Assoc.add, which adds a new binding to an -association list, as shown here:     

+

We’ll start with an example: a utility that reads lines from stdin, computes a frequency count of the lines, and prints out the ten most frequent lines. We’ll start with a simple implementation, which we’ll save as the file freq.ml.   

+

This implementation will use two functions from the List.Assoc module, which provides utility functions for interacting with association lists, i.e., lists of key/value pairs. In particular, we use the function List.Assoc.find, which looks up a key in an association list; and List.Assoc.add, which adds a new binding to an association list, as shown here:     

open Base;;
 let assoc = [("one", 1); ("two",2); ("three",3)];;
@@ -38,9 +19,7 @@ 
Single-File Programs

 >- : (string, int) Base.List.Assoc.t = [("two", 4); ("one", 1); ("three", 3)]
-

Note that List.Assoc.add doesn’t modify the original -list, but instead allocates a new list with the requisite key/value pair -added.

+

Note that List.Assoc.add doesn’t modify the original list, but instead allocates a new list with the requisite key/value pair added.

Now we can write freq.ml.

open Base
@@ -61,39 +40,14 @@ 
Single-File Programs

   |> (fun l -> List.take l 10)
   |> List.iter ~f:(fun (line, count) -> printf "%3d: %s\n" count line)
-

The function build_counts reads in lines from -stdin, constructing from those lines an association list -with the frequencies of each line. It does this by invoking -In_channel.fold_lines (similar to the function -List.fold described in Chapter 3, Lists -And Patterns), which reads through the lines one by one, calling the -provided fold function for each line to update the -accumulator. That accumulator is initialized to the empty list.

-

With build_counts defined, we then call the function to -build the association list, sort that list by frequency in descending -order, grab the first 10 elements off the list, and then iterate over -those 10 elements and print them to the screen. These operations are -tied together using the |> operator described in Chapter 2, Variables And Functions.  

-
-

Where Is main?

-

Unlike programs in C, Java or C#, programs in OCaml don’t have a -unique main function. When an OCaml program is evaluated, -all the statements in the implementation files are evaluated in the -order in which they were linked together. These implementation files can -contain arbitrary expressions, not just function definitions. In this -example, the declaration starting with let () = plays the -role of the main function, kicking off the processing. But -really the entire file is evaluated at startup, and so in some sense the -full codebase is one big main function.

-

The idiom of writing let () = may seem a bit odd, but it -has a purpose. The let binding here is a pattern-match to a -value of type unit, which is there to ensure that the -expression on the right-hand side returns unit, as is -common for functions that operate primarily by side effect.

-
-

If we weren’t using Base or any other external -libraries, we could build the executable like this:

+

The function build_counts reads in lines from stdin, constructing from those lines an association list with the frequencies of each line. It does this by invoking In_channel.fold_lines (similar to the function List.fold described in Chapter 3, Lists And Patterns), which reads through the lines one by one, calling the provided fold function for each line to update the accumulator. That accumulator is initialized to the empty list.

+

With build_counts defined, we then call the function to build the association list, sort that list by frequency in descending order, grab the first 10 elements off the list, and then iterate over those 10 elements and print them to the screen. These operations are tied together using the |> operator described in Chapter 2, Variables And Functions.  

+
+

Where Is main?

+

Unlike programs in C, Java or C#, programs in OCaml don’t have a unique main function. When an OCaml program is evaluated, all the statements in the implementation files are evaluated in the order in which they were linked together. These implementation files can contain arbitrary expressions, not just function definitions. In this example, the declaration starting with let () = plays the role of the main function, kicking off the processing. But really the entire file is evaluated at startup, and so in some sense the full codebase is one big main function.

+

The idiom of writing let () = may seem a bit odd, but it has a purpose. The let binding here is a pattern-match to a value of type unit, which is there to ensure that the expression on the right-hand side returns unit, as is common for functions that operate primarily by side effect.

+
+

If we weren’t using Base or any other external libraries, we could build the executable like this:

ocamlopt freq.ml -o freq
 >File "freq.ml", line 1, characters 5-9:
@@ -102,36 +56,18 @@ 
Where Is main?

 >Error: Unbound module Base
 [2]
-

But as you can see, it fails because it can’t find Base -and Stdio. We need a somewhat more complex invocation to -get them linked in:    

+

But as you can see, it fails because it can’t find Base and Stdio. We need a somewhat more complex invocation to get them linked in:    

ocamlfind ocamlopt -linkpkg -package base -package stdio freq.ml -o freq
-

This uses ocamlfind, a tool which itself invokes other -parts of the OCaml toolchain (in this case, ocamlopt) with -the appropriate flags to link in particular libraries and packages. -Here, -package base is asking ocamlfind to -link in the Base library; -linkpkg asks -ocamlfind to link in the packages as is necessary for building an -executable.  

-

While this works well enough for a one-file project, more complicated -projects require a tool to orchestrate the build. One good tool for this -task is dune. To invoke dune, you need to have -two files: a dune-project file for the overall project, and -a dune file that configures the particular directory. This -is a single-directory project, so we’ll just have one of each, but more -realistic projects will have one dune-project and many -dune files.    

-

At its simplest, the dune-project just specifies the -version of the dune configuration-language in use.

+

This uses ocamlfind, a tool which itself invokes other parts of the OCaml toolchain (in this case, ocamlopt) with the appropriate flags to link in particular libraries and packages. Here, -package base is asking ocamlfind to link in the Base library; -linkpkg asks ocamlfind to link in the packages as is necessary for building an executable.  

+

While this works well enough for a one-file project, more complicated projects require a tool to orchestrate the build. One good tool for this task is dune. To invoke dune, you need to have two files: a dune-project file for the overall project, and a dune file that configures the particular directory. This is a single-directory project, so we’ll just have one of each, but more realistic projects will have one dune-project and many dune files.    

+

At its simplest, the dune-project just specifies the version of the dune configuration-language in use.

(lang dune 3.0)
-

We also need a dune file to declare the executable we -want to build, along with the libraries it depends on.

+

We also need a dune file to declare the executable we want to build, along with the libraries it depends on.

(executable
   (name      freq)
@@ -142,12 +78,7 @@ 
Where Is main?

 
dune build freq.exe
-

We can run the resulting executable, freq.exe, from the -command line. Executables built with dune will be left in -the _build/default directory, from which they can be -invoked. The specific invocation below will count the words that come up -in the file freq.ml itself.  

+

We can run the resulting executable, freq.exe, from the command line. Executables built with dune will be left in the _build/default directory, from which they can be invoked. The specific invocation below will count the words that come up in the file freq.ml itself.  

grep -Eo '[[:alpha:]]+' freq.ml | ./_build/default/freq.exe
 >  5: line
@@ -162,9 +93,7 @@ 
Where Is main?

 >  2: l
-

Conveniently, dune allows us to combine the building and -running an executable into a single operation, which we can do using -dune exec.

+

Conveniently, dune allows us to combine the building and running an executable into a single operation, which we can do using dune exec.

grep -Eo '[[:alpha:]]+' freq.ml | dune exec ./freq.exe
 >  5: line
@@ -179,64 +108,20 @@ 
Where Is main?

 >  2: l
-

We’ve really just scratched the surface of what can be done with -dune. We’ll discuss dune in more detail in Chapter 21, The OCaml -Platform.

-
-

Bytecode Versus Native Code

-

OCaml ships with two compilers: the ocamlopt native code -compiler and the ocamlc bytecode compiler. Programs -compiled with ocamlc are interpreted by a virtual machine, -while programs compiled with ocamlopt are compiled to -machine code to be run on a specific operating system and processor -architecture. With dune, targets ending with -.bc are built as bytecode executables, and those ending -with .exe are built as native code.

-

Aside from performance, executables generated by the two compilers -have nearly identical behavior. There are a few things to be aware of. -First, the bytecode compiler can be used on more architectures, and has -some tools that are not available for native code. For example, the -OCaml debugger only works with bytecode (although gdb, the -GNU Debugger, works with some limitations on OCaml native-code -applications). The bytecode compiler is also quicker than the -native-code compiler. In addition, in order to run a bytecode -executable, you typically need to have OCaml installed on the system in -question. That’s not strictly required, though, since you can build a -bytecode executable with an embedded runtime, using the --custom compiler flag.

-

As a general matter, production executables should usually be built -using the native-code compiler, but it sometimes makes sense to use -bytecode for development builds. And, of course, bytecode makes sense -when targeting a platform not supported by the native-code compiler. -We’ll cover both compilers in more detail in Chapter 26, The Compiler Backend: Byte Code And Native -Code.

-
+

We’ve really just scratched the surface of what can be done with dune. We’ll discuss dune in more detail in Chapter 21, The OCaml Platform.

+
+

Bytecode Versus Native Code

+

OCaml ships with two compilers: the ocamlopt native code compiler and the ocamlc bytecode compiler. Programs compiled with ocamlc are interpreted by a virtual machine, while programs compiled with ocamlopt are compiled to machine code to be run on a specific operating system and processor architecture. With dune, targets ending with .bc are built as bytecode executables, and those ending with .exe are built as native code.

+

Aside from performance, executables generated by the two compilers have nearly identical behavior. There are a few things to be aware of. First, the bytecode compiler can be used on more architectures, and has some tools that are not available for native code. For example, the OCaml debugger only works with bytecode (although gdb, the GNU Debugger, works with some limitations on OCaml native-code applications). The bytecode compiler is also quicker than the native-code compiler. In addition, in order to run a bytecode executable, you typically need to have OCaml installed on the system in question. That’s not strictly required, though, since you can build a bytecode executable with an embedded runtime, using the -custom compiler flag.

+

As a general matter, production executables should usually be built using the native-code compiler, but it sometimes makes sense to use bytecode for development builds. And, of course, bytecode makes sense when targeting a platform not supported by the native-code compiler. We’ll cover both compilers in more detail in Chapter 26, The Compiler Backend: Byte Code And Native Code.

+

Multifile Programs and Modules

-

Source files in OCaml are tied into the module system, with each file -compiling down into a module whose name is derived from the name of the -file. We’ve encountered modules before, such as when we used functions -like find and add from the -List.Assoc module. At its simplest, you can think of a -module as a collection of definitions that are stored within a -namespace.    

-

Let’s consider how we can use modules to refactor the implementation -of freq.ml. Remember that the variable counts -contains an association list representing the counts of the lines seen -so far. But updating an association list takes time linear in the length -of the list, meaning that the time complexity of processing a file is -quadratic in the number of distinct lines in the file.

-

We can fix this problem by replacing association lists with a more -efficient data structure. To do that, we’ll first factor out the key -functionality into a separate module with an explicit interface. We can -consider alternative (and more efficient) implementations once we have a -clear interface to program against.

-

We’ll start by creating a file, counter.ml, that -contains the logic for maintaining the association list used to -represent the frequency counts. The key function, called -touch, bumps the frequency count of a given line by -one.

+

Source files in OCaml are tied into the module system, with each file compiling down into a module whose name is derived from the name of the file. We’ve encountered modules before, such as when we used functions like find and add from the List.Assoc module. At its simplest, you can think of a module as a collection of definitions that are stored within a namespace.    

+

Let’s consider how we can use modules to refactor the implementation of freq.ml. Remember that the variable counts contains an association list representing the counts of the lines seen so far. But updating an association list takes time linear in the length of the list, meaning that the time complexity of processing a file is quadratic in the number of distinct lines in the file.

+

We can fix this problem by replacing association lists with a more efficient data structure. To do that, we’ll first factor out the key functionality into a separate module with an explicit interface. We can consider alternative (and more efficient) implementations once we have a clear interface to program against.

+

We’ll start by creating a file, counter.ml, that contains the logic for maintaining the association list used to represent the frequency counts. The key function, called touch, bumps the frequency count of a given line by one.

open Base
 
@@ -248,12 +133,8 @@ 
Multifile Programs and Modules

   in
   List.Assoc.add ~equal:String.equal counts line (count + 1)
-

The file counter.ml will be compiled into a module named -Counter, where the name of the module is derived -automatically from the filename. The module name is capitalized even if -the file is not. Indeed, module names are always capitalized.  

-

We can now rewrite freq.ml to use -Counter.

+

The file counter.ml will be compiled into a module named Counter, where the name of the module is derived automatically from the filename. The module name is capitalized even if the file is not. Indeed, module names are always capitalized.  

+

We can now rewrite freq.ml to use Counter.

open Base
 open Stdio
@@ -267,9 +148,7 @@ 
Multifile Programs and Modules

   |> (fun l -> List.take l 10)
   |> List.iter ~f:(fun (line, count) -> printf "%3d: %s\n" count line)
-

The resulting code can still be built with dune, which -will discover dependencies and realize that counter.ml -needs to be compiled.

+

The resulting code can still be built with dune, which will discover dependencies and realize that counter.ml needs to be compiled.

dune build freq.exe
@@ -277,44 +156,21 @@

Multifile Programs and Modules

Signatures and Abstract Types

-

While we’ve pushed some of the logic to the Counter -module, the code in freq.ml can still depend on the details -of the implementation of Counter. Indeed, if you look at -the definition of build_counts, you’ll see that it depends -on the fact that the empty set of frequency counts is represented as an -empty list. We’d like to prevent this kind of dependency, so we can -change the implementation of Counter without needing to -change client code like that in freq.ml.      

-

The implementation details of a module can be hidden by attaching an -interface. (Note that in the context of OCaml, the terms -interface, signature, and module type are all -used interchangeably.) A module defined by a file -filename.ml can be constrained by a signature placed in a -file called filename.mli.  

-

For counter.mli, we’ll start by writing down an -interface that describes what’s currently available in -counter.ml, without hiding anything. val -declarations are used to specify values in a signature. The syntax of a -val declaration is as follows:

+

While we’ve pushed some of the logic to the Counter module, the code in freq.ml can still depend on the details of the implementation of Counter. Indeed, if you look at the definition of build_counts, you’ll see that it depends on the fact that the empty set of frequency counts is represented as an empty list. We’d like to prevent this kind of dependency, so we can change the implementation of Counter without needing to change client code like that in freq.ml.      

+

The implementation details of a module can be hidden by attaching an interface. (Note that in the context of OCaml, the terms interface, signature, and module type are all used interchangeably.) A module defined by a file filename.ml can be constrained by a signature placed in a file called filename.mli.  

+

For counter.mli, we’ll start by writing down an interface that describes what’s currently available in counter.ml, without hiding anything. val declarations are used to specify values in a signature. The syntax of a val declaration is as follows:

val <identifier> : <type>
-

Using this syntax, we can write the signature of -counter.ml as follows.

+

Using this syntax, we can write the signature of counter.ml as follows.

open Base
 
 (** Bump the frequency count for the given string. *)
 val touch : (string * int) list -> string -> (string * int) list
-

Note that dune will detect the presence of the -mli file automatically and include it in the build.

-

To hide the fact that frequency counts are represented as association -lists, we’ll need to make the type of frequency counts -abstract. A type is abstract if its name is exposed in the -interface, but its definition is not. Here’s an abstract interface for -Counter:

+

Note that dune will detect the presence of the mli file automatically and include it in the build.

+

To hide the fact that frequency counts are represented as association lists, we’ll need to make the type of frequency counts abstract. A type is abstract if its name is exposed in the interface, but its definition is not. Here’s an abstract interface for Counter:

open Base
 
@@ -331,17 +187,9 @@ 
Signatures and Abstract Types

     string shows up at most once, and the counts are >= 1. *)
 val to_list : t -> (string * int) list
-

We added empty and to_list to -Counter, since without them there would be no way to create -a Counter.t or get data out of one.

-

We also used this opportunity to document the module. The -mli file is the place where you specify your module’s -interface, and as such is a natural place to put documentation. We -started our comments with a double asterisk to cause them to be picked -up by the odoc tool when generating API documentation. -We’ll discuss odoc more in Chapter 21, The OCaml Platform.

-

Here’s a rewrite of counter.ml to match the new -counter.mli:

+

We added empty and to_list to Counter, since without them there would be no way to create a Counter.t or get data out of one.

+

We also used this opportunity to document the module. The mli file is the place where you specify your module’s interface, and as such is a natural place to put documentation. We started our comments with a double asterisk to cause them to be picked up by the odoc tool when generating API documentation. We’ll discuss odoc more in Chapter 21, The OCaml Platform.

+

Here’s a rewrite of counter.ml to match the new counter.mli:

open Base
 
@@ -358,8 +206,7 @@ 
Signatures and Abstract Types

   in
   List.Assoc.add ~equal:String.equal counts line (count + 1)
-

If we now try to compile freq.ml, we’ll get the -following error:

+

If we now try to compile freq.ml, we’ll get the following error:

dune build freq.exe
 >File "freq.ml", line 5, characters 53-66:
@@ -371,12 +218,7 @@ 
Signatures and Abstract Types

 >       Type Counter.t is not compatible with type 'a list
 [1]
-

This is because freq.ml depends on the fact that -frequency counts are represented as association lists, a fact that we’ve -just hidden. We just need to fix build_counts to use -Counter.empty instead of [] and to use -Counter.to_list to convert the completed counts to an -association list. The resulting implementation is shown below.

+

This is because freq.ml depends on the fact that frequency counts are represented as association lists, a fact that we’ve just hidden. We just need to fix build_counts to use Counter.empty instead of [] and to use Counter.to_list to convert the completed counts to an association list. The resulting implementation is shown below.

open Base
 open Stdio
@@ -399,10 +241,7 @@ 
Signatures and Abstract Types

 
dune build freq.exe
-

Now we can turn to optimizing the implementation of -Counter. Here’s an alternate and far more efficient -implementation, based on Base’s Map data -structure.

+

Now we can turn to optimizing the implementation of Counter. Here’s an alternate and far more efficient implementation, based on Base’s Map data structure.

open Base
 
@@ -419,27 +258,12 @@ 
Signatures and Abstract Types

   in
   Map.set t ~key:s ~data:(count + 1)
-

There’s some unfamiliar syntax in the above example, in particular -the use of int Map.M(String).t to indicate the type of a -map, and Map.empty (module String) to generate an empty -map. Here, we’re making use of a more advanced feature of the language -(specifically, functors and first-class modules, which we’ll get to in -later chapters). The use of these features for the Map data-structure in -particular is covered in Chapter 14, Maps And Hash Tables.

+

There’s some unfamiliar syntax in the above example, in particular the use of int Map.M(String).t to indicate the type of a map, and Map.empty (module String) to generate an empty map. Here, we’re making use of a more advanced feature of the language (specifically, functors and first-class modules, which we’ll get to in later chapters). The use of these features for the Map data-structure in particular is covered in Chapter 14, Maps And Hash Tables.

Concrete Types in Signatures

-

In our frequency-count example, the module Counter had -an abstract type Counter.t for representing a collection of -frequency counts. Sometimes, you’ll want to make a type in your -interface concrete, by including the type definition in the -interface.   

-

For example, imagine we wanted to add a function to -Counter for returning the line with the median frequency -count. If the number of lines is even, then there is no single median, -and the function would return the lines before and after the median -instead. We’ll use a custom type to represent the fact that there are -two possible return values. Here’s a possible implementation:

+

In our frequency-count example, the module Counter had an abstract type Counter.t for representing a collection of frequency counts. Sometimes, you’ll want to make a type in your interface concrete, by including the type definition in the interface.   

+

For example, imagine we wanted to add a function to Counter for returning the line with the median frequency count. If the number of lines is even, then there is no single median, and the function would return the lines before and after the median instead. We’ll use a custom type to represent the fact that there are two possible return values. Here’s a possible implementation:

type median =
   | Median of string
@@ -457,15 +281,8 @@ 
Concrete Types in Signatures

   then Median (nth (len / 2))
   else Before_and_after (nth ((len / 2) - 1), nth (len / 2))
-

In the above, we use failwith to throw an exception for -the case of the empty list. We’ll discuss exceptions more in Chapter 7, Error -Handling. Note also that the function fst simply -returns the first element of any two-tuple.

-

Now, to expose this usefully in the interface, we need to expose both -the function and the type median with its definition. Note -that values (of which functions are an example) and types have distinct -namespaces, so there’s no name clash here. Adding the following two -lines to counter.mli does the trick.

+

In the above, we use failwith to throw an exception for the case of the empty list. We’ll discuss exceptions more in Chapter 7, Error Handling. Note also that the function fst simply returns the first element of any two-tuple.

+

Now, to expose this usefully in the interface, we need to expose both the function and the type median with its definition. Note that values (of which functions are an example) and types have distinct namespaces, so there’s no name clash here. Adding the following two lines to counter.mli does the trick.

(** Represents the median computed from a set of strings. In the case
     where there is an even number of choices, the one before and after
@@ -476,29 +293,13 @@ 
Concrete Types in Signatures

 
 val median : t -> median
-

The decision of whether a given type should be abstract or concrete -is an important one. Abstract types give you more control over how -values are created and accessed, and make it easier to enforce -invariants beyond what is enforced by the type itself; concrete types -let you expose more detail and structure to client code in a lightweight -way. The right choice depends very much on the context.

+

The decision of whether a given type should be abstract or concrete is an important one. Abstract types give you more control over how values are created and accessed, and make it easier to enforce invariants beyond what is enforced by the type itself; concrete types let you expose more detail and structure to client code in a lightweight way. The right choice depends very much on the context.

Nested Modules

-

Up until now, we’ve only considered modules that correspond to files, -like counter.ml. But modules (and module signatures) can be -nested inside other modules. As a simple example, consider a program -that needs to deal with multiple identifiers like usernames and -hostnames. If you just represent these as strings, then it becomes easy -to confuse one with the other.    

-

A better approach is to mint new abstract types for each identifier, -where those types are under the covers just implemented as strings. That -way, the type system will prevent you from confusing a username with a -hostname, and if you do need to convert, you can do so using explicit -conversions to and from the string type.

-

Here’s how you might create such an abstract type, within a -submodule:  

+

Up until now, we’ve only considered modules that correspond to files, like counter.ml. But modules (and module signatures) can be nested inside other modules. As a simple example, consider a program that needs to deal with multiple identifiers like usernames and hostnames. If you just represent these as strings, then it becomes easy to confuse one with the other.    

+

A better approach is to mint new abstract types for each identifier, where those types are under the covers just implemented as strings. That way, the type system will prevent you from confusing a username with a hostname, and if you do need to convert, you can do so using explicit conversions to and from the string type.

+

Here’s how you might create such an abstract type, within a submodule:  

open Base
 
@@ -516,22 +317,12 @@ 
Nested Modules

   let ( = ) = String.( = )
 end
-

Note that the to_string and of_string -functions above are implemented simply as the identity function, which -means they have no runtime effect. They are there purely as part of the -discipline that they enforce on the code through the type system. We -also chose to put in an equality function, so you can check if two -usernames match. In a real application, we might want more -functionality, like the ability to hash and compare usernames, but we’ve -kept this example purposefully simple.

+

Note that the to_string and of_string functions above are implemented simply as the identity function, which means they have no runtime effect. They are there purely as part of the discipline that they enforce on the code through the type system. We also chose to put in an equality function, so you can check if two usernames match. In a real application, we might want more functionality, like the ability to hash and compare usernames, but we’ve kept this example purposefully simple.

The basic structure of a module declaration like this is:

module <name> : <signature> = <implementation>
-

We could have written this slightly differently, by giving the -signature its own top-level module type declaration, making -it possible to create multiple distinct types with the same underlying -implementation in a lightweight way:

+

We could have written this slightly differently, by giving the signature its own top-level module type declaration, making it possible to create multiple distinct types with the same underlying implementation in a lightweight way:

open Base
 module Time = Core.Time
@@ -563,10 +354,7 @@ 
Nested Modules

 
 let sessions_have_same_user s1 s2 = Username.( = ) s1.user s2.host
-

The preceding code has a bug: it compares the username in one session -to the host in the other session, when it should be comparing the -usernames in both cases. Because of how we defined our types, however, -the compiler will flag this bug for us.

+

The preceding code has a bug: it compares the username in one session to the host in the other session, when it should be comparing the usernames in both cases. Because of how we defined our types, however, the compiler will flag this bug for us.

dune build session_info.exe
 >File "session_info.ml", line 29, characters 59-66:
@@ -576,24 +364,12 @@ 
Nested Modules

 >       but an expression was expected of type Username.t
 [1]
-

This is a trivial example, but confusing different kinds of -identifiers is a very real source of bugs, and the approach of minting -abstract types for different classes of identifiers is an effective way -of avoiding such issues.

+

This is a trivial example, but confusing different kinds of identifiers is a very real source of bugs, and the approach of minting abstract types for different classes of identifiers is an effective way of avoiding such issues.

Opening Modules

-

Most of the time, you refer to values and types within a module by -using the module name as an explicit qualifier. For example, you write -List.map to refer to the map function in the -List module. Sometimes, though, you want to be able to -refer to the contents of a module without this explicit qualification. -That’s what the open statement is for.   

-

We’ve encountered open already, specifically where we’ve -written open Base to get access to the standard definitions -in the Base library. In general, opening a module adds the -contents of that module to the environment that the compiler looks at to -find the definition of various identifiers. Here’s an example:

+

Most of the time, you refer to values and types within a module by using the module name as an explicit qualifier. For example, you write List.map to refer to the map function in the List module. Sometimes, though, you want to be able to refer to the contents of a module without this explicit qualification. That’s what the open statement is for.   

+

We’ve encountered open already, specifically where we’ve written open Base to get access to the standard definitions in the Base library. In general, opening a module adds the contents of that module to the environment that the compiler looks at to find the definition of various identifiers. Here’s an example:

module M = struct let foo = 3 end;;
 >module M : sig val foo : int end
@@ -605,29 +381,15 @@ 
Opening Modules

 >- : int = 3
-

Here’s some general advice on how to use open -effectively.

+

Here’s some general advice on how to use open effectively.

Open Modules Rarely

-

open is essential when you’re using an alternative -standard library like Base, but it’s generally good style -to keep the opening of modules to a minimum. Opening a module is -basically a trade-off between terseness and explicitness—the more -modules you open, the fewer module qualifications you need, and the -harder it is to look at an identifier and figure out where it comes -from.

-

When you do use open, it should mostly be with modules -that were designed to be opened, like Base itself, or -Option.Monad_infix or Float.O within -Base..

+

open is essential when you’re using an alternative standard library like Base, but it’s generally good style to keep the opening of modules to a minimum. Opening a module is basically a trade-off between terseness and explicitness—the more modules you open, the fewer module qualifications you need, and the harder it is to look at an identifier and figure out where it comes from.

+

When you do use open, it should mostly be with modules that were designed to be opened, like Base itself, or Option.Monad_infix or Float.O within Base..

Prefer Local Opens

-

It’s generally better to keep down the amount of code affected by an -open. One great tool for this is local opens, -which let you restrict the scope of an open to an arbitrary expression. -There are two syntaxes for local opens. The following example shows the -let open syntax;  

+

It’s generally better to keep down the amount of code affected by an open. One great tool for this is local opens, which let you restrict the scope of an open to an arbitrary expression. There are two syntaxes for local opens. The following example shows the let open syntax;  

let average x y =
   let open Int64 in
@@ -635,10 +397,8 @@ 
Prefer Local Opens

 >val average : int64 -> int64 -> int64 = <fun>
-

Here, of_int and the infix operators are the ones from -the Int64 module.

-

The following shows off a more lightweight syntax which is -particularly useful for small expressions.

+

Here, of_int and the infix operators are the ones from the Int64 module.

+

The following shows off a more lightweight syntax which is particularly useful for small expressions.

let average x y =
   Int64.((x + y) / of_int 2);;
@@ -648,10 +408,7 @@ 
Prefer Local Opens

Using Module Shortcuts Instead

-

An alternative to local opens that makes your code -terser without giving up on explicitness is to locally rebind the name -of a module. So, when using the Counter.median type, -instead of writing:

+

An alternative to local opens that makes your code terser without giving up on explicitness is to locally rebind the name of a module. So, when using the Counter.median type, instead of writing:

let print_median m =
   match m with
@@ -668,19 +425,12 @@ 
Using Module Shortcuts Instead

   | C.Before_and_after (before, after) ->
     printf "Before and after median:\n   %s\n   %s\n" before after
-

Because the module name C only exists for a short scope, -it’s easy to read and remember what C stands for. Rebinding -modules to very short names at the top level of your module is usually a -mistake.

+

Because the module name C only exists for a short scope, it’s easy to read and remember what C stands for. Rebinding modules to very short names at the top level of your module is usually a mistake.

Including Modules

-

While opening a module affects the environment used to search for -identifiers, including a module is a way of adding new -identifiers to a module proper. Consider the following simple module for -representing a range of integer values:   

+

While opening a module affects the environment used to search for identifiers, including a module is a way of adding new identifiers to a module proper. Consider the following simple module for representing a range of integer values:   

module Interval = struct
   type t = | Interval of int * int
@@ -693,8 +443,7 @@ 
Including Modules

 >  sig type t = Interval of int * int | Empty val create : int -> int -> t end
-

We can use the include directive to create a new, -extended version of the Interval module:

+

We can use the include directive to create a new, extended version of the Interval module:

module Extended_interval = struct
   include Interval
@@ -714,10 +463,7 @@ 
Including Modules

 >- : bool = true
-

The difference between include and open is -that we’ve done more than change how identifiers are searched for: we’ve -changed what’s in the module. If we’d used open, we’d have -gotten a quite different result:

+

The difference between include and open is that we’ve done more than change how identifiers are searched for: we’ve changed what’s in the module. If we’d used open, we’d have gotten a quite different result:

module Extended_interval = struct
   open Interval
@@ -734,10 +480,7 @@ 
Including Modules

 >Error: Unbound value Extended_interval.create
-

To consider a more realistic example, imagine you wanted to build an -extended version of the Option module, where you’ve added -some functionality not present in the module as distributed in -Base. That’s a job for include.

+

To consider a more realistic example, imagine you wanted to build an extended version of the Option module, where you’ve added some functionality not present in the module as distributed in Base. That’s a job for include.

open Base
@@ -751,13 +494,7 @@ 
Including Modules

   | None -> None
   | Some f -> Some (f x)
-

Now, how do we write an interface for this new module? It turns out -that include works on signatures as well, so we can pull -essentially the same trick to write our mli. The only issue -is that we need to get our hands on the signature for the -Option module. This can be done using -module type of, which computes a signature from a -module:

+

Now, how do we write an interface for this new module? It turns out that include works on signatures as well, so we can pull essentially the same trick to write our mli. The only issue is that we need to get our hands on the signature for the Option module. This can be done using module type of, which computes a signature from a module:

open Base
 
@@ -767,23 +504,12 @@ 
Including Modules

 (* Signature of function we're adding *)
 val apply : ('a -> 'b) t -> 'a -> 'b t
-

The order of declarations in the mli doesn’t need to -match the order of declarations in the ml. The order of -declarations in the ml mostly matters insofar as it affects -which values are shadowed. If we wanted to replace a function in -Option with a new function of the same name, the -declaration of that function in the ml would have to come -after the include Option declaration.

-

We can now use Ext_option as a replacement for -Option. If we want to use Ext_option in -preference to Option in our project, we can create a file -of common definitions, which in this case we’ll call -import.ml.

+

The order of declarations in the mli doesn’t need to match the order of declarations in the ml. The order of declarations in the ml mostly matters insofar as it affects which values are shadowed. If we wanted to replace a function in Option with a new function of the same name, the declaration of that function in the ml would have to come after the include Option declaration.

+

We can now use Ext_option as a replacement for Option. If we want to use Ext_option in preference to Option in our project, we can create a file of common definitions, which in this case we’ll call import.ml.

module Option = Ext_option
-

Then, by opening Import, we can shadow -Base’s Option module with our extension.

+

Then, by opening Import, we can shadow Base’s Option module with our extension.

open Base
 open Import
@@ -793,17 +519,10 @@ 
Including Modules

Common Errors with Modules

-

When OCaml compiles a program with an ml and an -mli, it will complain if it detects a mismatch between the -two. Here are some of the common errors you’ll run into.

+

When OCaml compiles a program with an ml and an mli, it will complain if it detects a mismatch between the two. Here are some of the common errors you’ll run into.

Type Mismatches

-

The simplest kind of error is where the type specified in the -signature does not match the type in the implementation of the module. -As an example, if we replace the val declaration in -counter.mli by swapping the types of the first two -arguments:    

+

The simplest kind of error is where the type specified in the signature does not match the type in the implementation of the module. As an example, if we replace the val declaration in counter.mli by swapping the types of the first two arguments:    

(** Bump the frequency count for the given string. *)
 val touch : string -> t -> t
@@ -829,15 +548,12 @@

Type Mismatches

Missing Definitions

-

We might decide that we want a new function in Counter -for pulling out the frequency count of a given string. We could add that -to the mli by adding the following line.   

+

We might decide that we want a new function in Counter for pulling out the frequency count of a given string. We could add that to the mli by adding the following line.   

(** Returns the frequency count for the given string *)
 val count : t -> string -> int
-

Now if we try to compile without actually adding the implementation, -we’ll get this error.

+

Now if we try to compile without actually adding the implementation, we’ll get this error.

dune build freq.exe
 >File "counter.ml", line 1:
@@ -851,15 +567,7 @@ 
Missing Definitions

Type Definition Mismatches

-

Type definitions that show up in an mli need to match up -with corresponding definitions in the ml. Consider again -the example of the type median. The order of the -declaration of variants matters to the OCaml compiler, so the definition -of median in the implementation listing those options in a -different order:    

+

Type definitions that show up in an mli need to match up with corresponding definitions in the ml. Consider again the example of the type median. The order of the declaration of variants matters to the OCaml compiler, so the definition of median in the implementation listing those options in a different order:    

(** Represents the median computed from a set of strings. In the case
     where there is an even number of choices, the one before and after
@@ -885,28 +593,13 @@ 
Type Definition Mismatches

 >       File "counter.ml", lines 17-19, characters 0-39: Actual declaration
 [1]
-

Order is similarly important to other type declarations, including -the order in which record fields are declared and the order of arguments -(including labeled and optional arguments) to a function.

+

Order is similarly important to other type declarations, including the order in which record fields are declared and the order of arguments (including labeled and optional arguments) to a function.

Cyclic Dependencies

-

In most cases, OCaml doesn’t allow cyclic dependencies, i.e., a -collection of definitions that all refer to one another. If you want to -create such definitions, you typically have to mark them specially. For -example, when defining a set of mutually recursive values (like the -definition of is_even and is_odd in Chapter 2, Recursive Functions), you need to define them using -let rec rather than ordinary let.     

-

The same is true at the module level. By default, cyclic dependencies -between modules are not allowed, and cyclic dependencies among files are -never allowed. Recursive modules are possible but are a rare case, and -we won’t discuss them further here.

-

The simplest example of a forbidden circular reference is a module -referring to its own module name. So, if we tried to add a reference to -Counter from within counter.ml.

+

In most cases, OCaml doesn’t allow cyclic dependencies, i.e., a collection of definitions that all refer to one another. If you want to create such definitions, you typically have to mark them specially. For example, when defining a set of mutually recursive values (like the definition of is_even and is_odd in Chapter 2, Recursive Functions), you need to define them using let rec rather than ordinary let.     

+

The same is true at the module level. By default, cyclic dependencies between modules are not allowed, and cyclic dependencies among files are never allowed. Recursive modules are possible but are a rare case, and we won’t discuss them further here.

+

The simplest example of a forbidden circular reference is a module referring to its own module name. So, if we tried to add a reference to Counter from within counter.ml.

let singleton l = Counter.touch Counter.empty
@@ -919,15 +612,11 @@

Cyclic Dependencies

>Error: The module Counter is an alias for module Dune__exe__Counter, which is the current compilation unit [1]
-

The problem manifests in a different way if we create cyclic -references between files. We could create such a situation by adding a -reference to Freq from counter.ml, e.g., by -adding the following line.

+

The problem manifests in a different way if we create cyclic references between files. We could create such a situation by adding a reference to Freq from counter.ml, e.g., by adding the following line.

let _build_counts = Freq.build_counts
-

In this case, dune will notice the error and complain -explicitly about the cycle:

+

In this case, dune will notice the error and complain explicitly about the cycle:

dune build freq.exe
 >Error: Dependency cycle between the following files:
@@ -940,122 +629,40 @@ 
Cyclic Dependencies

Designing with Modules

-

The module system is a key part of how an OCaml program is -structured. As such, we’ll close this chapter with some advice on how to -think about designing that structure effectively.

+

The module system is a key part of how an OCaml program is structured. As such, we’ll close this chapter with some advice on how to think about designing that structure effectively.

Expose Concrete Types Rarely

-

When designing an mli, one choice that you need to make -is whether to expose the concrete definition of your types or leave them -abstract. Most of the time, abstraction is the right choice, for two -reasons: it enhances the flexibility of your design, and it makes it -possible to enforce invariants on the use of your module.

-

Abstraction enhances flexibility by restricting how users can -interact with your types, thus reducing the ways in which users can -depend on the details of your implementation. If you expose types -explicitly, then users can depend on any and every detail of the types -you choose. If they’re abstract, then only the specific operations you -want to expose are available. This means that you can freely change the -implementation without affecting clients, as long as you preserve the -semantics of those operations.

-

In a similar way, abstraction allows you to enforce invariants on -your types. If your types are exposed, then users of the module can -create new instances of that type (or if mutable, modify existing -instances) in any way allowed by the underlying type. That may violate a -desired invariant i.e., a property about your type that is -always supposed to be true. Abstract types allow you to protect -invariants by making sure that you only expose functions that preserve -your invariants.

-

Despite these benefits, there is a trade-off here. In particular, -exposing types concretely makes it possible to use pattern-matching with -those types, which as we saw in Chapter 3, Lists -And Patterns is a powerful and important tool. You should generally -only expose the concrete implementation of your types when there’s -significant value in the ability to pattern match, and when the -invariants that you care about are already enforced by the data type -itself.

+

When designing an mli, one choice that you need to make is whether to expose the concrete definition of your types or leave them abstract. Most of the time, abstraction is the right choice, for two reasons: it enhances the flexibility of your design, and it makes it possible to enforce invariants on the use of your module.

+

Abstraction enhances flexibility by restricting how users can interact with your types, thus reducing the ways in which users can depend on the details of your implementation. If you expose types explicitly, then users can depend on any and every detail of the types you choose. If they’re abstract, then only the specific operations you want to expose are available. This means that you can freely change the implementation without affecting clients, as long as you preserve the semantics of those operations.

+

In a similar way, abstraction allows you to enforce invariants on your types. If your types are exposed, then users of the module can create new instances of that type (or if mutable, modify existing instances) in any way allowed by the underlying type. That may violate a desired invariant i.e., a property about your type that is always supposed to be true. Abstract types allow you to protect invariants by making sure that you only expose functions that preserve your invariants.

+

Despite these benefits, there is a trade-off here. In particular, exposing types concretely makes it possible to use pattern-matching with those types, which as we saw in Chapter 3, Lists And Patterns is a powerful and important tool. You should generally only expose the concrete implementation of your types when there’s significant value in the ability to pattern match, and when the invariants that you care about are already enforced by the data type itself.

Design for the Call Site

-

When writing an interface, you should think not just about how easy -it is to understand the interface for someone who reads your carefully -documented mli file, but more importantly, you want the -call to be as obvious as possible for someone who is reading it at the -call site.

-

The reason for this is that most of the time, people interacting with -your API will be doing so by reading and modifying code that uses the -API, not by reading the interface definition. By making your API as -obvious as possible from that perspective, you simplify the lives of -your users.

-

There are many ways of improving readability of client code. One -example is labeled arguments (discussed in Chapter 2, Labeled Arguments), which act as documentation that -is available at the call site.

-

You can also improve readability simply by choosing good names for -your functions, variant tags and record fields. Good names aren’t always -long, to be clear. If you wanted to write an anonymous function for -doubling a number: (fun x -> x * 2), a short variable -name like x is best. A good rule of thumb is that names -that have a small scope should be short, whereas names that have a large -scope, like the name of a function in a module interface, should be -longer and more descriptive.

-

There is of course a tradeoff here, in that making your APIs more -explicit tends to make them more verbose as well. Another useful rule of -thumb is that more rarely used names should be longer and more explicit, -since the cost of verbosity goes down and the benefit of explicitness -goes up the less often a name is used.

+

When writing an interface, you should think not just about how easy it is to understand the interface for someone who reads your carefully documented mli file, but more importantly, you want the call to be as obvious as possible for someone who is reading it at the call site.

+

The reason for this is that most of the time, people interacting with your API will be doing so by reading and modifying code that uses the API, not by reading the interface definition. By making your API as obvious as possible from that perspective, you simplify the lives of your users.

+

There are many ways of improving readability of client code. One example is labeled arguments (discussed in Chapter 2, Labeled Arguments), which act as documentation that is available at the call site.

+

You can also improve readability simply by choosing good names for your functions, variant tags and record fields. Good names aren’t always long, to be clear. If you wanted to write an anonymous function for doubling a number: (fun x -> x * 2), a short variable name like x is best. A good rule of thumb is that names that have a small scope should be short, whereas names that have a large scope, like the name of a function in a module interface, should be longer and more descriptive.

+

There is of course a tradeoff here, in that making your APIs more explicit tends to make them more verbose as well. Another useful rule of thumb is that more rarely used names should be longer and more explicit, since the cost of verbosity goes down and the benefit of explicitness goes up the less often a name is used.

Create Uniform Interfaces

-

Designing the interface of a module is a task that should not be -thought of in isolation. The interfaces that appear in your codebase -should play together harmoniously. Part of achieving that is -standardizing aspects of those interfaces.

-

Base, Core and related libraries have been -designed with a uniform set of standards in mind around the design of -module interfaces. Here are some of the guidelines that they use.

+

Designing the interface of a module is a task that should not be thought of in isolation. The interfaces that appear in your codebase should play together harmoniously. Part of achieving that is standardizing aspects of those interfaces.

+

Base, Core and related libraries have been designed with a uniform set of standards in mind around the design of module interfaces. Here are some of the guidelines that they use.

    -

  • A module for (almost) every type. You should mint a -module for almost every type in your program, and the primary type of a -given module should be called t.

    • -

  • Put t first. If you have a module -M whose primary type is M.t, the functions in -M that take a value of type M.t should take it -as their first argument.

    • -

  • Functions that routinely throw an exception should end in -_exn. Otherwise, errors should be signaled by returning an -option or an Or_error.t (both of which are -discussed in Chapter 7, Error Handling ).

    • +

  • A module for (almost) every type. You should mint a module for almost every type in your program, and the primary type of a given module should be called t.

    • +

  • Put t first. If you have a module M whose primary type is M.t, the functions in M that take a value of type M.t should take it as their first argument.

    • +

  • Functions that routinely throw an exception should end in _exn. Otherwise, errors should be signaled by returning an option or an Or_error.t (both of which are discussed in Chapter 7, Error Handling ).

-

There are also standards in Base about what the type signature for -specific functions should be. For example, the signature for -map is always essentially the same, no matter what the -underlying type it is applied to. This kind of function-by-function API -uniformity is achieved through the use of signature includes, -which allow for different modules to share components of their -interface. This approach is described in Chapter 10, Using -Multiple Interfaces.

-

Base’s standards may or may not fit your projects, but you can -improve the usability of your codebase by finding some consistent set of -standards to apply.

+

There are also standards in Base about what the type signature for specific functions should be. For example, the signature for map is always essentially the same, no matter what the underlying type it is applied to. This kind of function-by-function API uniformity is achieved through the use of signature includes, which allow for different modules to share components of their interface. This approach is described in Chapter 10, Using Multiple Interfaces.

+

Base’s standards may or may not fit your projects, but you can improve the usability of your codebase by finding some consistent set of standards to apply.

Interfaces Before Implementations

-

OCaml’s concise and flexible type language enables a type-oriented -approach to software design. Such an approach involves thinking through -and writing out the types you’re going to use before embarking on the -implementation itself.

-

This is a good approach both when working in the core language, where -you would write your type definitions before writing the logic of your -computations, as well as at the module level, where you would write a -first draft of your mli before working on the -ml.

-

Of course, the design process goes in both directions. You’ll often -find yourself going back and modifying your types in response to things -you learn by working on the implementation. But types and signatures -provide a lightweight tool for constructing a skeleton of your design in -a way that helps clarify your goals and intent, before you spend a lot -of time and effort fleshing it out.

+

OCaml’s concise and flexible type language enables a type-oriented approach to software design. Such an approach involves thinking through and writing out the types you’re going to use before embarking on the implementation itself.

+

This is a good approach both when working in the core language, where you would write your type definitions before writing the logic of your computations, as well as at the module level, where you would write a first draft of your mli before working on the ml.

+

Of course, the design process goes in both directions. You’ll often find yourself going back and modifying your types in response to things you learn by working on the implementation. But types and signatures provide a lightweight tool for constructing a skeleton of your design in a way that helps clarify your goals and intent, before you spend a lot of time and effort fleshing it out.

-

Next: Chapter 05Records

 \ No newline at end of file +

Next: Chapter 05Records

 \ No newline at end of file diff --git a/first-class-modules.html b/first-class-modules.html index 1c57ee6ac..1499d9e8e 100644 --- a/first-class-modules.html +++ b/first-class-modules.html @@ -1,29 +1,14 @@ -First-Class Modules - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+First-Class Modules - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

First-Class Modules

-

You can think of OCaml as being broken up into two parts: a core -language that is concerned with values and types, and a module language -that is concerned with modules and module signatures. These sublanguages -are stratified, in that modules can contain types and values, but -ordinary values can’t contain modules or module types. That means you -can’t do things like define a variable whose value is a module, or a -function that takes a module as an argument.  

-

OCaml provides a way around this stratification in the form of -first-class modules. First-class modules are ordinary values -that can be created from and converted back to regular modules.  

-

First-class modules are a sophisticated technique, and you’ll need to -get comfortable with some advanced aspects of the language to use them -effectively. But it’s worth learning, because letting modules into the -core language is quite powerful, increasing the range of what you can -express and making it easier to build flexible and modular systems.

+

You can think of OCaml as being broken up into two parts: a core language that is concerned with values and types, and a module language that is concerned with modules and module signatures. These sublanguages are stratified, in that modules can contain types and values, but ordinary values can’t contain modules or module types. That means you can’t do things like define a variable whose value is a module, or a function that takes a module as an argument.  

+

OCaml provides a way around this stratification in the form of first-class modules. First-class modules are ordinary values that can be created from and converted back to regular modules.  

+

First-class modules are a sophisticated technique, and you’ll need to get comfortable with some advanced aspects of the language to use them effectively. But it’s worth learning, because letting modules into the core language is quite powerful, increasing the range of what you can express and making it easier to build flexible and modular systems.

Working with First-Class Modules

-

We’ll start out by covering the basic mechanics of first-class -modules by working through some toy examples. We’ll get to more -realistic examples in the next section.

+

We’ll start out by covering the basic mechanics of first-class modules by working through some toy examples. We’ll get to more realistic examples in the next section.

Creating First-Class Modules

-

In that light, consider the following signature of a module with a -single integer variable:

+

In that light, consider the following signature of a module with a single integer variable:

open Base;;
 module type X_int = sig val x : int end;;
@@ -38,14 +23,11 @@ 
Creating First-Class Modules

 >- : int = 3
-

A first-class module is created by packaging up a module with a -signature that it satisfies. This is done using the module -keyword.  

+

A first-class module is created by packaging up a module with a signature that it satisfies. This is done using the module keyword.  

(module <Module> : <Module_type>)
-

We can convert Three into a first-class module as -follows:

+

We can convert Three into a first-class module as follows:

let three = (module Three : X_int);;
 >val three : (module X_int) = <module>
@@ -54,8 +36,7 @@ 
Creating First-Class Modules

Inference and Anonymous Modules

-

The module type doesn’t need to be part of the construction of a -first-class module if it can be inferred. Thus, we can write:

+

The module type doesn’t need to be part of the construction of a first-class module if it can be inferred. Thus, we can write:

module Four = struct let x = 4 end;;
 >module Four : sig val x : int end
@@ -72,9 +53,7 @@ 
Inference and Anonymous Modules

Unpacking First-Class Modules

-

In order to access the contents of a first-class module, you need to -unpack it into an ordinary module. This can be done using the -val keyword, using this syntax:

+

In order to access the contents of a first-class module, you need to unpack it into an ordinary module. This can be done using the val keyword, using this syntax:

(val <first_class_module> : <Module_type>)
@@ -89,12 +68,7 @@

Unpacking First-Class Modules

Functions for Manipulating First-Class Modules

-

We can also write ordinary functions which consume and create -first-class modules. The following shows the definition of two -functions: to_int, which converts a -(module X_int) into an int; and -plus, which returns the sum of two -(module X_int):

+

We can also write ordinary functions which consume and create first-class modules. The following shows the definition of two functions: to_int, which converts a (module X_int) into an int; and plus, which returns the sum of two (module X_int):

let to_int m =
   let module M = (val m : X_int) in
@@ -107,16 +81,13 @@ 
Functions for Manipulating First-Class Modules

 >val plus : (module X_int) -> (module X_int) -> (module X_int) = <fun>
-

You can also unpack a first-class module with a pattern match, which -lets us write to_int more concisely:

+

You can also unpack a first-class module with a pattern match, which lets us write to_int more concisely:

let to_int (module M : X_int) = M.x;;
 >val to_int : (module X_int) -> int = <fun>
-

With these functions in hand, we can now work with values of type -(module X_int) in a more natural style, taking advantage of -the concision and simplicity of the core language:

+

With these functions in hand, we can now work with values of type (module X_int) in a more natural style, taking advantage of the concision and simplicity of the core language:

let six = plus three three;;
 >val six : (module X_int) = <module>
@@ -127,10 +98,7 @@ 
Functions for Manipulating First-Class Modules

Richer First-Class Modules

-

First-class modules can contain types and functions in addition to -simple values like int. Here’s an interface that contains a -type and a corresponding bump operation that takes a value -of the type and produces a new one:

+

First-class modules can contain types and functions in addition to simple values like int. Here’s an interface that contains a type and a corresponding bump operation that takes a value of the type and produces a new one:

module type Bumpable = sig
   type t
@@ -139,8 +107,7 @@ 
Richer First-Class Modules

 >module type Bumpable = sig type t val bump : t -> t end
-

We can create multiple instances of this module with different -underlying types:

+

We can create multiple instances of this module with different underlying types:

module Int_bumper = struct
   type t = int
@@ -163,10 +130,7 @@ 
Richer First-Class Modules

Exposing types

-

You can’t do much with int_bumper because it’s fully -abstract, so we can’t take advantage of the fact that the type in -question is int, which makes it impossible to construct or -really do anything with values of type Bumper.t.

+

You can’t do much with int_bumper because it’s fully abstract, so we can’t take advantage of the fact that the type in question is int, which makes it impossible to construct or really do anything with values of type Bumper.t.

let (module Bumper) = int_bumper in
 Bumper.bump 3;;
@@ -175,10 +139,7 @@ 
Exposing types

 >         Bumper.t
-

To make int_bumper usable, we need to expose that the -type Bumpable.t is actually equal to int. -Below we’ll do that for int_bumper, and also provide the -corresponding definition for float_bumper.

+

To make int_bumper usable, we need to expose that the type Bumpable.t is actually equal to int. Below we’ll do that for int_bumper, and also provide the corresponding definition for float_bumper.

let int_bumper = (module Int_bumper : Bumpable with type t = int);;
 >val int_bumper : (module Bumpable with type t = int) = <module>
@@ -186,9 +147,7 @@ 
Exposing types

 >val float_bumper : (module Bumpable with type t = float) = <module>
-

The addition of the sharing constraint has exposed the type -t, which lets us actually use the values within the -module.

+

The addition of the sharing constraint has exposed the type t, which lets us actually use the values within the module.

let (module Bumper) = int_bumper in
 Bumper.bump 3;;
@@ -198,12 +157,7 @@ 
Exposing types

 >- : float = 4.5
-

We can also use these first-class modules polymorphically. The -following function takes two arguments: a Bumpable module -and a list of elements of the same type as the type t of -the module:   

+

We can also use these first-class modules polymorphically. The following function takes two arguments: a Bumpable module and a list of elements of the same type as the type t of the module:   

let bump_list
       (type a)
@@ -215,17 +169,8 @@ 
Exposing types

 >  <fun>
-

In this example, a is a locally abstract type. -For any function, you can declare a pseudoparameter of the form -(type a) which introduces a fresh type named -a. This type acts like an abstract type within the context -of the function. In the example above, the locally abstract type was -used as part of a sharing constraint that ties the type B.t -with the type of the elements of the list passed in.      

-

The resulting function is polymorphic in both the type of the list -element and the type Bumpable.t. We can see this function -in action:

+

In this example, a is a locally abstract type. For any function, you can declare a pseudoparameter of the form (type a) which introduces a fresh type named a. This type acts like an abstract type within the context of the function. In the example above, the locally abstract type was used as part of a sharing constraint that ties the type B.t with the type of the elements of the list passed in.      

+

The resulting function is polymorphic in both the type of the list element and the type Bumpable.t. We can see this function in action:

bump_list int_bumper [1;2;3];;
 >- : int list = [2; 3; 4]
@@ -233,34 +178,24 @@ 
Exposing types

 >- : float list = [2.5; 3.5; 4.5]
-

Polymorphic first-class modules are important because they allow you -to connect the types associated with a first-class module to the types -of other values you’re working with.

-
-

More on Locally Abstract Types

-

One of the key properties of locally abstract types is that they’re -dealt with as abstract types in the function they’re defined within, but -are polymorphic from the outside. Consider the following example:  

+

Polymorphic first-class modules are important because they allow you to connect the types associated with a first-class module to the types of other values you’re working with.

+
+

More on Locally Abstract Types

+

One of the key properties of locally abstract types is that they’re dealt with as abstract types in the function they’re defined within, but are polymorphic from the outside. Consider the following example:  

let wrap_in_list (type a) (x:a) = [x];;
 >val wrap_in_list : 'a -> 'a list = <fun>
-

The type a is used in a way that is compatible with it -being abstract, but the type of the function that is inferred is -polymorphic.

-

If, on the other hand, we try to use the type a as if it -were equivalent to some concrete type, say, int, then the -compiler will complain.

+

The type a is used in a way that is compatible with it being abstract, but the type of the function that is inferred is polymorphic.

+

If, on the other hand, we try to use the type a as if it were equivalent to some concrete type, say, int, then the compiler will complain.

let double_int (type a) (x:a) = x + x;;
 >Line 1, characters 33-34:
 >Error: This expression has type a but an expression was expected of type int
-

One common use of locally abstract types is to create a new type that -can be used in constructing a module. Here’s an example of doing this to -create a new first-class module:

+

One common use of locally abstract types is to create a new type that can be used in constructing a module. Here’s an example of doing this to create a new first-class module:

module type Comparable = sig
   type t
@@ -280,30 +215,15 @@ 
More on Locally Abstract Types

 >- : (module Comparable with type t = float) = <module>
-

This technique is useful beyond first-class modules. For example, we -can use the same approach to construct a local module to be fed to a -functor.

-
+

This technique is useful beyond first-class modules. For example, we can use the same approach to construct a local module to be fed to a functor.

+

Example: A Query-Handling Framework

-

Now let’s look at first-class modules in the context of a more -complete and realistic example. In particular, we’re going to implement -a system for responding to user-generated queries.

-

This system will use s-expressions for formatting queries -and responses, as well as the configuration for the query handler. -S-expressions are a simple, flexible, and human-readable serialization -format commonly used in Base and related libraries. For -now, it’s enough to think of them as balanced parenthetical expressions -whose atomic values are strings, e.g., -(this (is an) (s expression)). S-expressions are covered in -more detail in Chapter 20, Data Serialization With S-Expressions.  

-

The following signature for a module that implements a system for -responding to user-generated queries. Here, we use Base’s -Sexp module for handling s-expressions. Note that we could -just as easily have used another serialization format, like JSON, as -discussed in Chapter 18, Handling JSON Data.  

+

Now let’s look at first-class modules in the context of a more complete and realistic example. In particular, we’re going to implement a system for responding to user-generated queries.

+

This system will use s-expressions for formatting queries and responses, as well as the configuration for the query handler. S-expressions are a simple, flexible, and human-readable serialization format commonly used in Base and related libraries. For now, it’s enough to think of them as balanced parenthetical expressions whose atomic values are strings, e.g., (this (is an) (s expression)). S-expressions are covered in more detail in Chapter 20, Data Serialization With S-Expressions.  

+

The following signature for a module that implements a system for responding to user-generated queries. Here, we use Base’s Sexp module for handling s-expressions. Note that we could just as easily have used another serialization format, like JSON, as discussed in Chapter 18, Handling JSON Data.  

module type Query_handler = sig
 
@@ -327,21 +247,12 @@ 
Example: A Query-Handling Framework

   val eval : t -> Sexp.t -> Sexp.t Or_error.t
 end;;
-

Implementing s-expression converters by hand is tedious and -error-prone, but happily, we have an alternative. -ppx_sexp_conv is a syntax extension which can be used to -automatically generate s-expression converters based on their type -definition. We’ll enable ppx_sexp_conv by enabling -ppx_jane, which brings in a larger family of syntax -extensions.   -     

+

Implementing s-expression converters by hand is tedious and error-prone, but happily, we have an alternative. ppx_sexp_conv is a syntax extension which can be used to automatically generate s-expression converters based on their type definition. We’ll enable ppx_sexp_conv by enabling ppx_jane, which brings in a larger family of syntax extensions.        

#require "ppx_jane";;
-

Here’s an example of the extension in action. Note that we need the -annotation [@@deriving sexp] to kick off the generation of -the converters.

+

Here’s an example of the extension in action. Note that we need the annotation [@@deriving sexp] to kick off the generation of the converters.

type u = { a: int; b: float } [@@deriving sexp];;
 >type u = { a : int; b : float; }
@@ -353,8 +264,7 @@ 
Example: A Query-Handling Framework

 >- : u = {a = 43; b = 3.4}
-

The same annotations can be attached within a signature to add the -appropriate type signature.

+

The same annotations can be attached within a signature to add the appropriate type signature.

module type M = sig type t [@@deriving sexp] end;;
 >module type M =
@@ -363,12 +273,7 @@ 
Example: A Query-Handling Framework

Implementing a Query Handler

-

Now we can construct an example of a query handler that satisfies the -Query_handler interface. We’ll start with a handler that -produces unique integer IDs, which works by keeping an internal counter -that’s bumped every time a new value is requested. The input to the -query in this case is just the trivial s-expression (), -otherwise known as Sexp.unit:  

+

Now we can construct an example of a query handler that satisfies the Query_handler interface. We’ll start with a handler that produces unique integer IDs, which works by keeping an internal counter that’s bumped every time a new value is requested. The input to the query in this case is just the trivial s-expression (), otherwise known as Sexp.unit:  

module Unique = struct
   type config = int [@@deriving sexp]
@@ -386,8 +291,7 @@ 
Implementing a Query Handler

       response
 end;;
-

We can use this module to create an instance of the -Unique query handler and interact with it directly:

+

We can use this module to create an instance of the Unique query handler and interact with it directly:

let unique = Unique.create 0;;
 >val unique : Unique.t = {Unique.next_id = 0}
@@ -397,9 +301,7 @@ 
Implementing a Query Handler

 >- : (Sexp.t, Error.t) result = Ok 1
-

Here’s another example: a query handler that does directory listings. -Here, the config is the default directory that relative paths are -interpreted within:

+

Here’s another example: a query handler that does directory listings. Here, the config is the default directory that relative paths are interpreted within:

#require "core_unix.sys_unix";;
@@ -428,8 +330,7 @@ 
Implementing a Query Handler

       Ok (Array.sexp_of_t String.sexp_of_t (Sys_unix.readdir dir))
 end;;
-

Again, we can create an instance of this query handler and interact -with it directly:

+

Again, we can create an instance of this query handler and interact with it directly:

let list_dir = List_dir.create "/var";;
 >val list_dir : List_dir.t = {List_dir.cwd = "/var"}
@@ -445,14 +346,7 @@ 
Implementing a Query Handler

Dispatching to Multiple Query Handlers

-

Now, what if we want to dispatch queries to any of an arbitrary -collection of handlers? Ideally, we’d just like to pass in the handlers -as a simple data structure like a list. This is awkward to do with -modules and functors alone, but it’s quite natural with first-class -modules. The first thing we’ll need to do is create a signature that -combines a Query_handler module with an instantiated query -handler: 

+

Now, what if we want to dispatch queries to any of an arbitrary collection of handlers? Ideally, we’d just like to pass in the handlers as a simple data structure like a list. This is awkward to do with modules and functors alone, but it’s quite natural with first-class modules. The first thing we’ll need to do is create a signature that combines a Query_handler module with an instantiated query handler: 

module type Query_handler_instance = sig
   module Query_handler : Query_handler
@@ -462,9 +356,7 @@ 
Dispatching to Multiple Query Handlers

 >  sig module Query_handler : Query_handler val this : Query_handler.t end
-

With this signature, we can create a first-class module that -encompasses both an instance of the query and the matching operations -for working with that query.

+

With this signature, we can create a first-class module that encompasses both an instance of the query and the matching operations for working with that query.

We can create an instance as follows:

let unique_instance =
@@ -475,9 +367,7 @@ 
Dispatching to Multiple Query Handlers

 >val unique_instance : (module Query_handler_instance) = <module>
-

Constructing instances in this way is a little verbose, but we can -write a function that eliminates most of this boilerplate. Note that we -are again making use of a locally abstract type:

+

Constructing instances in this way is a little verbose, but we can write a function that eliminates most of this boilerplate. Note that we are again making use of a locally abstract type:

let build_instance
       (type a)
@@ -493,8 +383,7 @@ 
Dispatching to Multiple Query Handlers

 >  'a -> (module Query_handler_instance) = <fun>
-

Using build_instance, constructing a new instance -becomes a one-liner:

+

Using build_instance, constructing a new instance becomes a one-liner:

let unique_instance = build_instance (module Unique) 0;;
 >val unique_instance : (module Query_handler_instance) = <module>
@@ -502,17 +391,12 @@ 
Dispatching to Multiple Query Handlers

 >val list_dir_instance : (module Query_handler_instance) = <module>
-

We can now write code that lets you dispatch queries to one of a list -of query handler instances. We assume that the shape of the query is as -follows:

+

We can now write code that lets you dispatch queries to one of a list of query handler instances. We assume that the shape of the query is as follows:

(query-name query)
-

where query-name is the name used to determine -which query handler to dispatch the query to, and -query is the body of the query.

-

The first thing we’ll need is a function that takes a list of query -handler instances and constructs a dispatch table from it:

+

where query-name is the name used to determine which query handler to dispatch the query to, and query is the body of the query.

+

The first thing we’ll need is a function that takes a list of query handler instances and constructs a dispatch table from it:

let build_dispatch_table handlers =
   let table = Hashtbl.create (module String) in
@@ -525,8 +409,7 @@ 
Dispatching to Multiple Query Handlers

 >  (string, (module Query_handler_instance)) Hashtbl.Poly.t = <fun>
-

Next, we’ll need a function that dispatches to a handler using a -dispatch table:

+

Next, we’ll need a function that dispatches to a handler using a dispatch table:

let dispatch dispatch_table name_and_query =
   match name_and_query with
@@ -545,19 +428,9 @@ 
Dispatching to Multiple Query Handlers

 >  Sexp.t -> Sexp.t Or_error.t = <fun>
-

This function interacts with an instance by unpacking it into a -module I and then using the query handler instance -(I.this) in concert with the associated module -(I.Query_handler).

-

The bundling together of the module and the value is in many ways -reminiscent of object-oriented languages. One key difference is that -first-class modules allow you to package up more than just functions or -methods. As we’ve seen, you can also include types and even modules. -We’ve only used it in a small way here, but this extra power allows you -to build more sophisticated components that involve multiple -interdependent types and values.

-

We can turn this into a complete, running example by adding a -command-line interface:

+

This function interacts with an instance by unpacking it into a module I and then using the query handler instance (I.this) in concert with the associated module (I.Query_handler).

+

The bundling together of the module and the value is in many ways reminiscent of object-oriented languages. One key difference is that first-class modules allow you to package up more than just functions or methods. As we’ve seen, you can also include types and even modules. We’ve only used it in a small way here, but this extra power allows you to build more sophisticated components that involve multiple interdependent types and values.

+

We can turn this into a complete, running example by adding a command-line interface:

open Stdio;;
 let rec cli dispatch_table =
@@ -586,9 +459,7 @@ 
Dispatching to Multiple Query Handlers

 >  <fun>
-

We’ll run this command-line interface from a standalone program by -putting the above code in a file, and adding the following to launch the -interface.

+

We’ll run this command-line interface from a standalone program by putting the above code in a file, and adding the following to launch the interface.

let () =
   cli (build_dispatch_table [unique_instance; list_dir_instance])
@@ -611,15 +482,8 @@

Dispatching to Multiple Query Handlers

Loading and Unloading Query Handlers

-

One of the advantages of first-class modules is that they afford a -great deal of dynamism and flexibility. For example, it’s a fairly -simple matter to change our design to allow query handlers to be loaded -and unloaded at runtime.  

-

We’ll do this by creating a query handler whose job is to control the -set of active query handlers. The module in question will be called -Loader, and its configuration is a list of known -Query_handler modules. Here are the basic types:

+

One of the advantages of first-class modules is that they afford a great deal of dynamism and flexibility. For example, it’s a fairly simple matter to change our design to allow query handlers to be loaded and unloaded at runtime.  

+

We’ll do this by creating a query handler whose job is to control the set of active query handlers. The module in question will be called Loader, and its configuration is a list of known Query_handler modules. Here are the basic types:

module Loader = struct
   type config = (module Query_handler) list [@sexp.opaque]
@@ -631,14 +495,8 @@ 
Loading and Unloading Query Handlers

 
   let name = "loader"
-

Note that a Loader.t has two tables: one containing the -known query handler modules, and one containing the active query handler -instances. The Loader.t will be responsible for creating -new instances and adding them to the table, as well as for removing -instances, all in response to user queries.

-

Next, we’ll need a function for creating a Loader.t. -This function requires the list of known query handler modules. Note -that the table of active modules starts out as empty:

+

Note that a Loader.t has two tables: one containing the known query handler modules, and one containing the active query handler instances. The Loader.t will be responsible for creating new instances and adding them to the table, as well as for removing instances, all in response to user queries.

+

Next, we’ll need a function for creating a Loader.t. This function requires the list of known query handler modules. Note that the table of active modules starts out as empty:

let create known_list =
     let active = String.Table.create () in
@@ -648,13 +506,7 @@ 
Loading and Unloading Query Handlers

         Hashtbl.set known ~key:Q.name ~data:q);
     { known; active }
-

Now we can write the functions for manipulating the table of active -query handlers. We’ll start with the function for loading an instance. -Note that it takes as an argument both the name of the query handler and -the configuration for instantiating that handler in the form of an -s-expression. These are used for creating a first-class module of type -(module Query_handler_instance), which is then added to the -active table:

+

Now we can write the functions for manipulating the table of active query handlers. We’ll start with the function for loading an instance. Note that it takes as an argument both the name of the query handler and the configuration for instantiating that handler in the form of an s-expression. These are used for creating a first-class module of type (module Query_handler_instance), which is then added to the active table:

let load t handler_name config =
     if Hashtbl.mem t.active handler_name then
@@ -674,9 +526,7 @@ 
Loading and Unloading Query Handlers

         Hashtbl.set t.active ~key:handler_name ~data:instance;
         Ok Sexp.unit
-

Since the load function will refuse to load -an already active handler, we also need the ability to unload a handler. -Note that the handler explicitly refuses to unload itself:

+

Since the load function will refuse to load an already active handler, we also need the ability to unload a handler. Note that the handler explicitly refuses to unload itself:

let unload t handler_name =
     if not (Hashtbl.mem t.active handler_name) then
@@ -688,10 +538,7 @@ 
Loading and Unloading Query Handlers

       Ok Sexp.unit
     )
-

Finally, we need to implement the eval function, which -will determine the query interface presented to the user. We’ll do this -by creating a variant type, and using the s-expression converter -generated for that type to parse the query from the user:

+

Finally, we need to implement the eval function, which will determine the query interface presented to the user. We’ll do this by creating a variant type, and using the s-expression converter generated for that type to parse the query from the user:

type request =
     | Load of string * Sexp.t
@@ -700,14 +547,8 @@ 
Loading and Unloading Query Handlers

     | Active_services
   [@@deriving sexp]
-

The eval function itself is fairly straightforward, -dispatching to the appropriate functions to respond to each type of -query. Note that we write -<:sexp_of<string list>> to autogenerate a -function for converting a list of strings to an s-expression, as -described in Chapter 20, Data Serialization With S Expressions.

-

This function ends the definition of the Loader -module:

+

The eval function itself is fairly straightforward, dispatching to the appropriate functions to respond to each type of query. Note that we write <:sexp_of<string list>> to autogenerate a function for converting a list of strings to an s-expression, as described in Chapter 20, Data Serialization With S Expressions.

+

This function ends the definition of the Loader module:

let eval t sexp =
     match Or_error.try_with (fun () -> request_of_sexp sexp) with
@@ -722,10 +563,7 @@ 
Loading and Unloading Query Handlers

         Ok ([%sexp_of: string list] (Hashtbl.keys t.active))
 end
-

Finally, we can put this all together with the command-line -interface. We first create an instance of the loader query handler and -then add that instance to the loader’s active table. We can then launch -the command-line interface, passing it the active table.

+

Finally, we can put this all together with the command-line interface. We first create an instance of the loader query handler and then add that instance to the loader’s active table. We can then launch the command-line interface, passing it the active table.

let () =
   let loader = Loader.create [(module Unique); (module List_dir)] in
@@ -739,11 +577,7 @@ 
Loading and Unloading Query Handlers

     ~key:Loader.name ~data:loader_instance;
   cli loader.active
-

The resulting command-line interface behaves much as you’d expect, -starting out with no query handlers available but giving you the ability -to load and unload them. Here’s an example of it in action. As you can -see, we start out with loader itself as the only active -handler.

+

The resulting command-line interface behaves much as you’d expect, starting out with no query handlers available but giving you the ability to load and unload them. Here’s an example of it in action. As you can see, we start out with loader itself as the only active handler.

dune exec -- ./query_handler_loader.exe
 >>>> (loader known_services)
@@ -757,10 +591,7 @@ 
Loading and Unloading Query Handlers

 
>>> (ls .)
 Could not find matching handler: ls
-

But, we can load the ls handler with a config of our -choice, at which point it will be available for use. And once we unload -it, it will be unavailable yet again and could be reloaded with a -different config.

+

But, we can load the ls handler with a config of our choice, at which point it will be available for use. And once we unload it, it will be unavailable yet again and could be reloaded with a different config.

>>> (loader (load ls /var))
 ()
@@ -772,26 +603,17 @@ 
Loading and Unloading Query Handlers

 >>> (ls .)
 Could not find matching handler: ls
-

Notably, the loader can’t be loaded (since it’s not on the list of -known handlers) and can’t be unloaded either:

+

Notably, the loader can’t be loaded (since it’s not on the list of known handlers) and can’t be unloaded either:

>>> (loader (unload loader))
 It's unwise to unload yourself
-

Although we won’t describe the details here, we can push this -dynamism yet further using OCaml’s dynamic linking facilities, which -allow you to compile and link in new code to a running program. This can -be automated using libraries like ocaml_plugin, which can -be installed via OPAM, and which takes care of much of the workflow -around setting up dynamic linking.

+

Although we won’t describe the details here, we can push this dynamism yet further using OCaml’s dynamic linking facilities, which allow you to compile and link in new code to a running program. This can be automated using libraries like ocaml_plugin, which can be installed via OPAM, and which takes care of much of the workflow around setting up dynamic linking.

Living Without First-Class Modules

-

It’s worth noting that most designs that can be done with first-class -modules can be simulated without them, with some level of awkwardness. -For example, we could rewrite our query handler example without -first-class modules using the following types: 

+

It’s worth noting that most designs that can be done with first-class modules can be simulated without them, with some level of awkwardness. For example, we could rewrite our query handler example without first-class modules using the following types: 

type query_handler_instance =
   { name : string
@@ -804,10 +626,7 @@ 
Living Without First-Class Modules

 >type query_handler = Sexp.t -> query_handler_instance
-

The idea here is that we hide the true types of the objects in -question behind the functions stored in the closure. Thus, we could put -the Unique query handler into this framework as -follows:

+

The idea here is that we hide the true types of the objects in question behind the functions stored in the closure. Thus, we could put the Unique query handler into this framework as follows:

let unique_handler config_sexp =
   let config = Unique.config_of_sexp config_sexp in
@@ -818,12 +637,7 @@ 
Living Without First-Class Modules

 >val unique_handler : Sexp.t -> query_handler_instance = <fun>
-

For an example on this scale, the preceding approach is completely -reasonable, and first-class modules are not really necessary. But the -more functionality you need to hide away behind a set of closures, and -the more complicated the relationships between the different types in -question, the more awkward this approach becomes, and the better it is -to use first-class modules.

+

For an example on this scale, the preceding approach is completely reasonable, and first-class modules are not really necessary. But the more functionality you need to hide away behind a set of closures, and the more complicated the relationships between the different types in question, the more awkward this approach becomes, and the better it is to use first-class modules.

-

Next: Chapter 12Objects

 \ No newline at end of file +

Next: Chapter 12Objects

 \ No newline at end of file diff --git a/foreign-function-interface.html b/foreign-function-interface.html index 1c82bca00..c694a25bc 100644 --- a/foreign-function-interface.html +++ b/foreign-function-interface.html @@ -1,63 +1,31 @@ -Foreign Function Interface - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Foreign Function Interface - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Foreign Function Interface

This chapter includes contributions from Jeremy Yallop.

-

OCaml has several options available to interact with non-OCaml code. -The compiler can link with external system libraries via C code and also -can produce standalone native object files that can be embedded within -other non-OCaml applications.   

-

The mechanism by which code in one programming language can invoke -routines in a different programming language is called a foreign -function interface. This chapter will:

+

OCaml has several options available to interact with non-OCaml code. The compiler can link with external system libraries via C code and also can produce standalone native object files that can be embedded within other non-OCaml applications.   

+

The mechanism by which code in one programming language can invoke routines in a different programming language is called a foreign function interface. This chapter will:

    -

  • Show how to call routines in C libraries directly from your OCaml -code

    • -

  • Teach you how to build higher-level abstractions in OCaml from -the low-level C bindings

    • -

  • Work through some full examples for binding a terminal interface -and UNIX date/time functions

    • +

  • Show how to call routines in C libraries directly from your OCaml code

    • +

  • Teach you how to build higher-level abstractions in OCaml from the low-level C bindings

    • +

  • Work through some full examples for binding a terminal interface and UNIX date/time functions

-

The simplest foreign function interface in OCaml doesn’t even require -you to write any C code at all! The Ctypes library lets you define the C -interface in pure OCaml, and the library then takes care of loading the -C symbols and invoking the foreign function call.   

-

Let’s dive straight into a realistic example to show you how the -library looks. We’ll create a binding to the Ncurses terminal toolkit, -as it’s widely available on most systems and doesn’t have any complex -dependencies.

-
-

Installing the Ctypes Library

-

If you want to use Ctypes interactively, you’ll also need to install -the libffi library as a prerequisite to using Ctypes. It’s -a fairly popular library and should be available in your OS package -manager. If you’re using opam 2.1 or higher, it will prompt you to -install it automatically when you install -ctypes-foreign.

+

The simplest foreign function interface in OCaml doesn’t even require you to write any C code at all! The Ctypes library lets you define the C interface in pure OCaml, and the library then takes care of loading the C symbols and invoking the foreign function call.   

+

Let’s dive straight into a realistic example to show you how the library looks. We’ll create a binding to the Ncurses terminal toolkit, as it’s widely available on most systems and doesn’t have any complex dependencies.

+
+

Installing the Ctypes Library

+

If you want to use Ctypes interactively, you’ll also need to install the libffi library as a prerequisite to using Ctypes. It’s a fairly popular library and should be available in your OS package manager. If you’re using opam 2.1 or higher, it will prompt you to install it automatically when you install ctypes-foreign.

$ opam install ctypes ctypes-foreign
 $ utop
 # require "ctypes-foreign" ;;
-

You’ll also need the Ncurses library for the first example. This -comes preinstalled on many operating systems such as macOS, and Debian -Linux provides it as the libncurses5-dev package.

-
+

You’ll also need the Ncurses library for the first example. This comes preinstalled on many operating systems such as macOS, and Debian Linux provides it as the libncurses5-dev package.

+

Example: A Terminal Interface

-

Ncurses is a library to help build terminal-independent text -interfaces in a reasonably efficient way. It’s used in console mail -clients like Mutt and Pine, and console web browsers such as Lynx. 

-

The full C interface is quite large and is explained in the online documentation. We’ll -just use the small excerpt, since we just want to demonstrate Ctypes in -action: 

+

Ncurses is a library to help build terminal-independent text interfaces in a reasonably efficient way. It’s used in console mail clients like Mutt and Pine, and console web browsers such as Lynx. 

+

The full C interface is quite large and is explained in the online documentation. We’ll just use the small excerpt, since we just want to demonstrate Ctypes in action: 

typedef struct _win_st WINDOW;
 typedef unsigned int chtype;
@@ -73,26 +41,10 @@ 
Example: A Terminal Interface

 void    box (WINDOW *, chtype, chtype);
 int     cbreak (void);
-

The Ncurses functions either operate on the current pseudoterminal or -on a window that has been created via newwin. The -WINDOW structure holds the internal library state and is -considered abstract outside of Ncurses. Ncurses clients just need to -store the pointer somewhere and pass it back to Ncurses library calls, -which in turn dereference its contents.

-

Note that there are over 200 library calls in Ncurses, so we’re only -binding a select few for this example. The initscr and -newwin create WINDOW pointers for the global -and subwindows, respectively. The mvwaddrstr takes a -window, x/y offsets, and a string and writes to the screen at that -location. The terminal is only updated after refresh or -wrefresh are called.

-

Ctypes provides an OCaml interface that lets you map these C -functions to equivalent OCaml functions. The library takes care of -converting OCaml function calls and arguments into the C calling -convention, invoking the foreign call within the C library and finally -returning the result as an OCaml value.

-

Let’s begin by defining the basic values we need, starting with the -WINDOW state pointer:

+

The Ncurses functions either operate on the current pseudoterminal or on a window that has been created via newwin. The WINDOW structure holds the internal library state and is considered abstract outside of Ncurses. Ncurses clients just need to store the pointer somewhere and pass it back to Ncurses library calls, which in turn dereference its contents.

+

Note that there are over 200 library calls in Ncurses, so we’re only binding a select few for this example. The initscr and newwin create WINDOW pointers for the global and subwindows, respectively. The mvwaddrstr takes a window, x/y offsets, and a string and writes to the screen at that location. The terminal is only updated after refresh or wrefresh are called.

+

Ctypes provides an OCaml interface that lets you map these C functions to equivalent OCaml functions. The library takes care of converting OCaml function calls and arguments into the C calling convention, invoking the foreign call within the C library and finally returning the result as an OCaml value.

+

Let’s begin by defining the basic values we need, starting with the WINDOW state pointer:

open Ctypes
 
@@ -100,29 +52,18 @@ 
Example: A Terminal Interface

 
 let window : window typ = ptr void
-

We don’t know the internal representation of the window pointer, so -we treat it as a C void pointer. We’ll improve on this later on in the -chapter, but it’s good enough for now. The second statement defines an -OCaml value that represents the WINDOW C pointer. This -value is used later in the Ctypes function definitions:

+

We don’t know the internal representation of the window pointer, so we treat it as a C void pointer. We’ll improve on this later on in the chapter, but it’s good enough for now. The second statement defines an OCaml value that represents the WINDOW C pointer. This value is used later in the Ctypes function definitions:

open Foreign
 
 let initscr = foreign "initscr" (void @-> returning window)
-

That’s all we need to invoke our first function call to -initscr to initialize the terminal. The -foreign function accepts two parameters:

+

That’s all we need to invoke our first function call to initscr to initialize the terminal. The foreign function accepts two parameters:

    -

  • The C function call name, which is looked up using the -dlsym POSIX function.

    • -

  • A value that defines the complete set of C function arguments and -its return type. The @-> operator adds an argument to -the C parameter list, and returning terminates the -parameter list with the return type.

    • +

  • The C function call name, which is looked up using the dlsym POSIX function.

    • +

  • A value that defines the complete set of C function arguments and its return type. The @-> operator adds an argument to the C parameter list, and returning terminates the parameter list with the return type.

-

The remainder of the Ncurses binding simply expands on these -definitions:

+

The remainder of the Ncurses binding simply expands on these definitions:

let newwin =
   foreign "newwin" (int @-> int @-> int @-> int @-> returning window)
@@ -145,20 +86,9 @@ 
Example: A Terminal Interface

 let box = foreign "box" (window @-> char @-> char @-> returning void)
 let cbreak = foreign "cbreak" (void @-> returning int)
-

These definitions are all straightforward mappings from the C -declarations in the Ncurses header file. Note that the -string and int values here are nothing to do -with OCaml type declarations; instead, they are values that come from -opening the Ctypes module at the top of the file.

-

Most of the parameters in the Ncurses example represent fairly simple -scalar C types, except for window (a pointer to the library -state) and string, which maps from OCaml strings that have -a specific length onto C character buffers whose length is defined by a -terminating null character that immediately follows the string data.

-

The module signature for ncurses.mli looks much like a -normal OCaml signature. You can infer it directly from the -ncurses.ml by running a command called -ocaml-print-intf, which you can install with opam.

+

These definitions are all straightforward mappings from the C declarations in the Ncurses header file. Note that the string and int values here are nothing to do with OCaml type declarations; instead, they are values that come from opening the Ctypes module at the top of the file.

+

Most of the parameters in the Ncurses example represent fairly simple scalar C types, except for window (a pointer to the library state) and string, which maps from OCaml strings that have a specific length onto C character buffers whose length is defined by a terminating null character that immediately follows the string data.

+

The module signature for ncurses.mli looks much like a normal OCaml signature. You can infer it directly from the ncurses.ml by running a command called ocaml-print-intf, which you can install with opam.

ocaml-print-intf ncurses.ml
 >type window = unit Ctypes.ptr
@@ -175,13 +105,8 @@ 
Example: A Terminal Interface

 >val cbreak : unit -> int
-

The ocaml-print-intf tool examines the default signature -inferred by the compiler for a module file and prints it out as -human-readable output. You can copy this into a corresponding -mli file and customize it to improve its safety for -external callers by making some of its internals more abstract.

-

Here’s the customized ncurses.mli interface that we can -safely use from other libraries:

+

The ocaml-print-intf tool examines the default signature inferred by the compiler for a module file and prints it out as human-readable output. You can copy this into a corresponding mli file and customize it to improve its safety for external callers by making some of its internals more abstract.

+

Here’s the customized ncurses.mli interface that we can safely use from other libraries:

type window
 
@@ -197,13 +122,8 @@ 
Example: A Terminal Interface

 val box : window -> char -> char -> unit
 val cbreak : unit -> int
-

Note that the window type is now abstract in the -signature, to ensure that window pointers can only be constructed via -the Ncurses.initscr function. This prevents void pointers -obtained from other sources from being mistakenly passed to an Ncurses -library call.

-

Now compile a “hello world” terminal drawing program to tie this all -together:

+

Note that the window type is now abstract in the signature, to ensure that window pointers can only be constructed via the Ncurses.initscr function. This prevents void pointers obtained from other sources from being mistakenly passed to an Ncurses library call.

+

Now compile a “hello world” terminal drawing program to tie this all together:

open Ncurses
 
@@ -220,14 +140,7 @@ 
Example: A Terminal Interface

   Unix.sleep 5;
   endwin ()
-

The hello executable is compiled by linking with the -ctypes-foreign package.   We also add in a -(flags) directive to instruct the compiler to link in the -system ncurses C library to the executable. If you do not -specify the C library in the dune file, then the program may build -successfully, but attempting to invoke the executable will fail as not -all of the dependencies will be available.

+

The hello executable is compiled by linking with the ctypes-foreign package.   We also add in a (flags) directive to instruct the compiler to link in the system ncurses C library to the executable. If you do not specify the C library in the dune file, then the program may build successfully, but attempting to invoke the executable will fail as not all of the dependencies will be available.

(executable
   (name      hello)
@@ -239,86 +152,36 @@ 
Example: A Terminal Interface

 
dune build hello.exe
-

Running hello.exe should now display a Hello World in -your terminal!

-

Ctypes wouldn’t be very useful if it were limited to only defining -simple C types, of course. It provides full support for C pointer -arithmetic, pointer conversions, and reading and writing through -pointers, using OCaml functions as function pointers to C code, as well -as struct and union definitions.

-

We’ll go over some of these features in more detail for the remainder -of the chapter by using some POSIX date functions as running -examples.

-
-

Linking Modes: libffi and Stub Generation

-

The core of ctypes is a set of OCaml combinators for describing the -structure of C types (numeric types, arrays, pointers, structs, unions -and functions). You can then use these combinators to describe the types -of the C functions that you want to call. There are two entirely -distinct ways to actually link to the system libraries that contain the -function definitions: dynamic linking and stub -generation.

-

The ctypes-foreign package used in this chapter uses the -low-level libffi library to dynamically open C libraries, -search for the relevant symbols for the function call being invoked, and -marshal the function parameters according to the operating system’s -application binary interface (ABI). While much of this happens -behind-the-scenes and permits convenient interactive programming while -developing bindings, it is not always the solution you want to use in -production.

-

The ctypes-cstubs package provides an alternative -mechanism to shift much of the linking work to be done once at build -time, instead of doing it on every invocation of the function. It does -this by taking the same OCaml binding descriptions, and -generating intermediate C source files that contain the corresponding -C/OCaml glue code. When these are compiled with a normal dune build, the -generated C code is treated just as any handwritten code might be, and -compiled against the system header files. This allows certain C values -to be used that cannot be dynamically probed (e.g. preprocessor macro -definitions), and can also catch definition errors if there is a C -header mismatch at compile time.

-

C rarely makes life easier though. There are some definitions that -cannot be entirely expressed as static C code (e.g. dynamic function -pointers), and those require the use of ctypes-foreign (and -libffi). Using ctypes does make it possible to share the -majority of definitions across both linking modes, all while avoiding -writing C code directly.

-

While we do not cover the details of C stub generation further in -this chapter, you can read more about how to use this mode in the -“Dealing with foreign libraries” chapter in the dune manual.

-
+

Running hello.exe should now display a Hello World in your terminal!

+

Ctypes wouldn’t be very useful if it were limited to only defining simple C types, of course. It provides full support for C pointer arithmetic, pointer conversions, and reading and writing through pointers, using OCaml functions as function pointers to C code, as well as struct and union definitions.

+

We’ll go over some of these features in more detail for the remainder of the chapter by using some POSIX date functions as running examples.

+
+

Linking Modes: libffi and Stub Generation

+

The core of ctypes is a set of OCaml combinators for describing the structure of C types (numeric types, arrays, pointers, structs, unions and functions). You can then use these combinators to describe the types of the C functions that you want to call. There are two entirely distinct ways to actually link to the system libraries that contain the function definitions: dynamic linking and stub generation.

+

The ctypes-foreign package used in this chapter uses the low-level libffi library to dynamically open C libraries, search for the relevant symbols for the function call being invoked, and marshal the function parameters according to the operating system’s application binary interface (ABI). While much of this happens behind-the-scenes and permits convenient interactive programming while developing bindings, it is not always the solution you want to use in production.

+

The ctypes-cstubs package provides an alternative mechanism to shift much of the linking work to be done once at build time, instead of doing it on every invocation of the function. It does this by taking the same OCaml binding descriptions, and generating intermediate C source files that contain the corresponding C/OCaml glue code. When these are compiled with a normal dune build, the generated C code is treated just as any handwritten code might be, and compiled against the system header files. This allows certain C values to be used that cannot be dynamically probed (e.g. preprocessor macro definitions), and can also catch definition errors if there is a C header mismatch at compile time.

+

C rarely makes life easier though. There are some definitions that cannot be entirely expressed as static C code (e.g. dynamic function pointers), and those require the use of ctypes-foreign (and libffi). Using ctypes does make it possible to share the majority of definitions across both linking modes, all while avoiding writing C code directly.

+

While we do not cover the details of C stub generation further in this chapter, you can read more about how to use this mode in the “Dealing with foreign libraries” chapter in the dune manual.

+

Basic Scalar C Types

-

First, let’s look at how to define basic scalar C types. Every C type -is represented by an OCaml equivalent via the single type -definition:  

+

First, let’s look at how to define basic scalar C types. Every C type is represented by an OCaml equivalent via the single type definition:  

type 'a typ
-

Ctypes.typ is the type of values that represents C -types to OCaml. There are two types associated with each instance of -typ:

+

Ctypes.typ is the type of values that represents C types to OCaml. There are two types associated with each instance of typ:

    -

  • The C type used to store and pass values to the foreign -library.

    • -

  • The corresponding OCaml type. The 'a type parameter -contains the OCaml type such that a value of type t typ is -used to read and write OCaml values of type t.

    • +

  • The C type used to store and pass values to the foreign library.

    • +

  • The corresponding OCaml type. The 'a type parameter contains the OCaml type such that a value of type t typ is used to read and write OCaml values of type t.

-

There are various other uses of typ values within -Ctypes, such as:

+

There are various other uses of typ values within Ctypes, such as:

  • Constructing function types for binding native functions

    • -

  • Constructing pointers for reading and writing locations in -C-managed storage

    • -

  • Describing component fields of structures, unions, and -arrays

    • +

  • Constructing pointers for reading and writing locations in C-managed storage

    • +

  • Describing component fields of structures, unions, and arrays

-

Here are the definitions for most of the standard C99 scalar types, -including some platform-dependent ones:  

+

Here are the definitions for most of the standard C99 scalar types, including some platform-dependent ones:  

val void      : unit typ
 val char      : char typ
@@ -350,66 +213,35 @@ 
Basic Scalar C Types

 val complex32 : Complex.t typ
 val complex64 : Complex.t typ
-

These values are all of type 'a typ, where the value -name (e.g., void) tells you the C type and the -'a component (e.g., unit) is the OCaml -representation of that C type. Most of the mappings are straightforward, -but some of them need a bit more explanation:

+

These values are all of type 'a typ, where the value name (e.g., void) tells you the C type and the 'a component (e.g., unit) is the OCaml representation of that C type. Most of the mappings are straightforward, but some of them need a bit more explanation:

    -

  • Void values appear in OCaml as the unit type. Using -void in an argument or result type specification produces -an OCaml function that accepts or returns unit. -Dereferencing a pointer to void is an error, as in C, and -will raise the IncompleteType exception.

    • -

  • The C size_t type is an alias for one of the -unsigned integer types. The actual size and alignment requirements for -size_t varies between platforms. Ctypes provides an OCaml -size_t type that is aliased to the appropriate integer -type.

    • -

  • OCaml only supports double-precision floating-point numbers, and -so the C float and double types both map onto -the OCaml float type, and the C float complex -and double complex types both map onto the OCaml -double-precision Complex.t type.

    • +

  • Void values appear in OCaml as the unit type. Using void in an argument or result type specification produces an OCaml function that accepts or returns unit. Dereferencing a pointer to void is an error, as in C, and will raise the IncompleteType exception.

    • +

  • The C size_t type is an alias for one of the unsigned integer types. The actual size and alignment requirements for size_t varies between platforms. Ctypes provides an OCaml size_t type that is aliased to the appropriate integer type.

    • +

  • OCaml only supports double-precision floating-point numbers, and so the C float and double types both map onto the OCaml float type, and the C float complex and double complex types both map onto the OCaml double-precision Complex.t type.

Pointers and Arrays

-

Pointers are at the heart of C, so they are necessarily part of -Ctypes, which provides support for pointer arithmetic, pointer -conversions, reading and writing through pointers, and passing and -returning pointers to and from functions.    

-

We’ve already seen a simple use of pointers in the Ncurses example. -Let’s start a new example by binding the following POSIX functions:

+

Pointers are at the heart of C, so they are necessarily part of Ctypes, which provides support for pointer arithmetic, pointer conversions, reading and writing through pointers, and passing and returning pointers to and from functions.    

+

We’ve already seen a simple use of pointers in the Ncurses example. Let’s start a new example by binding the following POSIX functions:

time_t time(time_t *);
 double difftime(time_t, time_t);
 char *ctime(const time_t *timep);
-

The time function returns the current calendar time and -is a simple start. The first step is to open some of the Ctypes -modules:

+

The time function returns the current calendar time and is a simple start. The first step is to open some of the Ctypes modules:

Ctypes
-
-The Ctypes module provides functions for describing C types -in OCaml. +
The Ctypes module provides functions for describing C types in OCaml.
PosixTypes
-
-The PosixTypes module includes some extra POSIX-specific -types (such as time_t). +
The PosixTypes module includes some extra POSIX-specific types (such as time_t).
Foreign
-
-The Foreign module exposes the foreign -function that makes it possible to invoke C functions. +
The Foreign module exposes the foreign function that makes it possible to invoke C functions.
-

With these opens in place, we can now create a binding to -time directly from the toplevel.

+

With these opens in place, we can now create a binding to time directly from the toplevel.

#require "ctypes-foreign";;
 #require "ctypes.top";;
@@ -421,33 +253,20 @@ 
Pointers and Arrays

 >val time : time_t Ctypes_static.ptr -> time_t = <fun>
-

The foreign function is the main link between OCaml and -C. It takes two arguments: the name of the C function to bind, and a -value describing the type of the bound function. In the -time binding, the function type specifies one argument of -type ptr time_t and a return type of -time_t.

-

We can now call time immediately in the same toplevel. -The argument is actually optional, so we’ll just pass a null pointer -that has been coerced into becoming a null pointer to -time_t:

+

The foreign function is the main link between OCaml and C. It takes two arguments: the name of the C function to bind, and a value describing the type of the bound function. In the time binding, the function type specifies one argument of type ptr time_t and a return type of time_t.

+

We can now call time immediately in the same toplevel. The argument is actually optional, so we’ll just pass a null pointer that has been coerced into becoming a null pointer to time_t:

let cur_time = time (from_voidp time_t null);;
 >val cur_time : time_t = <abstr>
-

Since we’re going to call time a few times, let’s create -a wrapper function that passes the null pointer through:

+

Since we’re going to call time a few times, let’s create a wrapper function that passes the null pointer through:

let time' () = time (from_voidp time_t null);;
 >val time' : unit -> time_t = <fun>
-

Since time_t is an abstract type, we can’t actually do -anything useful with it directly. We need to bind a second function to -do anything useful with the return values from time. We’ll -move on to difftime; the second C function in our prototype -list:

+

Since time_t is an abstract type, we can’t actually do anything useful with it directly. We need to bind a second function to do anything useful with the return values from time. We’ll move on to difftime; the second C function in our prototype list:

let difftime = foreign "difftime" (time_t @-> time_t @-> returning double);;
 >val difftime : time_t -> time_t -> float = <fun>
@@ -468,22 +287,16 @@ 
Pointers and Arrays

 >val delta : float = 2.
-

The binding to difftime above is sufficient to compare -two time_t values.

+

The binding to difftime above is sufficient to compare two time_t values.

Allocating Typed Memory for Pointers

-

Let’s look at a slightly less trivial example where we pass a nonnull -pointer to a function. Continuing with the theme from earlier, we’ll -bind to the ctime function, which converts a -time_t value to a human-readable string:  

+

Let’s look at a slightly less trivial example where we pass a nonnull pointer to a function. Continuing with the theme from earlier, we’ll bind to the ctime function, which converts a time_t value to a human-readable string:  

let ctime = foreign "ctime" (ptr time_t @-> returning string);;
 >val ctime : time_t Ctypes_static.ptr -> string = <fun>
-

The binding is continued in the toplevel to add to our growing -collection. However, we can’t just pass the result of time -to ctime:

+

The binding is continued in the toplevel to add to our growing collection. However, we can’t just pass the result of time to ctime:

ctime (time' ());;
 >Line 1, characters 7-17:
@@ -491,19 +304,13 @@ 
Allocating Typed Memory for Pointers

 >         time_t Ctypes_static.ptr = (time_t, [ `C ]) pointer
-

This is because ctime needs a pointer to the -time_t rather than passing it by value. We thus need to -allocate some memory for the time_t and obtain its memory -address:

+

This is because ctime needs a pointer to the time_t rather than passing it by value. We thus need to allocate some memory for the time_t and obtain its memory address:

let t_ptr = allocate time_t (time' ());;
 ...
-

The allocate function takes the type of the memory to be -allocated and the initial value and it returns a suitably typed pointer. -We can now call ctime passing the pointer as an -argument:

+

The allocate function takes the type of the memory to be allocated and the initial value and it returns a suitably typed pointer. We can now call ctime passing the pointer as an argument:

ctime t_ptr;;
 ...
@@ -512,77 +319,45 @@ 
Allocating Typed Memory for Pointers

Using Views to Map Complex Values

-

While scalar types typically have a 1:1 representation, other C types -require extra work to convert them into OCaml. Views create new C type -descriptions that have special behavior when used to read or write C -values.   

-

We’ve already used one view in the definition of ctime -earlier. The string view wraps the C type -char * (written in OCaml as ptr char) and -converts between the C and OCaml string representations each time the -value is written or read.

-

Here is the type signature of the Ctypes.view -function:

+

While scalar types typically have a 1:1 representation, other C types require extra work to convert them into OCaml. Views create new C type descriptions that have special behavior when used to read or write C values.   

+

We’ve already used one view in the definition of ctime earlier. The string view wraps the C type char * (written in OCaml as ptr char) and converts between the C and OCaml string representations each time the value is written or read.

+

Here is the type signature of the Ctypes.view function:

val view :
   read:('a -> 'b) ->
   write:('b -> 'a) ->
   'a typ -> 'b typ
-

Ctypes has some internal low-level conversion functions that map -between an OCaml string and a C character buffer by copying -the contents into the respective data structure. They have the following -type signature:

+

Ctypes has some internal low-level conversion functions that map between an OCaml string and a C character buffer by copying the contents into the respective data structure. They have the following type signature:

val string_of_char_ptr : char ptr -> string
 val char_ptr_of_string : string -> char ptr
-

Given these functions, the definition of the -Ctypes.string value that uses views is quite simple:

+

Given these functions, the definition of the Ctypes.string value that uses views is quite simple:

let string =
   view (char ptr)
     ~read:string_of_char_ptr
     ~write:char_ptr_of_string
-

The type of this string function is a normal -typ with no external sign of the use of the view -function:

+

The type of this string function is a normal typ with no external sign of the use of the view function:

val string    : string typ
-
-

OCaml Strings Versus C Character Buffers

-

Although OCaml strings may look like C character buffers from an -interface perspective, they’re very different in terms of their memory -representations.

-

OCaml strings are stored in the OCaml heap with a header that -explicitly defines their length. C buffers are also fixed-length, but by -convention, a C string is terminated by a null (a \0 byte) -character. The C string functions calculate their length by scanning the -buffer until the first null character is encountered.

-

This means that you need to be careful that OCaml strings that you -pass to C functions don’t contain any null values, since the first -occurrence of a null character will be treated as the end of the C -string. Ctypes also defaults to a copying interface for -strings, which means that you shouldn’t use them when you want the -library to mutate the buffer in-place. In that situation, use the Ctypes -Bigarray support to pass memory by reference instead.

-
+
+

OCaml Strings Versus C Character Buffers

+

Although OCaml strings may look like C character buffers from an interface perspective, they’re very different in terms of their memory representations.

+

OCaml strings are stored in the OCaml heap with a header that explicitly defines their length. C buffers are also fixed-length, but by convention, a C string is terminated by a null (a \0 byte) character. The C string functions calculate their length by scanning the buffer until the first null character is encountered.

+

This means that you need to be careful that OCaml strings that you pass to C functions don’t contain any null values, since the first occurrence of a null character will be treated as the end of the C string. Ctypes also defaults to a copying interface for strings, which means that you shouldn’t use them when you want the library to mutate the buffer in-place. In that situation, use the Ctypes Bigarray support to pass memory by reference instead.

+

Structs and Unions

-

The C constructs struct and union make it -possible to build new types from existing types. Ctypes contains -counterparts that work similarly.   

+

The C constructs struct and union make it possible to build new types from existing types. Ctypes contains counterparts that work similarly.   

Defining a Structure

-

Let’s improve the timer function that we wrote earlier. The POSIX -function gettimeofday retrieves the time with microsecond -resolution. The signature of gettimeofday is as follows, -including the structure definitions:

+

Let’s improve the timer function that we wrote earlier. The POSIX function gettimeofday retrieves the time with microsecond resolution. The signature of gettimeofday is as follows, including the structure definitions:

struct timeval {
   long tv_sec;
@@ -591,8 +366,7 @@ 
Defining a Structure

 
 int gettimeofday(struct timeval *, struct timezone *tv);
-

Using Ctypes, we can describe this type as follows in our toplevel, -continuing on from the previous definitions:

+

Using Ctypes, we can describe this type as follows in our toplevel, continuing on from the previous definitions:

type timeval;;
 >type timeval
@@ -603,23 +377,12 @@ 
Defining a Structure

 >    spec = Ctypes_static.Incomplete {Ctypes_static.isize = 0}; fields = []}
-

The first command defines a new OCaml type timeval that -we’ll use to instantiate the OCaml version of the struct. This is a -phantom type that exists only to distinguish the underlying C -type from other pointer types. The particular timeval -structure now has a distinct type from other structures we define -elsewhere, which helps to avoid getting them mixed up.  

-

The second command calls structure to create a fresh -structure type. At this point, the structure type is incomplete: we can -add fields but cannot yet use it in foreign calls or use it -to create values.

+

The first command defines a new OCaml type timeval that we’ll use to instantiate the OCaml version of the struct. This is a phantom type that exists only to distinguish the underlying C type from other pointer types. The particular timeval structure now has a distinct type from other structures we define elsewhere, which helps to avoid getting them mixed up.  

+

The second command calls structure to create a fresh structure type. At this point, the structure type is incomplete: we can add fields but cannot yet use it in foreign calls or use it to create values.

Adding Fields to Structures

-

The timeval structure definition still doesn’t have any -fields, so we need to add those next:   

+

The timeval structure definition still doesn’t have any fields, so we need to add those next:   

let tv_sec  = field timeval "tv_sec" long;;
 >val tv_sec : (Signed.long, timeval structure) field =
@@ -633,22 +396,12 @@ 
Adding Fields to Structures

 >- : unit = ()
-

The field function appends a field to the structure, as -shown with tv_sec and tv_usec. Structure -fields are typed accessors that are associated with a particular -structure, and they correspond to the labels in C.

-

Every field addition mutates the structure variable and records a new -size (the exact value of which depends on the type of the field that was -just added). Once we seal the structure, we will be able to -create values using it, but adding fields to a sealed structure is an -error.

+

The field function appends a field to the structure, as shown with tv_sec and tv_usec. Structure fields are typed accessors that are associated with a particular structure, and they correspond to the labels in C.

+

Every field addition mutates the structure variable and records a new size (the exact value of which depends on the type of the field that was just added). Once we seal the structure, we will be able to create values using it, but adding fields to a sealed structure is an error.

Incomplete Structure Definitions

-

Since gettimeofday needs a struct timezone -pointer for its second argument, we also need to define a second -structure type:  

+

Since gettimeofday needs a struct timezone pointer for its second argument, we also need to define a second structure type:  

type timezone;;
 >type timezone
@@ -659,11 +412,7 @@ 
Incomplete Structure Definitions

 >    spec = Ctypes_static.Incomplete {Ctypes_static.isize = 0}; fields = []}
-

We don’t ever need to create struct timezone values, so -we can leave this struct as incomplete without adding any fields or -sealing it. If you ever try to use it in a situation where its concrete -size needs to be known, the library will raise an -IncompleteType exception.

+

We don’t ever need to create struct timezone values, so we can leave this struct as incomplete without adding any fields or sealing it. If you ever try to use it in a situation where its concrete size needs to be known, the library will raise an IncompleteType exception.

We’re finally ready to bind to gettimeofday now:

let gettimeofday = foreign "gettimeofday" ~check_errno:true
@@ -673,16 +422,8 @@ 
Incomplete Structure Definitions

 >  timezone structure Ctypes_static.ptr -> int = <fun>
-

There’s one other new feature here: the -returning_checking_errno function behaves like -returning, except that it checks whether the bound C -function modifies the C error flag. Changes to errno are -mapped into OCaml exceptions and raise a Unix.Unix_error -exception just as the standard library functions do.

-

As before, we can create a wrapper to make gettimeofday -easier to use. The functions make, addr, and -getf create a structure value, retrieve the address of a -structure value, and retrieve the value of a field from a structure.

+

There’s one other new feature here: the returning_checking_errno function behaves like returning, except that it checks whether the bound C function modifies the C error flag. Changes to errno are mapped into OCaml exceptions and raise a Unix.Unix_error exception just as the standard library functions do.

+

As before, we can create a wrapper to make gettimeofday easier to use. The functions make, addr, and getf create a structure value, retrieve the address of a structure value, and retrieve the value of a field from a structure.

let gettimeofday' () =
   let tv = make timeval in
@@ -701,10 +442,7 @@ 
Incomplete Structure Definitions

Recap: a Time-Printing Command

-

We built up a lot of bindings in the previous section, so let’s recap -them with a complete example that ties it together with a command-line -frontend:  

+

We built up a lot of bindings in the previous section, so let’s recap them with a complete example that ties it together with a command-line frontend:  

open Core
 open Ctypes
@@ -774,30 +512,22 @@ 
Recap: a Time-Printing Command

 >Mon Oct 11 15:57:38 2021
-
-
-

Why Do We Need to Use returning?

-

The alert reader may be curious about why all these function -definitions have to be terminated by returning:

+
+

Why Do We Need to Use returning?

+

The alert reader may be curious about why all these function definitions have to be terminated by returning:

(* correct types *)
 val time: ptr time_t @-> returning time_t
 val difftime: time_t @-> time_t @-> returning double
-

The returning function may appear superfluous here. Why -couldn’t we simply give the types as follows?

+

The returning function may appear superfluous here. Why couldn’t we simply give the types as follows?

(* incorrect types *)
 val time: ptr time_t @-> time_t
 val difftime: time_t @-> time_t @-> double
-

The reason involves higher types and two differences between the way -that functions are treated in OCaml and C. Functions are first-class -values in OCaml, but not in C. For example, in C it is possible to -return a function pointer from a function, but not to return an actual -function.

-

Secondly, OCaml functions are typically defined in a curried style. -The signature of a two-argument function is written as follows:

+

The reason involves higher types and two differences between the way that functions are treated in OCaml and C. Functions are first-class values in OCaml, but not in C. For example, in C it is possible to return a function pointer from a function, but not to return an actual function.

+

Secondly, OCaml functions are typically defined in a curried style. The signature of a two-argument function is written as follows:

val curried : int -> int -> int
@@ -805,9 +535,7 @@

Why Do We Need to Use returning?

val curried : int -> (int -> int)
-

and the arguments can be supplied one at a time to create a closure. -In contrast, C functions receive their arguments all at once. The -equivalent C function type is the following:

+

and the arguments can be supplied one at a time to create a closure. In contrast, C functions receive their arguments all at once. The equivalent C function type is the following:

int uncurried_C(int, int);
@@ -815,8 +543,7 @@

Why Do We Need to Use returning?

uncurried_C(3, 4);
-

A C function that’s written in curried style looks very -different:

+

A C function that’s written in curried style looks very different:

/* A function that accepts an int, and returns a function
    pointer that accepts a second int and returns an int. */
@@ -829,23 +556,14 @@ 
Why Do We Need to Use returning?

 /* supply one argument at a time */
 function_t *f = curried_C(3); f(4);
-

The OCaml type of uncurried_C when bound by Ctypes is -int -> int -> int: a two-argument function. The OCaml -type of curried_C when bound by ctypes is -int -> (int -> int): a one-argument function that -returns a one-argument function.

-

In OCaml, of course, these types are absolutely equivalent. Since the -OCaml types are the same but the C semantics are quite different, we -need some kind of marker to distinguish the cases. This is the purpose -of returning in function definitions.

+

The OCaml type of uncurried_C when bound by Ctypes is int -> int -> int: a two-argument function. The OCaml type of curried_C when bound by ctypes is int -> (int -> int): a one-argument function that returns a one-argument function.

+

In OCaml, of course, these types are absolutely equivalent. Since the OCaml types are the same but the C semantics are quite different, we need some kind of marker to distinguish the cases. This is the purpose of returning in function definitions.

+

Defining Arrays

-

Arrays in C are contiguous blocks of the same type of value. Any of -the basic types defined previously can be allocated as blocks via the -Array module:  

+

Arrays in C are contiguous blocks of the same type of value. Any of the basic types defined previously can be allocated as blocks via the Array module:  

module Array : sig
   type 'a t = 'a array
@@ -860,64 +578,36 @@ 
Defining Arrays

   val make : 'a typ -> ?initial:'a -> int -> 'a t
 end
-

The array functions are similar to those in the standard library -Array module except that they operate on arrays stored -using the flat C representation rather than the OCaml representation -described in Chapter 23, Memory Representation Of Values.

-

As with standard OCaml arrays, the conversion between arrays and -lists requires copying the values, which can be expensive for large data -structures. Notice that you can also convert an array into a -ptr pointer to the head of the underlying buffer, which can -be useful if you need to pass the pointer and size arguments separately -to a C function.

-

Unions in C are named structures that can be mapped onto the same -underlying memory. They are also fully supported in Ctypes, but we won’t -go into more detail here.   

+

The array functions are similar to those in the standard library Array module except that they operate on arrays stored using the flat C representation rather than the OCaml representation described in Chapter 23, Memory Representation Of Values.

+

As with standard OCaml arrays, the conversion between arrays and lists requires copying the values, which can be expensive for large data structures. Notice that you can also convert an array into a ptr pointer to the head of the underlying buffer, which can be useful if you need to pass the pointer and size arguments separately to a C function.

+

Unions in C are named structures that can be mapped onto the same underlying memory. They are also fully supported in Ctypes, but we won’t go into more detail here.   

Pointer Operators for Dereferencing and Arithmetic
-

Ctypes defines a number of operators that let you manipulate pointers -and arrays just as you would in C. The Ctypes equivalents do have the -benefit of being more strongly typed, of course.

+

Ctypes defines a number of operators that let you manipulate pointers and arrays just as you would in C. The Ctypes equivalents do have the benefit of being more strongly typed, of course.

  • !@ p will dereference the pointer p.
    • -
  • p <-@ v will write the value v to the -address p.
    • -
  • p +@ n computes the address of the nth -next element, if p points to an array element.
    • -
  • p -@ n computes the address of the nth -previous element, if p points to an array element.
    • +
  • p <-@ v will write the value v to the address p.
    • +
  • p +@ n computes the address of the nth next element, if p points to an array element.
    • +
  • p -@ n computes the address of the nth previous element, if p points to an array element.
-

There are also other useful non-operator functions available (see the -Ctypes documentation), such as pointer differencing and comparison.

+

There are also other useful non-operator functions available (see the Ctypes documentation), such as pointer differencing and comparison.

Passing Functions to C

-

It’s also straightforward to pass OCaml function values to C. The C -standard library function qsort sorts arrays of elements -using a comparison function passed in as a function pointer. The -signature for qsort is:  

+

It’s also straightforward to pass OCaml function values to C. The C standard library function qsort sorts arrays of elements using a comparison function passed in as a function pointer. The signature for qsort is:  

void qsort(void *base, size_t nmemb, size_t size,
            int(*compar)(const void *, const void *));
-

C programmers often use typedef to make type definitions -involving function pointers easier to read. Using a typedef, the type of -qsort looks a little more palatable:

+

C programmers often use typedef to make type definitions involving function pointers easier to read. Using a typedef, the type of qsort looks a little more palatable:

typedef int(compare_t)(const void *, const void *);
 
 void qsort(void *base, size_t nmemb, size_t size, compare_t *);
-

This also happens to be a close mapping to the corresponding Ctypes -definition. Since type descriptions are regular values, we can just use -let in place of typedef and end up with -working OCaml bindings to qsort:

+

This also happens to be a close mapping to the corresponding Ctypes definition. Since type descriptions are regular values, we can just use let in place of typedef and end up with working OCaml bindings to qsort:

open Core open Ctypes open PosixTypes open Foreign open Ctypes_static;;
 let compare_t = ptr void @-> ptr void @-> returning int;;
@@ -935,20 +625,11 @@ 
Passing Functions to C

 >  <fun>
-

We only use compare_t once (in the qsort -definition), so you can choose to inline it in the OCaml code if you -prefer. As the type shows, the resulting qsort value is a -higher-order function, since the fourth argument is itself a -function.

-

Arrays created using Ctypes have a richer runtime structure than C -arrays, so we don’t need to pass size information around. Furthermore, -we can use OCaml polymorphism in place of the unsafe -void ptr type.

+

We only use compare_t once (in the qsort definition), so you can choose to inline it in the OCaml code if you prefer. As the type shows, the resulting qsort value is a higher-order function, since the fourth argument is itself a function.

+

Arrays created using Ctypes have a richer runtime structure than C arrays, so we don’t need to pass size information around. Furthermore, we can use OCaml polymorphism in place of the unsafe void ptr type.

Example: A Command-Line Quicksort

-

The following is a command-line tool that uses the qsort -binding to sort all of the integers supplied on the standard input: - 

+

The following is a command-line tool that uses the qsort binding to sort all of the integers supplied on the standard input:  

open Core
 open Ctypes
@@ -995,9 +676,7 @@ 
Example: A Command-Line Quicksort

     sort_stdin
   |> Command_unix.run
-

Compile it in the usual way with dune and test it against -some input data, and also build the inferred interface so we can examine -it more closely:

+

Compile it in the usual way with dune and test it against some input data, and also build the inferred interface so we can examine it more closely:

(executable
   (name      qsort)
@@ -1008,8 +687,7 @@ 
Example: A Command-Line Quicksort

 >1 2 3 4
-

The inferred mli shows us the types of the raw qsort -binding and also the qsort' wrapper function.

+

The inferred mli shows us the types of the raw qsort binding and also the qsort' wrapper function.

ocaml-print-intf qsort.ml
 >val compare_t :
@@ -1023,124 +701,40 @@ 
Example: A Command-Line Quicksort

 >val sort_stdin : unit -> unit
-

The qsort' wrapper function has a much more canonical -OCaml interface than the raw binding. It accepts a comparator function -and a Ctypes array, and returns unit.

-

Using qsort' to sort arrays is straightforward. Our -example code reads the standard input as a list, converts it to a C -array, passes it through qsort, and outputs the result to the standard -output. Again, remember to not confuse the Ctypes.Array -module with the Core.Array module: the former is in scope -since we opened Ctypes at the start of the file.    

-
-

Lifetime of Allocated Ctypes

-

Values allocated via Ctypes (i.e., using allocate, -Array.make, and so on) will not be garbage-collected as -long as they are reachable from OCaml values. The system memory they -occupy is freed when they do become unreachable, via a finalizer -function registered with the garbage collector (GC).

-

The definition of reachability for Ctypes values is a little -different from conventional OCaml values, though. The allocation -functions return an OCaml-managed pointer to the value, and as long as -some derivative pointer is still reachable by the GC, the value won’t be -collected.

-

“Derivative” means a pointer that’s computed from the original -pointer via arithmetic, so a reachable reference to an array element or -a structure field protects the whole object from collection.

-

A corollary of the preceding rule is that pointers written into the C -heap don’t have any effect on reachability. For example, if you have a -C-managed array of pointers to structs, then you’ll need some additional -way of keeping the structs themselves around to protect them from -collection. You could achieve this via a global array of values on the -OCaml side that would keep them live until they’re no longer needed.

-

Functions passed to C have similar considerations regarding lifetime. -On the OCaml side, functions created at runtime may be collected when -they become unreachable. As we’ve seen, OCaml functions passed to C are -converted to function pointers, and function pointers written into the C -heap have no effect on the reachability of the OCaml functions they -reference. With qsort things are straightforward, since the -comparison function is only used during the call to qsort -itself. However, other C libraries may store function pointers in global -variables or elsewhere, in which case you’ll need to take care that the -OCaml functions you pass to them aren’t prematurely -garbage-collected.

-
+

The qsort' wrapper function has a much more canonical OCaml interface than the raw binding. It accepts a comparator function and a Ctypes array, and returns unit.

+

Using qsort' to sort arrays is straightforward. Our example code reads the standard input as a list, converts it to a C array, passes it through qsort, and outputs the result to the standard output. Again, remember to not confuse the Ctypes.Array module with the Core.Array module: the former is in scope since we opened Ctypes at the start of the file.    

+
+

Lifetime of Allocated Ctypes

+

Values allocated via Ctypes (i.e., using allocate, Array.make, and so on) will not be garbage-collected as long as they are reachable from OCaml values. The system memory they occupy is freed when they do become unreachable, via a finalizer function registered with the garbage collector (GC).

+

The definition of reachability for Ctypes values is a little different from conventional OCaml values, though. The allocation functions return an OCaml-managed pointer to the value, and as long as some derivative pointer is still reachable by the GC, the value won’t be collected.

+

“Derivative” means a pointer that’s computed from the original pointer via arithmetic, so a reachable reference to an array element or a structure field protects the whole object from collection.

+

A corollary of the preceding rule is that pointers written into the C heap don’t have any effect on reachability. For example, if you have a C-managed array of pointers to structs, then you’ll need some additional way of keeping the structs themselves around to protect them from collection. You could achieve this via a global array of values on the OCaml side that would keep them live until they’re no longer needed.

+

Functions passed to C have similar considerations regarding lifetime. On the OCaml side, functions created at runtime may be collected when they become unreachable. As we’ve seen, OCaml functions passed to C are converted to function pointers, and function pointers written into the C heap have no effect on the reachability of the OCaml functions they reference. With qsort things are straightforward, since the comparison function is only used during the call to qsort itself. However, other C libraries may store function pointers in global variables or elsewhere, in which case you’ll need to take care that the OCaml functions you pass to them aren’t prematurely garbage-collected.

+

Learning More About C Bindings

-

The Ctypes distribution -contains a number of larger-scale examples, including:  

+

The Ctypes distribution contains a number of larger-scale examples, including:  

    -

  • Bindings to the POSIX fts API, which demonstrates C -callbacks more comprehensively

    • -

  • A more complete Ncurses binding than the example we opened the -chapter with

    • -

  • A comprehensive test suite that covers the complete library, and -can provide useful snippets for your own bindings

    • +

  • Bindings to the POSIX fts API, which demonstrates C callbacks more comprehensively

    • +

  • A more complete Ncurses binding than the example we opened the chapter with

    • +

  • A comprehensive test suite that covers the complete library, and can provide useful snippets for your own bindings

-

This chapter hasn’t really needed you to understand the innards of -OCaml at all. Ctypes does its best to make function bindings easy, but -the rest of this part will also fill you in about interactions with -OCaml memory layout in Chapter 23, Memory Representation Of Values and automatic -memory management in Chapter 24, Understanding The Garbage Collector.

-

Ctypes gives OCaml programs access to the C representation of values, -shielding you from the details of the OCaml value representation, and -introduces an abstraction layer that hides the details of foreign calls. -While this covers a wide variety of situations, it’s sometimes necessary -to look behind the abstraction to obtain finer control over the details -of the interaction between the two languages.

-

You can find more information about the C interface in several -places:

+

This chapter hasn’t really needed you to understand the innards of OCaml at all. Ctypes does its best to make function bindings easy, but the rest of this part will also fill you in about interactions with OCaml memory layout in Chapter 23, Memory Representation Of Values and automatic memory management in Chapter 24, Understanding The Garbage Collector.

+

Ctypes gives OCaml programs access to the C representation of values, shielding you from the details of the OCaml value representation, and introduces an abstraction layer that hides the details of foreign calls. While this covers a wide variety of situations, it’s sometimes necessary to look behind the abstraction to obtain finer control over the details of the interaction between the two languages.

+

You can find more information about the C interface in several places:

    -

  • The standard OCaml foreign function interface allows you to glue -OCaml and C together from the other side of the boundary, by writing C -functions that operate on the OCaml representation of values. You can -find details of the standard interface in the OCaml manual and in the -book Developing -Applications with Objective Caml.

    • -

  • Florent Monnier maintains an excellent online -tutorial that provides examples of how to call OCaml functions from -C. This covers a wide variety of OCaml data types and also more complex -callbacks between C and OCaml.

    • +

  • The standard OCaml foreign function interface allows you to glue OCaml and C together from the other side of the boundary, by writing C functions that operate on the OCaml representation of values. You can find details of the standard interface in the OCaml manual and in the book Developing Applications with Objective Caml.

    • +

  • Florent Monnier maintains an excellent online tutorial that provides examples of how to call OCaml functions from C. This covers a wide variety of OCaml data types and also more complex callbacks between C and OCaml.

Struct Memory Layout

-

The C language gives implementations a certain amount of freedom in -choosing how to lay out structs in memory. There may be padding between -members and at the end of the struct, in order to satisfy the memory -alignment requirements of the host platform. Ctypes uses -platform-appropriate size and alignment information to replicate the -struct layout process. OCaml and C will have consistent views about the -layout of the struct as long as you declare the fields of a struct in -the same order and with the same types as the C library you’re binding -to.   

-

However, this approach can lead to difficulties when the fields of a -struct aren’t fully specified in the interface of a library. The -interface may list the fields of a structure without specifying their -order, or make certain fields available only on certain platforms, or -insert undocumented fields into struct definitions for performance -reasons. For example, the struct timeval definition used in -this chapter accurately describes the layout of the struct on common -platforms, but implementations on some more unusual architectures -include additional padding members that will lead to strange behavior in -the examples.

-

The Cstubs subpackage of Ctypes addresses this issue. Rather than -simply assuming that struct definitions given by the user accurately -reflect the actual definitions of structs used in C libraries, Cstubs -generates code that uses the C library headers to discover the layout of -the struct. The good news is that the code that you write doesn’t need -to change much. Cstubs provides alternative implementations of the -field and seal functions that you’ve already -used to describe struct timeval; instead of computing -member offsets and sizes appropriate for the platform, these -implementations obtain them directly from C.

-

The details of using Cstubs are available in the online documentation, along -with instructions on integration with autoconf platform -portability instructions.

+

The C language gives implementations a certain amount of freedom in choosing how to lay out structs in memory. There may be padding between members and at the end of the struct, in order to satisfy the memory alignment requirements of the host platform. Ctypes uses platform-appropriate size and alignment information to replicate the struct layout process. OCaml and C will have consistent views about the layout of the struct as long as you declare the fields of a struct in the same order and with the same types as the C library you’re binding to.   

+

However, this approach can lead to difficulties when the fields of a struct aren’t fully specified in the interface of a library. The interface may list the fields of a structure without specifying their order, or make certain fields available only on certain platforms, or insert undocumented fields into struct definitions for performance reasons. For example, the struct timeval definition used in this chapter accurately describes the layout of the struct on common platforms, but implementations on some more unusual architectures include additional padding members that will lead to strange behavior in the examples.

+

The Cstubs subpackage of Ctypes addresses this issue. Rather than simply assuming that struct definitions given by the user accurately reflect the actual definitions of structs used in C libraries, Cstubs generates code that uses the C library headers to discover the layout of the struct. The good news is that the code that you write doesn’t need to change much. Cstubs provides alternative implementations of the field and seal functions that you’ve already used to describe struct timeval; instead of computing member offsets and sizes appropriate for the platform, these implementations obtain them directly from C.

+

The details of using Cstubs are available in the online documentation, along with instructions on integration with autoconf platform portability instructions.

-

Next: Chapter 23Memory Representation of Values

 \ No newline at end of file +

Next: Chapter 23Memory Representation of Values

 \ No newline at end of file diff --git a/functors.html b/functors.html index 6e598503c..6c7fcc762 100644 --- a/functors.html +++ b/functors.html @@ -1,62 +1,30 @@ -Functors - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Functors - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Functors

-

Up until now, we’ve seen OCaml’s modules play an important but -limited role. In particular, we’ve used modules to organize code into -units with specified interfaces. But OCaml’s module system can do much -more than that, serving as a powerful tool for building generic code and -structuring large-scale systems. Much of that power comes from functors. - 

-

Functors are, roughly speaking, functions from modules to modules, -and they can be used to solve a variety of code-structuring problems, -including:

+

Up until now, we’ve seen OCaml’s modules play an important but limited role. In particular, we’ve used modules to organize code into units with specified interfaces. But OCaml’s module system can do much more than that, serving as a powerful tool for building generic code and structuring large-scale systems. Much of that power comes from functors.  

+

Functors are, roughly speaking, functions from modules to modules, and they can be used to solve a variety of code-structuring problems, including:

Dependency injection
-
-Makes the implementations of some components of a system swappable. This -is particularly useful when you want to mock up parts of your system for -testing and simulation purposes. +
Makes the implementations of some components of a system swappable. This is particularly useful when you want to mock up parts of your system for testing and simulation purposes.
Autoextension of modules
-
-Functors give you a way of extending existing modules with new -functionality in a standardized way. For example, you might want to add -a slew of comparison operators derived from a base comparison function. -To do this by hand would require a lot of repetitive code for each type, -but functors let you write this logic just once and apply it to many -different types. +
Functors give you a way of extending existing modules with new functionality in a standardized way. For example, you might want to add a slew of comparison operators derived from a base comparison function. To do this by hand would require a lot of repetitive code for each type, but functors let you write this logic just once and apply it to many different types.
Instantiating modules with state
-
-Modules can contain mutable states, and that means that you’ll -occasionally want to have multiple instantiations of a particular -module, each with its own separate and independent mutable state. -Functors let you automate the construction of such modules. +
Modules can contain mutable states, and that means that you’ll occasionally want to have multiple instantiations of a particular module, each with its own separate and independent mutable state. Functors let you automate the construction of such modules.
-

These are really just some of the uses that you can put functors to. -We’ll make no attempt to provide examples of all of the uses of functors -here. Instead, this chapter will try to provide examples that illuminate -the language features and design patterns that you need to master in -order to use functors effectively.

+

These are really just some of the uses that you can put functors to. We’ll make no attempt to provide examples of all of the uses of functors here. Instead, this chapter will try to provide examples that illuminate the language features and design patterns that you need to master in order to use functors effectively.

A Trivial Example

-

Let’s create a functor that takes a module containing a single -integer variable x and returns a new module with -x incremented by one. This is intended to serve as a way to -walk through the basic mechanics of functors, even though it’s not -something you’d want to do in practice.  

-

First, let’s define a signature for a module that contains a single -value of type int:

+

Let’s create a functor that takes a module containing a single integer variable x and returns a new module with x incremented by one. This is intended to serve as a way to walk through the basic mechanics of functors, even though it’s not something you’d want to do in practice.  

+

First, let’s define a signature for a module that contains a single value of type int:

open Base;;
 module type X_int = sig val x : int end;;
 >module type X_int = sig val x : int end
-

Now we can define our functor. We’ll use X_int both to -constrain the argument to the functor and to constrain the module -returned by the functor:

+

Now we can define our functor. We’ll use X_int both to constrain the argument to the functor and to constrain the module returned by the functor:

module Increment (M : X_int) : X_int = struct
   let x = M.x + 1
@@ -64,15 +32,8 @@ 
A Trivial Example

 >module Increment : functor (M : X_int) -> X_int
-

One thing that immediately jumps out is that functors are more -syntactically heavyweight than ordinary functions. For one thing, -functors require explicit (module) type annotations, which ordinary -functions do not. Technically, only the type on the input is mandatory, -although in practice, you should usually constrain the module returned -by the functor, just as you should use an mli, even though -it’s not mandatory.

-

The following shows what happens when we omit the module type for the -output of the functor:

+

One thing that immediately jumps out is that functors are more syntactically heavyweight than ordinary functions. For one thing, functors require explicit (module) type annotations, which ordinary functions do not. Technically, only the type on the input is mandatory, although in practice, you should usually constrain the module returned by the functor, just as you should use an mli, even though it’s not mandatory.

+

The following shows what happens when we omit the module type for the output of the functor:

module Increment (M : X_int) = struct
   let x = M.x + 1
@@ -80,9 +41,7 @@ 
A Trivial Example

 >module Increment : functor (M : X_int) -> sig val x : int end
-

We can see that the inferred module type of the output is now written -out explicitly, rather than being a reference to the named signature -X_int.

+

We can see that the inferred module type of the output is now written out explicitly, rather than being a reference to the named signature X_int.

We can use Increment to define new modules:

module Three = struct let x = 3 end;;
@@ -93,14 +52,7 @@ 
A Trivial Example

 >- : int = 1
-

In this case, we applied Increment to a module whose -signature is exactly equal to X_int. But we can apply -Increment to any module that satisfies the -interface X_int, in the same way that the contents of an -ml file must satisfy the mli. That means that -the module type can omit some information available in the module, -either by dropping fields or by leaving some fields abstract. Here’s an -example:

+

In this case, we applied Increment to a module whose signature is exactly equal to X_int. But we can apply Increment to any module that satisfies the interface X_int, in the same way that the contents of an ml file must satisfy the mli. That means that the module type can omit some information available in the module, either by dropping fields or by leaving some fields abstract. Here’s an example:

module Three_and_more = struct
   let x = 3
@@ -111,29 +63,13 @@ 
A Trivial Example

 >module Four : sig val x : int end
-

The rules for determining whether a module matches a given signature -are similar in spirit to the rules in an object-oriented language that -determine whether an object satisfies a given interface. As in an -object-oriented context, the extra information that doesn’t match the -signature you’re looking for (in this case, the variable y) -is simply ignored.

+

The rules for determining whether a module matches a given signature are similar in spirit to the rules in an object-oriented language that determine whether an object satisfies a given interface. As in an object-oriented context, the extra information that doesn’t match the signature you’re looking for (in this case, the variable y) is simply ignored.

A Bigger Example: Computing with Intervals

-

Let’s consider a more realistic example of how to use functors: a -library for computing with intervals. Intervals are a common -computational object, and they come up in different contexts and for -different types. You might need to work with intervals of floating-point -values or strings or times, and in each of these cases, you want similar -operations: testing for emptiness, checking for containment, -intersecting intervals, and so on.

-

We can use functors to build a generic interval library that can be -used with any type that supports a total ordering on the underlying set. -  

-

First we’ll define a module type that captures the information we’ll -need about the endpoints of the intervals. This interface, which we’ll -call Comparable, contains just two things: a comparison -function and the type of the values to be compared:

+

Let’s consider a more realistic example of how to use functors: a library for computing with intervals. Intervals are a common computational object, and they come up in different contexts and for different types. You might need to work with intervals of floating-point values or strings or times, and in each of these cases, you want similar operations: testing for emptiness, checking for containment, intersecting intervals, and so on.

+

We can use functors to build a generic interval library that can be used with any type that supports a total ordering on the underlying set.   

+

First we’ll define a module type that captures the information we’ll need about the endpoints of the intervals. This interface, which we’ll call Comparable, contains just two things: a comparison function and the type of the values to be compared:

module type Comparable = sig
   type t
@@ -142,27 +78,14 @@ 
A Bigger Example: Computing with Intervals

 >module type Comparable = sig type t val compare : t -> t -> int end
-

The comparison function follows the standard OCaml idiom for such -functions, returning 0 if the two elements are equal, a -positive number if the first element is larger than the second, and a -negative number if the first element is smaller than the second. Thus, -we could rewrite the standard comparison functions on top of -compare.

+

The comparison function follows the standard OCaml idiom for such functions, returning 0 if the two elements are equal, a positive number if the first element is larger than the second, and a negative number if the first element is smaller than the second. Thus, we could rewrite the standard comparison functions on top of compare.

compare x y < 0     (* x < y *)
 compare x y = 0     (* x = y *)
 compare x y > 0     (* x > y *)
-

(This idiom is a bit of a historical error. It would be better if -compare returned a variant with three cases for less than, -greater than, and equal. But it’s a well-established idiom at this -point, and unlikely to change.)

-

The functor for creating the interval module follows. We represent an -interval with a variant type, which is either Empty or -Interval (x,y), where x and y are -the bounds of the interval. In addition to the type, the body of the -functor contains implementations of a number of useful primitives for -interacting with intervals:

+

(This idiom is a bit of a historical error. It would be better if compare returned a variant with three cases for less than, greater than, and equal. But it’s a well-established idiom at this point, and unlikely to change.)

+

The functor for creating the interval module follows. We represent an interval with a variant type, which is either Empty or Interval (x,y), where x and y are the bounds of the interval. In addition to the type, the body of the functor contains implementations of a number of useful primitives for interacting with intervals:

module Make_interval(Endpoint : Comparable) = struct
 
@@ -210,10 +133,7 @@ 
A Bigger Example: Computing with Intervals

 >    end
-

We can instantiate the functor by applying it to a module with the -right signature. In the following code, rather than name the module -first and then call the functor, we provide the functor input as an -anonymous module:

+

We can instantiate the functor by applying it to a module with the right signature. In the following code, rather than name the module first and then call the functor, we provide the functor input as an anonymous module:

module Int_interval =
   Make_interval(struct
@@ -230,11 +150,7 @@ 
A Bigger Example: Computing with Intervals

 >  end
-

If the input interface for your functor is aligned with the standards -of the libraries you use, then you don’t need to construct a custom -module to feed to the functor. In this case, we can directly use the -Int or String modules provided by -Base:

+

If the input interface for your functor is aligned with the standards of the libraries you use, then you don’t need to construct a custom module to feed to the functor. In this case, we can directly use the Int or String modules provided by Base:

module Int_interval = Make_interval(Int);;
 >module Int_interval :
@@ -259,14 +175,8 @@ 
A Bigger Example: Computing with Intervals

 >  end
-

This works because many modules in Base, including Int -and String, satisfy an extended version of the -Comparable signature described previously. Such -standardized signatures are good practice, both because they make -functors easier to use, and because they encourage standardization that -makes your codebase easier to navigate.

-

We can use the newly defined Int_interval module like -any ordinary module:

+

This works because many modules in Base, including Int and String, satisfy an extended version of the Comparable signature described previously. Such standardized signatures are good practice, both because they make functors easier to use, and because they encourage standardization that makes your codebase easier to navigate.

+

We can use the newly defined Int_interval module like any ordinary module:

let i1 = Int_interval.create 3 8;;
 >val i1 : Int_interval.t = Int_interval.Interval (3, 8)
@@ -276,11 +186,7 @@ 
A Bigger Example: Computing with Intervals

 >- : Int_interval.t = Int_interval.Interval (4, 8)
-

This design gives us the freedom to use any comparison function we -want for comparing the endpoints. We could, for example, create a type -of integer interval with the order of the comparison reversed, as -follows: 

+

This design gives us the freedom to use any comparison function we want for comparing the endpoints. We could, for example, create a type of integer interval with the order of the comparison reversed, as follows: 

module Rev_int_interval =
   Make_interval(struct
@@ -297,8 +203,7 @@ 
A Bigger Example: Computing with Intervals

 >  end
-

The behavior of Rev_int_interval is of course different -from Int_interval:

+

The behavior of Rev_int_interval is of course different from Int_interval:

let interval = Int_interval.create 4 3;;
 >val interval : Int_interval.t = Int_interval.Empty
@@ -306,10 +211,7 @@ 
A Bigger Example: Computing with Intervals

 >val rev_interval : Rev_int_interval.t = Rev_int_interval.Interval (4, 3)
-

Importantly, Rev_int_interval.t is a different type than -Int_interval.t, even though its physical representation is -the same. Indeed, the type system will prevent us from confusing -them.

+

Importantly, Rev_int_interval.t is a different type than Int_interval.t, even though its physical representation is the same. Indeed, the type system will prevent us from confusing them.

Int_interval.contains rev_interval 3;;
 >Line 1, characters 23-35:
@@ -317,18 +219,10 @@ 
A Bigger Example: Computing with Intervals

 >       but an expression was expected of type Int_interval.t
-

This is important, because confusing the two kinds of intervals would -be a semantic error, and it’s an easy one to make. The ability of -functors to mint new types is a useful trick that comes up a lot.

+

This is important, because confusing the two kinds of intervals would be a semantic error, and it’s an easy one to make. The ability of functors to mint new types is a useful trick that comes up a lot.

Making the Functor Abstract

-

There’s a problem with Make_interval. The code we wrote -depends on the invariant that the upper bound of an interval is greater -than its lower bound, but that invariant can be violated. The invariant -is enforced by the create function, but because -Int_interval.t is not abstract, we can bypass the -create function: 

+

There’s a problem with Make_interval. The code we wrote depends on the invariant that the upper bound of an interval is greater than its lower bound, but that invariant can be violated. The invariant is enforced by the create function, but because Int_interval.t is not abstract, we can bypass the create function: 

Int_interval.is_empty (* going through create *)
 (Int_interval.create 4 3);;
@@ -338,9 +232,7 @@ 
Making the Functor Abstract

 >- : bool = false
-

To make Int_interval.t abstract, we need to restrict the -output of Make_interval with an interface. Here’s an -explicit interface that we can use for that purpose:

+

To make Int_interval.t abstract, we need to restrict the output of Make_interval with an interface. Here’s an explicit interface that we can use for that purpose:

module type Interval_intf = sig
   type t
@@ -361,11 +253,7 @@ 
Making the Functor Abstract

 >  end
-

This interface includes the type endpoint to give us a -way of referring to the endpoint type. Given this interface, we can redo -our definition of Make_interval. Notice that we added the -type endpoint to the implementation of the module to match -Interval_intf:

+

This interface includes the type endpoint to give us a way of referring to the endpoint type. Given this interface, we can redo our definition of Make_interval. Notice that we added the type endpoint to the implementation of the module to match Interval_intf:

module Make_interval(Endpoint : Comparable) : Interval_intf = struct
   type endpoint = Endpoint.t
@@ -408,10 +296,7 @@ 
Making the Functor Abstract

Sharing Constraints

-

The resulting module is abstract, but it’s unfortunately too -abstract. In particular, we haven’t exposed the type -endpoint, which means that we can’t even construct an -interval anymore:  

+

The resulting module is abstract, but it’s unfortunately too abstract. In particular, we haven’t exposed the type endpoint, which means that we can’t even construct an interval anymore:  

module Int_interval = Make_interval(Int);;
 >module Int_interval :
@@ -429,26 +314,15 @@ 
Sharing Constraints

 >         Int_interval.endpoint
-

To fix this, we need to expose the fact that endpoint is -equal to Int.t (or more generally, Endpoint.t, -where Endpoint is the argument to the functor). One way of -doing this is through a sharing constraint, which allows you to -tell the compiler to expose the fact that a given type is equal to some -other type. The syntax for a simple sharing constraint is as -follows:

+

To fix this, we need to expose the fact that endpoint is equal to Int.t (or more generally, Endpoint.t, where Endpoint is the argument to the functor). One way of doing this is through a sharing constraint, which allows you to tell the compiler to expose the fact that a given type is equal to some other type. The syntax for a simple sharing constraint is as follows:

<Module_type> with type <type> = <type'>
-

The result of this expression is a new signature that’s been modified -so that it exposes the fact that type defined -inside of the module type is equal to type' whose -definition is outside of it. One can also apply multiple sharing -constraints to the same signature:

+

The result of this expression is a new signature that’s been modified so that it exposes the fact that type defined inside of the module type is equal to type' whose definition is outside of it. One can also apply multiple sharing constraints to the same signature:

<Module_type> with type <type1> = <type1'> and type <type2> = <type2'>
-

We can use a sharing constraint to create a specialized version of -Interval_intf for integer intervals:

+

We can use a sharing constraint to create a specialized version of Interval_intf for integer intervals:

module type Int_interval_intf =
 Interval_intf with type endpoint = int;;
@@ -463,14 +337,8 @@ 
Sharing Constraints

 >  end
-

We can also use sharing constraints in the context of a functor. The -most common use case is where you want to expose that some of the types -of the module being generated by the functor are related to the types in -the module fed to the functor.

-

In this case, we’d like to expose an equality between the type -endpoint in the new module and the type -Endpoint.t, from the module Endpoint that is -the functor argument. We can do this as follows:

+

We can also use sharing constraints in the context of a functor. The most common use case is where you want to expose that some of the types of the module being generated by the functor are related to the types in the module fed to the functor.

+

In this case, we’d like to expose an equality between the type endpoint in the new module and the type Endpoint.t, from the module Endpoint that is the functor argument. We can do this as follows:

module Make_interval(Endpoint : Comparable)
   : (Interval_intf with type endpoint = Endpoint.t)
@@ -522,10 +390,7 @@ 
Sharing Constraints

 >    end
-

Now the interface is as it was, except that endpoint is -known to be equal to Endpoint.t. As a result of that type -equality, we can again do things that require that endpoint -be exposed, like constructing intervals:

+

Now the interface is as it was, except that endpoint is known to be equal to Endpoint.t. As a result of that type equality, we can again do things that require that endpoint be exposed, like constructing intervals:

module Int_interval = Make_interval(Int);;
 >module Int_interval :
@@ -546,21 +411,11 @@ 
Sharing Constraints

Destructive Substitution

-

Sharing constraints basically do the job, but they have some -downsides. In particular, we’ve now been stuck with the useless type -declaration of endpoint that clutters up both the interface -and the implementation. A better solution would be to modify the -Interval_intf signature by replacing endpoint -with Endpoint.t everywhere it shows up, and deleting the -definition of endpoint from the signature. We can do just -this using what’s called destructive substitution. Here’s the -basic syntax: 

+

Sharing constraints basically do the job, but they have some downsides. In particular, we’ve now been stuck with the useless type declaration of endpoint that clutters up both the interface and the implementation. A better solution would be to modify the Interval_intf signature by replacing endpoint with Endpoint.t everywhere it shows up, and deleting the definition of endpoint from the signature. We can do just this using what’s called destructive substitution. Here’s the basic syntax: 

<Module_type> with type <type> := <type'>
-

This looks just like a sharing constraint, except that we use -:= instead of =. The following shows how we -could use this with Make_interval.

+

This looks just like a sharing constraint, except that we use := instead of =. The following shows how we could use this with Make_interval.

module type Int_interval_intf =
 Interval_intf with type endpoint := int;;
@@ -574,9 +429,7 @@ 
Destructive Substitution

 >  end
-

There’s now no endpoint type: all of its occurrences -have been replaced by int. As with sharing constraints, we -can also use this in the context of a functor:

+

There’s now no endpoint type: all of its occurrences have been replaced by int. As with sharing constraints, we can also use this in the context of a functor:

module Make_interval(Endpoint : Comparable)
   : Interval_intf with type endpoint := Endpoint.t =
@@ -626,11 +479,7 @@ 
Destructive Substitution

 >    end
-

The interface is precisely what we want: the type t is -abstract, and the type of the endpoint is exposed; so we can create -values of type Int_interval.t using the creation function, -but not directly using the constructors and thereby violating the -invariants of the module.

+

The interface is precisely what we want: the type t is abstract, and the type of the endpoint is exposed; so we can create values of type Int_interval.t using the creation function, but not directly using the constructors and thereby violating the invariants of the module.

module Int_interval = Make_interval(Int);;
 >module Int_interval :
@@ -649,44 +498,24 @@ 
Destructive Substitution

 >Error: Unbound constructor Int_interval.Interval
-

In addition, the endpoint type is gone from the -interface, meaning we no longer need to define the endpoint -type alias in the body of the module.

-

It’s worth noting that the name is somewhat misleading, in that -there’s nothing destructive about destructive substitution; it’s really -just a way of creating a new signature by transforming an existing -one.

+

In addition, the endpoint type is gone from the interface, meaning we no longer need to define the endpoint type alias in the body of the module.

+

It’s worth noting that the name is somewhat misleading, in that there’s nothing destructive about destructive substitution; it’s really just a way of creating a new signature by transforming an existing one.

Using Multiple Interfaces

-

Another feature that we might want for our interval module is the -ability to serialize, i.e., to be able to read and write -intervals as a stream of bytes. In this case, we’ll do this by -converting to and from s-expressions, which were mentioned already in Chapter 7, Error -Handling. To recall, an s-expression is essentially a parenthesized -expression whose atoms are strings, and it is a serialization format -that is used commonly in Base. Here’s an example:   

+

Another feature that we might want for our interval module is the ability to serialize, i.e., to be able to read and write intervals as a stream of bytes. In this case, we’ll do this by converting to and from s-expressions, which were mentioned already in Chapter 7, Error Handling. To recall, an s-expression is essentially a parenthesized expression whose atoms are strings, and it is a serialization format that is used commonly in Base. Here’s an example:   

Sexp.List [ Sexp.Atom "This"; Sexp.Atom "is"
 ; Sexp.List [Sexp.Atom "an"; Sexp.Atom "s-expression"]];;
 >- : Sexp.t = (This is (an s-expression))
-

Base is designed to work well with a syntax extension -called ppx_sexp_conv which will generate s-expression -conversion functions for any type annotated with -[@@deriving sexp]. We can enable ppx_sexp_conv -along with a collection of other useful extensions by enabling -ppx_jane:     -   

+

Base is designed to work well with a syntax extension called ppx_sexp_conv which will generate s-expression conversion functions for any type annotated with [@@deriving sexp]. We can enable ppx_sexp_conv along with a collection of other useful extensions by enabling ppx_jane:        

#require "ppx_jane";;
-

Now, we can use the deriving annotation to create sexp-converters -for a given type.

+

Now, we can use the deriving annotation to create sexp-converters for a given type.

type some_type = int * string list [@@deriving sexp];;
 >type some_type = int * string list
@@ -698,9 +527,7 @@ 
Using Multiple Interfaces

 >- : some_type = (44, ["five"; "six"])
-

We’ll discuss s-expressions and Sexplib in more detail in Chapter 20, Data Serialization With S Expressions, but for now, -let’s see what happens if we attach the [@@deriving sexp] -declaration to the definition of t within the functor:

+

We’ll discuss s-expressions and Sexplib in more detail in Chapter 20, Data Serialization With S Expressions, but for now, let’s see what happens if we attach the [@@deriving sexp] declaration to the definition of t within the functor:

module Make_interval(Endpoint : Comparable)
   : (Interval_intf with type endpoint := Endpoint.t) = struct
@@ -743,15 +570,8 @@ 
Using Multiple Interfaces

 >Error: Unbound value Endpoint.t_of_sexp
-

The problem is that [@@deriving sexp] adds code for -defining the s-expression converters, and that code assumes that -Endpoint has the appropriate sexp-conversion functions for -Endpoint.t. But all we know about Endpoint is -that it satisfies the Comparable interface, which doesn’t -say anything about s-expressions.

-

Happily, Base comes with a built-in interface for just -this purpose called Sexpable.S, which is defined as -follows:

+

The problem is that [@@deriving sexp] adds code for defining the s-expression converters, and that code assumes that Endpoint has the appropriate sexp-conversion functions for Endpoint.t. But all we know about Endpoint is that it satisfies the Comparable interface, which doesn’t say anything about s-expressions.

+

Happily, Base comes with a built-in interface for just this purpose called Sexpable.S, which is defined as follows:

sig
   type t
@@ -759,13 +579,7 @@ 
Using Multiple Interfaces

   val t_of_sexp : Sexp.t -> t
 end
-

We can modify Make_interval to use the -Sexpable.S interface, for both its input and its output. -First, let’s create an extended version of the -Interval_intf interface that includes the functions from -the Sexpable.S interface. We can do this using destructive -substitution on the Sexpable.S interface, to avoid having -multiple distinct type t’s clashing with each other:

+

We can modify Make_interval to use the Sexpable.S interface, for both its input and its output. First, let’s create an extended version of the Interval_intf interface that includes the functions from the Sexpable.S interface. We can do this using destructive substitution on the Sexpable.S interface, to avoid having multiple distinct type t’s clashing with each other:

module type Interval_intf_with_sexp = sig
   include Interval_intf
@@ -784,12 +598,7 @@ 
Using Multiple Interfaces

 >  end
-

Equivalently, we can define a type t within our new -module, and apply destructive substitutions to all of the included -interfaces, Interval_intf included, as shown in the -following example. This is somewhat cleaner when combining multiple -interfaces, since it correctly reflects that all of the signatures are -being handled equivalently:

+

Equivalently, we can define a type t within our new module, and apply destructive substitutions to all of the included interfaces, Interval_intf included, as shown in the following example. This is somewhat cleaner when combining multiple interfaces, since it correctly reflects that all of the signatures are being handled equivalently:

module type Interval_intf_with_sexp = sig
   type t
@@ -809,9 +618,7 @@ 
Using Multiple Interfaces

 >  end
-

Now we can write the functor itself. We have been careful to override -the sexp converter here to ensure that the data structure’s invariants -are still maintained when reading in from an s-expression:

+

Now we can write the functor itself. We have been careful to override the sexp converter here to ensure that the data structure’s invariants are still maintained when reading in from an s-expression:

module Make_interval(Endpoint : sig
     type t
@@ -904,12 +711,7 @@ 
Using Multiple Interfaces

Extending Modules

-

Another common use of functors is to generate type-specific -functionality for a given module in a standardized way. Let’s see how -this works in the context of a functional queue, which is just a -functional version of a FIFO (first-in, first-out) queue. Being -functional, operations on the queue return new queues, rather than -modifying the queues that were passed in.   

+

Another common use of functors is to generate type-specific functionality for a given module in a standardized way. Let’s see how this works in the context of a functional queue, which is just a functional version of a FIFO (first-in, first-out) queue. Being functional, operations on the queue return new queues, rather than modifying the queues that were passed in.   

Here’s a reasonable mli for such a module:

type 'a t
@@ -926,20 +728,8 @@ 
Extending Modules

 (** Folds over the queue, from front to back *)
 val fold : 'a t -> init:'acc -> f:('acc -> 'a -> 'acc) -> 'acc
-

The signature of the fold function requires some -explanation. It follows the same pattern as the List.fold -function we described in Chapter 3, Using The List Module Effectively. Essentially, -Fqueue.fold q ~init ~f walks over the elements of -q from front to back, starting with an accumulator of -init and using f to update the accumulator -value as it walks over the queue, returning the final value of the -accumulator at the end of the computation. fold is a quite -powerful operation, as we’ll see.

-

We’ll implement Fqueue using the well known trick of -maintaining an input and an output list so that one can both efficiently -enqueue on the input list and dequeue from the output list. If you -attempt to dequeue when the output list is empty, the input list is -reversed and becomes the new output list. Here’s the implementation:

+

The signature of the fold function requires some explanation. It follows the same pattern as the List.fold function we described in Chapter 3, Using The List Module Effectively. Essentially, Fqueue.fold q ~init ~f walks over the elements of q from front to back, starting with an accumulator of init and using f to update the accumulator value as it walks over the queue, returning the final value of the accumulator at the end of the computation. fold is a quite powerful operation, as we’ll see.

+

We’ll implement Fqueue using the well known trick of maintaining an input and an output list so that one can both efficiently enqueue on the input list and dequeue from the output list. If you attempt to dequeue when the output list is empty, the input list is reversed and becomes the new output list. Here’s the implementation:

open Base
 
@@ -962,26 +752,9 @@ 
Extending Modules

   let after_out = List.fold ~init ~f out_list in
   List.fold_right ~init:after_out ~f:(fun x acc -> f acc x) in_list
-

One problem with Fqueue is that the interface is quite -skeletal. There are lots of useful helper functions that one might want -that aren’t there. The List module, by way of contrast, has -functions like List.iter, which runs a function on each -element; and List.for_all, which returns true if and only -if the given predicate evaluates to true on every element -of the list. Such helper functions come up for pretty much every -container type, and implementing them over and over is a dull and -repetitive affair.

-

As it happens, many of these helper functions can be derived -mechanically from the fold function we already implemented. -Rather than write all of these helper functions by hand for every new -container type, we can instead use a functor to add this functionality -to any container that has a fold function.

-

We’ll create a new module, Foldable, that automates the -process of adding helper functions to a fold-supporting -container. As you can see, Foldable contains a module -signature S which defines the signature that is required to -support folding; and a functor Extend that allows one to -extend any module that matches Foldable.S:

+

One problem with Fqueue is that the interface is quite skeletal. There are lots of useful helper functions that one might want that aren’t there. The List module, by way of contrast, has functions like List.iter, which runs a function on each element; and List.for_all, which returns true if and only if the given predicate evaluates to true on every element of the list. Such helper functions come up for pretty much every container type, and implementing them over and over is a dull and repetitive affair.

+

As it happens, many of these helper functions can be derived mechanically from the fold function we already implemented. Rather than write all of these helper functions by hand for every new container type, we can instead use a functor to add this functionality to any container that has a fold function.

+

We’ll create a new module, Foldable, that automates the process of adding helper functions to a fold-supporting container. As you can see, Foldable contains a module signature S which defines the signature that is required to support folding; and a functor Extend that allows one to extend any module that matches Foldable.S:

open Base
 
@@ -1025,50 +798,27 @@ 
Extending Modules

     with Short_circuit -> true
 end
-

Now we can apply this to Fqueue. We can create an -interface for an extended version of Fqueue as follows:

+

Now we can apply this to Fqueue. We can create an interface for an extended version of Fqueue as follows:

type 'a t
 include (module type of Fqueue) with type 'a t := 'a t
 include Foldable.Extension with type 'a t := 'a t
-

In order to apply the functor, we’ll put the definition of -Fqueue in a submodule called T, and then call -Foldable.Extend on T:

+

In order to apply the functor, we’ll put the definition of Fqueue in a submodule called T, and then call Foldable.Extend on T:

include Fqueue
 include Foldable.Extend(Fqueue)
-

Base comes with a number of functors for extending -modules that follow this same basic pattern, including:     -   

+

Base comes with a number of functors for extending modules that follow this same basic pattern, including:        

    -

  • Container.Make : Very similar to -Foldable.Extend.

    • -

  • Comparable.Make : Adds support for functionality -that depends on the presence of a comparison function, including support -for containers like maps and sets.

    • -

  • Hashable.Make : Adds support for hashing-based data -structures including hash tables, hash sets, and hash heaps.

    • -

  • Monad.Make : For so-called monadic libraries, like -those discussed in Chapters Chapter 7, Error Handling and Chapter 16, Concurrent Programming With Async. Here, the -functor is used to provide a collection of standard helper functions -based on the bind and return -operators.

    • +

  • Container.Make : Very similar to Foldable.Extend.

    • +

  • Comparable.Make : Adds support for functionality that depends on the presence of a comparison function, including support for containers like maps and sets.

    • +

  • Hashable.Make : Adds support for hashing-based data structures including hash tables, hash sets, and hash heaps.

    • +

  • Monad.Make : For so-called monadic libraries, like those discussed in Chapters Chapter 7, Error Handling and Chapter 16, Concurrent Programming With Async. Here, the functor is used to provide a collection of standard helper functions based on the bind and return operators.

-

These functors come in handy when you want to add the same kind of -functionality that is commonly available in Base to your -own types.

-

We’ve really only covered some of the possible uses of functors. -Functors are really a quite powerful tool for modularizing your code. -The cost is that functors are syntactically heavyweight compared to the -rest of the language, and that there are some tricky issues you need to -understand to use them effectively, with sharing constraints and -destructive substitution being high on that list.

-

All of this means that for small and simple programs, heavy use of -functors is probably a mistake. But as your programs get more -complicated and you need more effective modular architectures, functors -become a highly valuable tool.

+

These functors come in handy when you want to add the same kind of functionality that is commonly available in Base to your own types.

+

We’ve really only covered some of the possible uses of functors. Functors are really a quite powerful tool for modularizing your code. The cost is that functors are syntactically heavyweight compared to the rest of the language, and that there are some tricky issues you need to understand to use them effectively, with sharing constraints and destructive substitution being high on that list.

+

All of this means that for small and simple programs, heavy use of functors is probably a mistake. But as your programs get more complicated and you need more effective modular architectures, functors become a highly valuable tool.

-

Next: Chapter 11First-Class Modules

 \ No newline at end of file +

Next: Chapter 11First-Class Modules

 \ No newline at end of file diff --git a/gadts.html b/gadts.html index 48153e17c..1bbda7bd0 100644 --- a/gadts.html +++ b/gadts.html @@ -1,42 +1,18 @@ -GADTs - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+GADTs - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

GADTs

-

Generalized Algebraic Data Types, or GADTs for short, are an -extension of the variants we saw in Chapter 6, Variants. GADTs are more expressive than regular -variants, which helps you create types that more precisely match the -shape of the program you want to write. That can help you write code -that’s safer, more concise, and more efficient.   

-

At the same time, GADTs are an advanced feature of OCaml, and their -power comes at a distinct cost. GADTs are harder to use and less -intuitive than ordinary variants, and it can sometimes be a bit of a -puzzle to figure out how to use them effectively. All of which is to say -that you should only use a GADT when it makes a big qualitative -improvement to your design.

-

That said, for the right use-case, GADTs can be really -transformative, and this chapter will walk through several examples that -demonstrate the range of use-cases that GADTs support.

-

At their heart, GADTs provide two extra features above and beyond -ordinary variants:

+

Generalized Algebraic Data Types, or GADTs for short, are an extension of the variants we saw in Chapter 6, Variants. GADTs are more expressive than regular variants, which helps you create types that more precisely match the shape of the program you want to write. That can help you write code that’s safer, more concise, and more efficient.   

+

At the same time, GADTs are an advanced feature of OCaml, and their power comes at a distinct cost. GADTs are harder to use and less intuitive than ordinary variants, and it can sometimes be a bit of a puzzle to figure out how to use them effectively. All of which is to say that you should only use a GADT when it makes a big qualitative improvement to your design.

+

That said, for the right use-case, GADTs can be really transformative, and this chapter will walk through several examples that demonstrate the range of use-cases that GADTs support.

+

At their heart, GADTs provide two extra features above and beyond ordinary variants:

    -
  • They let the compiler learn more type information when you descend -into a case of a pattern match.
    • -
  • They make it easy to use existential types, which let you -work with data of a specific but unknown type.  
    • +
  • They let the compiler learn more type information when you descend into a case of a pattern match.
    • +
  • They make it easy to use existential types, which let you work with data of a specific but unknown type.  
-

It’s a little hard to understand these features without working -through some examples, so we’ll do that next.

+

It’s a little hard to understand these features without working through some examples, so we’ll do that next.

A Little Language

-

One classic use-case for GADTs is for writing typed expression -languages, similar to the boolean expression language described in Chapter 6, Variants. In this section, we’ll create a slightly -richer language that lets us mix arithmetic and boolean expressions. -This means that we have to deal with the possibility of ill-typed -expressions, e.g., an expression that adds a bool and an -int.

-

Let’s first try to do this with an ordinary variant. We’ll declare -two types here: value, which represents a primitive value -in the language (i.e., an int or a bool), and -expr, which represents the full set of possible -expressions.

+

One classic use-case for GADTs is for writing typed expression languages, similar to the boolean expression language described in Chapter 6, Variants. In this section, we’ll create a slightly richer language that lets us mix arithmetic and boolean expressions. This means that we have to deal with the possibility of ill-typed expressions, e.g., an expression that adds a bool and an int.

+

Let’s first try to do this with an ordinary variant. We’ll declare two types here: value, which represents a primitive value in the language (i.e., an int or a bool), and expr, which represents the full set of possible expressions.

open Base
 
@@ -50,10 +26,7 @@ 
A Little Language

   | Plus of expr * expr
   | If of expr * expr * expr
-

We can write a recursive evaluator for this type in a pretty -straight-ahead style. First, we’ll declare an exception that can be -thrown when we hit an ill-typed expression, e.g., when encountering an -expression that tries to add a bool and an int.

+

We can write a recursive evaluator for this type in a pretty straight-ahead style. First, we’ll declare an exception that can be thrown when we hit an ill-typed expression, e.g., when encountering an expression that tries to add a bool and an int.

exception Ill_typed
@@ -77,9 +50,7 @@

A Little Language

>val eval : expr -> value = <fun>
-

This implementation is a bit ugly because it has a lot of dynamic -checks to detect type errors. Indeed, it’s entirely possible to create -an ill-typed expression which will trip these checks.

+

This implementation is a bit ugly because it has a lot of dynamic checks to detect type errors. Indeed, it’s entirely possible to create an ill-typed expression which will trip these checks.

let i x = Value (Int x)
 and b x = Value (Bool x)
@@ -91,16 +62,10 @@ 
A Little Language

 >Exception: Ill_typed.
-

This possibility of ill-typed expressions doesn’t just complicate the -implementation: it’s also a problem for users, since it’s all too easy -to create ill-typed expressions by mistake.

+

This possibility of ill-typed expressions doesn’t just complicate the implementation: it’s also a problem for users, since it’s all too easy to create ill-typed expressions by mistake.

Making the Language Type-Safe

-

Let’s consider what a type-safe version of this API might look like -in the absence of GADTs. To even express the type constraints, we’ll -need expressions to have a type parameter to distinguish integer -expressions from boolean expressions. Given such a parameter, the -signature for such a language might look like this.

+

Let’s consider what a type-safe version of this API might look like in the absence of GADTs. To even express the type constraints, we’ll need expressions to have a type parameter to distinguish integer expressions from boolean expressions. Given such a parameter, the signature for such a language might look like this.

module type Typesafe_lang_sig = sig
   type 'a t
@@ -119,15 +84,11 @@ 
Making the Language Type-Safe

   val bool_eval : bool t -> bool
 end
-

The functions int_eval and bool_eval -deserve some explanation. You might expect there to be a single -evaluation function, with this signature.

+

The functions int_eval and bool_eval deserve some explanation. You might expect there to be a single evaluation function, with this signature.

val eval : 'a t -> 'a
-

But as we’ll see, we’re not going to be able to implement that, at -least, not without using GADTs. So for now, we’re stuck with two -different evaluators, one for each type of expression.

+

But as we’ll see, we’re not going to be able to implement that, at least, not without using GADTs. So for now, we’re stuck with two different evaluators, one for each type of expression.

Now let’s write an implementation that matches this signature.

module Typesafe_lang : Typesafe_lang_sig = struct
@@ -150,8 +111,7 @@ 
Making the Language Type-Safe

     | Int _ -> raise Ill_typed
 end
-

As you can see, the ill-typed expression we had trouble with before -can’t be constructed, because it’s rejected by OCaml’s type-system.

+

As you can see, the ill-typed expression we had trouble with before can’t be constructed, because it’s rejected by OCaml’s type-system.

let expr = Typesafe_lang.(plus (int 3) (bool false));;
 >Line 1, characters 40-52:
@@ -160,34 +120,20 @@ 
Making the Language Type-Safe

 >       Type bool is not compatible with type int
-

So, what happened here? How did we add the type-safety we wanted? The -fundamental trick is to add what’s called a phantom type. In -this definition:  

+

So, what happened here? How did we add the type-safety we wanted? The fundamental trick is to add what’s called a phantom type. In this definition:  

type 'a t = expr
-

the type parameter 'a is the phantom type, since it -doesn’t show up in the body of the definition of t.

-

Because the type parameter is unused, it’s free to take on any value. -That means we can constrain the use of that type parameter arbitrarily -in the signature, which is a freedom we use to add the type-safety rules -that we wanted.

-

This all amounts to an improvement in terms of the API, but the -implementation is if anything worse. We still have the same evaluator -with all of its dynamic checking for type errors. But we’ve had to write -yet more wrapper code to make this work.

-

Also, the phantom-type discipline is quite error prone. You might -have missed the fact that the type on the eq function above -is wrong!

+

the type parameter 'a is the phantom type, since it doesn’t show up in the body of the definition of t.

+

Because the type parameter is unused, it’s free to take on any value. That means we can constrain the use of that type parameter arbitrarily in the signature, which is a freedom we use to add the type-safety rules that we wanted.

+

This all amounts to an improvement in terms of the API, but the implementation is if anything worse. We still have the same evaluator with all of its dynamic checking for type errors. But we’ve had to write yet more wrapper code to make this work.

+

Also, the phantom-type discipline is quite error prone. You might have missed the fact that the type on the eq function above is wrong!

Typesafe_lang.eq;;
 >- : 'a Typesafe_lang.t -> 'a Typesafe_lang.t -> bool Typesafe_lang.t = <fun>
-

It looks like it’s polymorphic over the type of expressions, but the -evaluator only supports checking equality on integers. As a result, we -can still construct an ill-typed expression, phantom-types -notwithstanding.

+

It looks like it’s polymorphic over the type of expressions, but the evaluator only supports checking equality on integers. As a result, we can still construct an ill-typed expression, phantom-types notwithstanding.

let expr = Typesafe_lang.(eq (bool true) (bool false));;
 >val expr : bool Typesafe_lang.t = <abstr>
@@ -195,23 +141,11 @@ 
Making the Language Type-Safe

 >Exception: Ill_typed.
-

This highlights why we still need the dynamic checks in the -implementation: the types within the implementation don’t necessarily -rule out ill-typed expressions. The same fact explains why we needed two -different eval functions: the implementation of eval -doesn’t have any type-level guarantee of when it’s handling a bool -expression versus an int expression, so it can’t safely give results -where the type of the result varies based on the result of the -expression.

+

This highlights why we still need the dynamic checks in the implementation: the types within the implementation don’t necessarily rule out ill-typed expressions. The same fact explains why we needed two different eval functions: the implementation of eval doesn’t have any type-level guarantee of when it’s handling a bool expression versus an int expression, so it can’t safely give results where the type of the result varies based on the result of the expression.

Trying to Do Better with Ordinary Variants

-

To see why we need GADTs, let’s see how far we can get without them. -In particular, let’s see what happens when we try to encode the typing -rules we want for our DSL directly into the definition of the expression -type. We’ll do that by putting an ordinary type parameter on our -expr and value types, in order to represent -the type of an expression or value.

+

To see why we need GADTs, let’s see how far we can get without them. In particular, let’s see what happens when we try to encode the typing rules we want for our DSL directly into the definition of the expression type. We’ll do that by putting an ordinary type parameter on our expr and value types, in order to represent the type of an expression or value.

type 'a value =
   | Int of 'a
@@ -223,8 +157,7 @@ 
Trying to Do Better with Ordinary Variants

   | Plus of 'a expr * 'a expr
   | If of bool expr * 'a expr * 'a expr
-

This looks promising at first, but it doesn’t quite do what we want. -Let’s experiment a little.

+

This looks promising at first, but it doesn’t quite do what we want. Let’s experiment a little.

let i x = Value (Int x)
 and b x = Value (Bool x)
@@ -240,11 +173,7 @@ 
Trying to Do Better with Ordinary Variants

 >- : int expr = Plus (Value (Int 3), Value (Int 4))
-

So far so good. But if you think about it for a minute, you’ll -realize this doesn’t actually do what we want. For one thing, the type -of the outer expression is always just equal to the type of the inner -expression, which means that some things that should type-check -don’t.

+

So far so good. But if you think about it for a minute, you’ll realize this doesn’t actually do what we want. For one thing, the type of the outer expression is always just equal to the type of the inner expression, which means that some things that should type-check don’t.

If (Eq (i 3, i 4), i 0, i 1);;
 >Line 1, characters 9-12:
@@ -259,17 +188,11 @@ 
Trying to Do Better with Ordinary Variants

 >- : int expr = Value (Bool 3)
-

The problem here is that the way we want to use the type parameter -isn’t supported by ordinary variants. In particular, we want the type -parameter to be populated in different ways in the different tags, and -to depend in non-trivial ways on the types of the data associated with -each tag. That’s where GADTs can help.

+

The problem here is that the way we want to use the type parameter isn’t supported by ordinary variants. In particular, we want the type parameter to be populated in different ways in the different tags, and to depend in non-trivial ways on the types of the data associated with each tag. That’s where GADTs can help.

GADTs to the Rescue

-

Now we’re ready to write our first GADT. Here’s a new version of our -value and expr types that correctly encode our -desired typing rules.

+

Now we’re ready to write our first GADT. Here’s a new version of our value and expr types that correctly encode our desired typing rules.

type _ value =
   | Int : int -> int value
@@ -281,23 +204,8 @@ 
GADTs to the Rescue

   | Plus : int expr * int expr -> int expr
   | If : bool expr * 'a expr * 'a expr -> 'a expr
-

The syntax here requires some decoding. The colon to the right of -each tag is what tells you that this is a GADT. To the right of the -colon, you’ll see what looks like an ordinary, single-argument function -type, and you can almost think of it that way; specifically, as the type -signature for that particular tag, viewed as a type constructor. The -left-hand side of the arrow states the types of the arguments to the -constructor, and the right-hand side determines the type of the -constructed value.

-

In the definition of each tag in a GADT, the right-hand side of the -arrow is an instance of the type of the overall GADT, with independent -choices for the type parameter in each case. Importantly, the type -parameter can depend both on the tag and on the type of the arguments. -Eq is an example where the type parameter is determined -entirely by the tag: it always corresponds to a bool expr. -If is an example where the type parameter depends on the -arguments to the tag, in particular the type parameter of the -If is the type parameter of the then and else clauses.

+

The syntax here requires some decoding. The colon to the right of each tag is what tells you that this is a GADT. To the right of the colon, you’ll see what looks like an ordinary, single-argument function type, and you can almost think of it that way; specifically, as the type signature for that particular tag, viewed as a type constructor. The left-hand side of the arrow states the types of the arguments to the constructor, and the right-hand side determines the type of the constructed value.

+

In the definition of each tag in a GADT, the right-hand side of the arrow is an instance of the type of the overall GADT, with independent choices for the type parameter in each case. Importantly, the type parameter can depend both on the tag and on the type of the arguments. Eq is an example where the type parameter is determined entirely by the tag: it always corresponds to a bool expr. If is an example where the type parameter depends on the arguments to the tag, in particular the type parameter of the If is the type parameter of the then and else clauses.

Let’s try some examples.

let i x = Value (Int x)
@@ -321,13 +229,8 @@ 
GADTs to the Rescue

 >       Type bool is not compatible with type int
-

What we see here is that the type-safety rules we previously enforced -with signature-level restrictions on phantom types are now directly -encoded in the definition of the expression type.

-

These type-safety rules apply not just when constructing an -expression, but also when deconstructing one, which means we can write a -simpler and more concise evaluator that doesn’t need any type-safety -checks.

+

What we see here is that the type-safety rules we previously enforced with signature-level restrictions on phantom types are now directly encoded in the definition of the expression type.

+

These type-safety rules apply not just when constructing an expression, but also when deconstructing one, which means we can write a simpler and more concise evaluator that doesn’t need any type-safety checks.

let eval_value : type a. a value -> a = function
   | Int x -> x
@@ -341,16 +244,11 @@ 
GADTs to the Rescue

 >val eval : 'a expr -> 'a = <fun>
-

Note that we now have a single polymorphic eval function, as opposed -to the two type-specific evaluators we needed when using phantom -types.

+

Note that we now have a single polymorphic eval function, as opposed to the two type-specific evaluators we needed when using phantom types.

GADTs, Locally Abstract Types, and Polymorphic Recursion

-

The above example lets us see one of the downsides of GADTs, which is -that code using them needs extra type annotations. Look at what happens -if we write the definition of value without the -annotation.

+

The above example lets us see one of the downsides of GADTs, which is that code using them needs extra type annotations. Look at what happens if we write the definition of value without the annotation.

let eval_value = function
   | Int x -> x
@@ -361,11 +259,7 @@ 
GADTs, Locally Abstract Types, and Polymorphic Recursion

 >       Type bool is not compatible with type int
-

The issue here is that OCaml by default isn’t willing to instantiate -ordinary type variables in different ways in the body of the same -function, which is what is required here. We can fix that by adding a -locally abstract type, which doesn’t have that restriction. - 

+

The issue here is that OCaml by default isn’t willing to instantiate ordinary type variables in different ways in the body of the same function, which is what is required here. We can fix that by adding a locally abstract type, which doesn’t have that restriction.  

let eval_value (type a) (v : a value) : a =
   match v with
@@ -374,9 +268,7 @@ 
GADTs, Locally Abstract Types, and Polymorphic Recursion

 >val eval_value : 'a value -> 'a = <fun>
-

This isn’t the same annotation we wrote earlier, and indeed, if we -try this approach with eval, we’ll see that it doesn’t -work.

+

This isn’t the same annotation we wrote earlier, and indeed, if we try this approach with eval, we’ll see that it doesn’t work.

let rec eval (type a) (e : a expr) : a =
   match e with
@@ -390,19 +282,12 @@ 
GADTs, Locally Abstract Types, and Polymorphic Recursion

 >       Type a is not compatible with type bool
-

This is a pretty unhelpful error message, but the basic problem is -that eval is recursive, and inference of GADTs doesn’t play -well with recursive calls.

+

This is a pretty unhelpful error message, but the basic problem is that eval is recursive, and inference of GADTs doesn’t play well with recursive calls.

-

More specifically, the issue is that the type-checker is trying to -merge the locally abstract type a into the type of the -recursive function eval, and merging it into the outer -scope within which eval is defined is the way in which -a is escaping its scope.

-

We can fix this by explicitly marking eval as -polymorphic, which OCaml has a handy type annotation for.

+

More specifically, the issue is that the type-checker is trying to merge the locally abstract type a into the type of the recursive function eval, and merging it into the outer scope within which eval is defined is the way in which a is escaping its scope.

+

We can fix this by explicitly marking eval as polymorphic, which OCaml has a handy type annotation for.

let rec eval : 'a. 'a expr -> 'a =
   fun (type a) (x : a expr) ->
@@ -414,25 +299,10 @@ 
GADTs, Locally Abstract Types, and Polymorphic Recursion

 >val eval : 'a expr -> 'a = <fun>
-

This works because by marking eval as polymorphic, the -type of eval isn’t specialized to a, and so -a doesn’t escape its scope.

-

It’s also helpful here because eval itself is an example -of polymorphic recursion, which is to say that -eval needs to call itself at multiple different types. This -comes up, for example, with If, since the If -itself must be of type bool, but the type of the then and -else clauses could be of type int. This means that when -evaluating If, we’ll dispatch eval at a -different type than it was called on.  

-

As such, eval needs to see itself as polymorphic. This -kind of polymorphism is basically impossible to infer automatically, -which is a second reason we need to annotate eval’s -polymorphism explicitly.

-

The above syntax is a bit verbose, so OCaml has syntactic sugar to -combine the polymorphism annotation and the creation of the locally -abstract types:

+

This works because by marking eval as polymorphic, the type of eval isn’t specialized to a, and so a doesn’t escape its scope.

+

It’s also helpful here because eval itself is an example of polymorphic recursion, which is to say that eval needs to call itself at multiple different types. This comes up, for example, with If, since the If itself must be of type bool, but the type of the then and else clauses could be of type int. This means that when evaluating If, we’ll dispatch eval at a different type than it was called on.  

+

As such, eval needs to see itself as polymorphic. This kind of polymorphism is basically impossible to infer automatically, which is a second reason we need to annotate eval’s polymorphism explicitly.

+

The above syntax is a bit verbose, so OCaml has syntactic sugar to combine the polymorphism annotation and the creation of the locally abstract types:

let rec eval : type a. a expr -> a = function
   | Value v -> eval_value v
@@ -442,31 +312,21 @@ 
GADTs, Locally Abstract Types, and Polymorphic Recursion

 >val eval : 'a expr -> 'a = <fun>
-

This type of annotation is the right one to pick when you write any -recursive function that makes use of GADTs.

+

This type of annotation is the right one to pick when you write any recursive function that makes use of GADTs.

When Are GADTs Useful?

-

The typed language we showed above is a perfectly reasonable example, -but GADTs are useful for a lot more than designing little languages. In -this section, we’ll try to give you a broader sampling of the kinds of -things you can do with GADTs.

+

The typed language we showed above is a perfectly reasonable example, but GADTs are useful for a lot more than designing little languages. In this section, we’ll try to give you a broader sampling of the kinds of things you can do with GADTs.

Varying Your Return Type

-

Sometimes, you want to write a single function that can effectively -have different types in different circumstances. In some sense, this is -totally ordinary. After all, OCaml’s polymorphism means that values can -take on different types in different contexts. List.find is -a fine example. The signature indicates that the type of the result -varies with the type of the input list.

+

Sometimes, you want to write a single function that can effectively have different types in different circumstances. In some sense, this is totally ordinary. After all, OCaml’s polymorphism means that values can take on different types in different contexts. List.find is a fine example. The signature indicates that the type of the result varies with the type of the input list.

List.find;;
 >- : 'a list -> f:('a -> bool) -> 'a option = <fun>
-

And of course you can use List.find to produce values -of different types.

+

And of course you can use List.find to produce values of different types.

List.find ~f:(fun x -> x > 3) [1;3;5;2];;
 >- : int option = Some 5
@@ -474,21 +334,14 @@ 
Varying Your Return Type

 >- : char option = Some 'B'
-

But this approach is limited to simple dependencies between types -that correspond to how data flows through your code. Sometimes you want -types to vary in a more flexible way.

-

To make this concrete, let’s say we wanted to create a version of -find that is configurable in terms of how it handles the -case of not finding an item. There are three different behaviors you -might want:

+

But this approach is limited to simple dependencies between types that correspond to how data flows through your code. Sometimes you want types to vary in a more flexible way.

+

To make this concrete, let’s say we wanted to create a version of find that is configurable in terms of how it handles the case of not finding an item. There are three different behaviors you might want:

  • Throw an exception.
  • Return None.
  • Return a default value.
-

Let’s try to write a function that exhibits these behaviors without -using GADTs. First, we’ll create a variant type that represents the -three possible behaviors.

+

Let’s try to write a function that exhibits these behaviors without using GADTs. First, we’ll create a variant type that represents the three possible behaviors.

module If_not_found = struct
   type 'a t =
@@ -497,9 +350,7 @@ 
Varying Your Return Type

     | Default_to of 'a
 end
-

Now we can write flexible_find, which takes an -If_not_found.t as a parameter and varies its behavior -accordingly.

+

Now we can write flexible_find, which takes an If_not_found.t as a parameter and varies its behavior accordingly.

let rec flexible_find list ~f (if_not_found : _ If_not_found.t) =
   match list with
@@ -526,15 +377,8 @@ 
Varying Your Return Type

 >- : int option = Some 20
-

This mostly does what we want, but the problem is that -flexible_find always returns an option, even when it’s -passed Raise or Default_to, which guarantees -that the None case is never used.

-

To eliminate the unnecessary option in the Raise and -Default_to cases, we’re going to turn -If_not_found.t into a GADT. In particular, we’ll mint it as -a GADT with two type parameters: one for the type of the list element, -and one for the return type of the function.

+

This mostly does what we want, but the problem is that flexible_find always returns an option, even when it’s passed Raise or Default_to, which guarantees that the None case is never used.

+

To eliminate the unnecessary option in the Raise and Default_to cases, we’re going to turn If_not_found.t into a GADT. In particular, we’ll mint it as a GADT with two type parameters: one for the type of the list element, and one for the return type of the function.

module If_not_found = struct
   type (_, _) t =
@@ -543,11 +387,8 @@ 
Varying Your Return Type

     | Default_to : 'a -> ('a, 'a) t
 end
-

As you can see, Raise and Default_to both -have the same element type and return type, but Return_none -provides an optional return value.

-

Here’s a definition of flexible_find that takes -advantage of this GADT.  

+

As you can see, Raise and Default_to both have the same element type and return type, but Return_none provides an optional return value.

+

Here’s a definition of flexible_find that takes advantage of this GADT.  

let rec flexible_find
  : type a b. f:(a -> bool) -> a list -> (a, b) If_not_found.t -> b =
@@ -570,11 +411,7 @@ 
Varying Your Return Type

 >  f:('a -> bool) -> 'a list -> ('a, 'b) If_not_found.t -> 'b = <fun>
-

As you can see from the signature of flexible_find, the -return value now depends on the type of If_not_found.t, -which means it can depend on the particular variant of -If_not_found.t that’s in use. As a result, -flexible_find only returns an option when it needs to.

+

As you can see from the signature of flexible_find, the return value now depends on the type of If_not_found.t, which means it can depend on the particular variant of If_not_found.t that’s in use. As a result, flexible_find only returns an option when it needs to.

flexible_find ~f:(fun x -> x > 10) [1;2;5] Return_none;;
 >- : int option = Base.Option.None
@@ -589,20 +426,14 @@ 
Varying Your Return Type

Capturing the Unknown

-

Code that works with unknown types is routine in OCaml, and comes up -in the simplest of examples:

+

Code that works with unknown types is routine in OCaml, and comes up in the simplest of examples:

let tuple x y = (x,y);;
 >val tuple : 'a -> 'b -> 'a * 'b = <fun>
-

The type variables 'a and 'b indicate that -there are two unknown types here, and these type variables are -universally quantified. Which is to say, the type of -tuple is: for all types a and -b, a -> b -> a * b.

-

And indeed, we can restrict the type of tuple to any -'a and 'b we want.

+

The type variables 'a and 'b indicate that there are two unknown types here, and these type variables are universally quantified. Which is to say, the type of tuple is: for all types a and b, a -> b -> a * b.

+

And indeed, we can restrict the type of tuple to any 'a and 'b we want.

(tuple : int -> float -> int * float);;
 >- : int -> float -> int * float = <fun>
@@ -610,33 +441,22 @@ 
Capturing the Unknown

 >- : string -> string * string -> string * (string * string) = <fun>
-

Sometimes, however, we want type variables that are existentially -quantified, meaning that instead of being compatible with all -types, the type represents a particular but unknown type.  

-

GADTs provide one natural way of encoding such type variables. Here’s -a simple example.

+

Sometimes, however, we want type variables that are existentially quantified, meaning that instead of being compatible with all types, the type represents a particular but unknown type.  

+

GADTs provide one natural way of encoding such type variables. Here’s a simple example.

type stringable =
   Stringable : { value: 'a; to_string: 'a -> string } -> stringable
-

This type packs together a value of some arbitrary type, along with -a function for converting values of that type to strings.

-

We can tell that 'a is existentially quantified because -it shows up on the left-hand side of the arrow but not on the right, so -the 'a that shows up internally doesn’t appear in a type -parameter for stringable itself. Essentially, the -existentially quantified type is bound within the definition of -stringable.

-

The following function can print an arbitrary -stringable:

+

This type packs together a value of some arbitrary type, along with a function for converting values of that type to strings.

+

We can tell that 'a is existentially quantified because it shows up on the left-hand side of the arrow but not on the right, so the 'a that shows up internally doesn’t appear in a type parameter for stringable itself. Essentially, the existentially quantified type is bound within the definition of stringable.

+

The following function can print an arbitrary stringable:

let print_stringable (Stringable s) =
   Stdio.print_endline (s.to_string s.value);;
 >val print_stringable : stringable -> unit = <fun>
-

We can use print_stringable on a collection of -stringables of different underlying types.

+

We can use print_stringable on a collection of stringables of different underlying types.

let stringables =
   (let s value to_string = Stringable { to_string; value } in
@@ -655,11 +475,7 @@ 
Capturing the Unknown

 >- : unit = ()
-

The thing that lets this all work is that the type of the underlying -object is existentially bound within the type stringable. -As such, the type of the underlying values can’t escape the scope of -stringable, and any function that tries to return such a -value won’t type-check.

+

The thing that lets this all work is that the type of the underlying object is existentially bound within the type stringable. As such, the type of the underlying values can’t escape the scope of stringable, and any function that tries to return such a value won’t type-check.

let get_value (Stringable s) = s.value;;
 >Line 1, characters 32-39:
@@ -668,30 +484,18 @@ 
Capturing the Unknown

 >       The type constructor $Stringable_'a would escape its scope
-

It’s worth spending a moment to decode this error message, and the -meaning of the type variable $Stringable_'a in particular. -You can think of this variable as having three parts:

+

It’s worth spending a moment to decode this error message, and the meaning of the type variable $Stringable_'a in particular. You can think of this variable as having three parts:

  • The $ marks the variable as an existential.
    • -
  • Stringable is the name of the GADT tag that this -variable came from.
    • -
  • 'a is the name of the type variable from inside that -tag.
    • +
  • Stringable is the name of the GADT tag that this variable came from.
    • +
  • 'a is the name of the type variable from inside that tag.

Abstracting Computational Machines

-

A common idiom in OCaml is to combine small components into larger -computational machines, using a collection of component-combining -functions, or combinators.  

-

GADTs can be helpful for writing such combinators. To see how, let’s -consider an example: pipelines. Here, a pipeline is a sequence -of steps where each step consumes the output of the previous step, -potentially does some side effects, and returns a value to be passed to -the next step. This is analogous to a shell pipeline, and is useful for -all sorts of system automation tasks.

-

But, can’t we write pipelines already? After all, OCaml comes with a -perfectly serviceable pipeline operator:

+

A common idiom in OCaml is to combine small components into larger computational machines, using a collection of component-combining functions, or combinators.  

+

GADTs can be helpful for writing such combinators. To see how, let’s consider an example: pipelines. Here, a pipeline is a sequence of steps where each step consumes the output of the previous step, potentially does some side effects, and returns a value to be passed to the next step. This is analogous to a shell pipeline, and is useful for all sorts of system automation tasks.

+

But, can’t we write pipelines already? After all, OCaml comes with a perfectly serviceable pipeline operator:

open Core;;
@@ -703,20 +507,13 @@ 
Abstracting Computational Machines

 >val sum_file_sizes : unit -> int = <fun>
-

This works well enough, but the advantage of a custom pipeline type -is that it lets you build extra services beyond basic execution of the -pipeline, e.g.:

+

This works well enough, but the advantage of a custom pipeline type is that it lets you build extra services beyond basic execution of the pipeline, e.g.:

    -
  • Profiling, so that when you run a pipeline, you get a report of how -long each step of the pipeline took.
    • -
  • Control over execution, like allowing users to pause the pipeline -mid-execution, and restart it later.
    • -
  • Custom error handling, so, for example, you could build a pipeline -that kept track of where it failed, and offered the possibility of -restarting it.
    • +
  • Profiling, so that when you run a pipeline, you get a report of how long each step of the pipeline took.
    • +
  • Control over execution, like allowing users to pause the pipeline mid-execution, and restart it later.
    • +
  • Custom error handling, so, for example, you could build a pipeline that kept track of where it failed, and offered the possibility of restarting it.
-

The type signature of such a pipeline type might look something like -this:

+

The type signature of such a pipeline type might look something like this:

module type Pipeline = sig
   type ('input,'output) t
@@ -725,16 +522,8 @@ 
Abstracting Computational Machines

   val empty : ('a,'a) t
 end
-

Here, the type ('a,'b) t represents a pipeline that -consumes values of type 'a and emits values of type -'b. The operator @> lets you add a step to -a pipeline by providing a function to prepend on to an existing -pipeline, and empty gives you an empty pipeline, which can -be used to seed the pipeline.

-

The following shows how we could use this API for building a pipeline -like our earlier example using |>. Here, we’re using a -functor, which we’ll see in more detail in Chapter 10, Functors, as a way to -write code using the pipeline API before we’ve implemented it.

+

Here, the type ('a,'b) t represents a pipeline that consumes values of type 'a and emits values of type 'b. The operator @> lets you add a step to a pipeline by providing a function to prepend on to an existing pipeline, and empty gives you an empty pipeline, which can be used to seed the pipeline.

+

The following shows how we could use this API for building a pipeline like our earlier example using |>. Here, we’re using a functor, which we’ll see in more detail in Chapter 10, Functors, as a way to write code using the pipeline API before we’ve implemented it.

module Example_pipeline (Pipeline : Pipeline) = struct
   open Pipeline
@@ -750,10 +539,7 @@ 
Abstracting Computational Machines

 >    sig val sum_file_sizes : (unit, int) Pipeline.t end
-

If all we want is a pipeline capable of a no-frills execution, we can -define our pipeline itself as a simple function, the @> -operator as function composition. Then executing the pipeline is just -function application.

+

If all we want is a pipeline capable of a no-frills execution, we can define our pipeline itself as a simple function, the @> operator as function composition. Then executing the pipeline is just function application.

module Basic_pipeline : sig
    include Pipeline
@@ -769,30 +555,16 @@ 
Abstracting Computational Machines

   let exec t input = t input
 end
-

But this way of implementing a pipeline doesn’t give us any of the -extra services we discussed. All we’re really doing is step-by-step -building up the same kind of function that we could have gotten using -the |> operator.

-

We could get a more powerful pipeline by simply enhancing the -pipeline type, providing it with extra runtime structures to track -profiles, or handle exceptions, or provide whatever else is needed for -the particular use-case. But this approach is awkward, since it requires -us to pre-commit to whatever services we’re going to support, and to -embed all of them in our pipeline representation.

-

GADTs provide a simpler approach. Instead of concretely building a -machine for executing a pipeline, we can use GADTs to abstractly -represent the pipeline we want, and then build the functionality we want -on top of that representation.

+

But this way of implementing a pipeline doesn’t give us any of the extra services we discussed. All we’re really doing is step-by-step building up the same kind of function that we could have gotten using the |> operator.

+

We could get a more powerful pipeline by simply enhancing the pipeline type, providing it with extra runtime structures to track profiles, or handle exceptions, or provide whatever else is needed for the particular use-case. But this approach is awkward, since it requires us to pre-commit to whatever services we’re going to support, and to embed all of them in our pipeline representation.

+

GADTs provide a simpler approach. Instead of concretely building a machine for executing a pipeline, we can use GADTs to abstractly represent the pipeline we want, and then build the functionality we want on top of that representation.

Here’s what such a representation might look like.

type (_, _) pipeline =
   | Step : ('a -> 'b) * ('b, 'c) pipeline -> ('a, 'c) pipeline
   | Empty : ('a, 'a) pipeline
-

The tags here represent the two building blocks of a pipeline: -Step corresponds to the @> operator, and -Empty corresponds to the empty pipeline, as -you can see below.

+

The tags here represent the two building blocks of a pipeline: Step corresponds to the @> operator, and Empty corresponds to the empty pipeline, as you can see below.

let ( @> ) f pipeline = Step (f,pipeline);;
 >val ( @> ) : ('a -> 'b) -> ('b, 'c) pipeline -> ('a, 'c) pipeline = <fun>
@@ -800,8 +572,7 @@ 
Abstracting Computational Machines

 >val empty : ('a, 'a) pipeline = Empty
-

With that in hand, we can do a no-frills pipeline execution easily -enough.

+

With that in hand, we can do a no-frills pipeline execution easily enough.

let rec exec : type a b. (a, b) pipeline -> a -> b =
  fun pipeline input ->
@@ -811,9 +582,7 @@ 
Abstracting Computational Machines

 >val exec : ('a, 'b) pipeline -> 'a -> 'b = <fun>
-

But we can also do more interesting things. For example, here’s a -function that executes a pipeline and produces a profile showing how -long each step of a pipeline took.

+

But we can also do more interesting things. For example, here’s a function that executes a pipeline and produces a profile showing how long each step of a pipeline took.

let exec_with_profile pipeline input =
   let rec loop
@@ -835,37 +604,19 @@ 
Abstracting Computational Machines

 >  <fun>
-

The more abstract GADT approach for creating a little combinator -library like this has several advantages over having combinators that -build a more concrete computational machine:  

+

The more abstract GADT approach for creating a little combinator library like this has several advantages over having combinators that build a more concrete computational machine:  

    -

  • The core types are simpler, since they are typically built out of -GADT tags that are just reflections of the types of the base -combinators.

    • -

  • The design is more modular, since your core types don’t need to -contemplate every possible use you want to make of them.

    • -

  • The code tends to be more efficient, since the more concrete -approach typically involves allocating closures to wrap up the necessary -functionality, and closures are more heavyweight than GADT -tags.

    • +

  • The core types are simpler, since they are typically built out of GADT tags that are just reflections of the types of the base combinators.

    • +

  • The design is more modular, since your core types don’t need to contemplate every possible use you want to make of them.

    • +

  • The code tends to be more efficient, since the more concrete approach typically involves allocating closures to wrap up the necessary functionality, and closures are more heavyweight than GADT tags.

Narrowing the Possibilities

-

Another use-case for GADTs is to narrow the set of possible states -for a given data-type in different circumstances.

-

One context where this can be useful is when managing complex -application state, where the available data changes over time. Let’s -consider a simple example, where we’re writing code to handle a logon -request from a user, and we want to check if the user in question is -authorized to logon.

-

We’ll assume that the user logging in is authenticated as a -particular name, but that in order to authenticate, we need to do two -things: to translate that user-name into a numeric user-id, and to fetch -permissions for the service in question; once we have both, we can check -if the user-id is permitted to log on.

-

Without GADTs, we might model the state of a single logon request as -follows.

+

Another use-case for GADTs is to narrow the set of possible states for a given data-type in different circumstances.

+

One context where this can be useful is when managing complex application state, where the available data changes over time. Let’s consider a simple example, where we’re writing code to handle a logon request from a user, and we want to check if the user in question is authorized to logon.

+

We’ll assume that the user logging in is authenticated as a particular name, but that in order to authenticate, we need to do two things: to translate that user-name into a numeric user-id, and to fetch permissions for the service in question; once we have both, we can check if the user-id is permitted to log on.

+

Without GADTs, we might model the state of a single logon request as follows.

type logon_request =
   { user_name : User_name.t
@@ -873,12 +624,8 @@ 
Narrowing the Possibilities

   ; permissions : Permissions.t option
   }
-

Here, User_name.t represents a textual name, -User_id.t represents an integer identifier associated with -a user, and a Permissions.t lets you determine which -User_id.t’s are authorized to log in.

-

Here’s how we might write a function for testing whether a given -request is authorized.

+

Here, User_name.t represents a textual name, User_id.t represents an integer identifier associated with a user, and a Permissions.t lets you determine which User_id.t’s are authorized to log in.

+

Here’s how we might write a function for testing whether a given request is authorized.

let authorized request =
   match request.user_id, request.permissions with
@@ -889,65 +636,33 @@ 
Narrowing the Possibilities

 >val authorized : logon_request -> (bool, string) result = <fun>
-

The intent is to only call this function once the data is complete, -i.e., when the user_id and permissions fields -have been filled in, which is why it errors out if the data is -incomplete.

-

The code above works just fine for a simple case like this. But in a -real system, your code can get more complicated in multiple ways, -e.g.,

+

The intent is to only call this function once the data is complete, i.e., when the user_id and permissions fields have been filled in, which is why it errors out if the data is incomplete.

+

The code above works just fine for a simple case like this. But in a real system, your code can get more complicated in multiple ways, e.g.,

  • more fields to manage, including more optional fields,
  • more operations that depend on these optional fields,
    • -
  • multiple requests to be handled in parallel, each of which might be -in a different state of completion.
    • +
  • multiple requests to be handled in parallel, each of which might be in a different state of completion.
-

As this kind of complexity creeps in, it can be useful to be able to -track the state of a given request at the type level, and to use that to -narrow the set of states a given request can be in, thereby removing -some extra case analysis and error handling, which can reduce the -complexity of the code and remove opportunities for mistakes.

-

One way of doing this is to mint different types to represent -different states of the request, e.g., one type for an incomplete -request where various fields are optional, and a different type where -all of the data is mandatory.

-

While this works, it can be awkward and verbose. With GADTs, we can -track the state of the request in a type parameter, and have that -parameter be used to narrow the set of available cases, without -duplicating the type.

+

As this kind of complexity creeps in, it can be useful to be able to track the state of a given request at the type level, and to use that to narrow the set of states a given request can be in, thereby removing some extra case analysis and error handling, which can reduce the complexity of the code and remove opportunities for mistakes.

+

One way of doing this is to mint different types to represent different states of the request, e.g., one type for an incomplete request where various fields are optional, and a different type where all of the data is mandatory.

+

While this works, it can be awkward and verbose. With GADTs, we can track the state of the request in a type parameter, and have that parameter be used to narrow the set of available cases, without duplicating the type.

A Completion-Sensitive Option Type

-

We’ll start by creating an option type that is sensitive to whether -our request is in a complete or incomplete state. To do that, we’ll mint -types to represent the states of being complete and incomplete.

+

We’ll start by creating an option type that is sensitive to whether our request is in a complete or incomplete state. To do that, we’ll mint types to represent the states of being complete and incomplete.

type incomplete = Incomplete
 type complete = Complete
-

The definition of the types doesn’t really matter, since we’re never -instantiating these types, just using them as markers of different -states. All that matters is that the types are distinct.

-

Now we can mint a completeness-sensitive option type. Note the two -type variables: the first indicates the type of the contents of the -option, and the second indicates whether this is being used in an -incomplete state.

+

The definition of the types doesn’t really matter, since we’re never instantiating these types, just using them as markers of different states. All that matters is that the types are distinct.

+

Now we can mint a completeness-sensitive option type. Note the two type variables: the first indicates the type of the contents of the option, and the second indicates whether this is being used in an incomplete state.

type (_, _) coption =
   | Absent : (_, incomplete) coption
   | Present : 'a -> ('a, _) coption
-

We use Absent and Present rather than -Some or None to make the code less confusing -when both option and coption are used -together.

-

You might notice that we haven’t used complete here -explicitly. Instead, what we’ve done is to ensure that only an -incomplete coption can be Absent. Accordingly, -a coption that’s complete (and therefore not -incomplete) can only be Present.

-

This is easier to understand with some examples. Consider the -following function for getting the value out of a coption, -returning a default value if Absent is found.

+

We use Absent and Present rather than Some or None to make the code less confusing when both option and coption are used together.

+

You might notice that we haven’t used complete here explicitly. Instead, what we’ve done is to ensure that only an incomplete coption can be Absent. Accordingly, a coption that’s complete (and therefore not incomplete) can only be Present.

+

This is easier to understand with some examples. Consider the following function for getting the value out of a coption, returning a default value if Absent is found.

let get ~default o =
    match o with
@@ -956,9 +671,7 @@ 
A Completion-Sensitive Option Type

 >val get : default:'a -> ('a, incomplete) coption -> 'a = <fun>
-

Note that the incomplete type was inferred here. If we -annotate the coption as complete, the code no -longer compiles.

+

Note that the incomplete type was inferred here. If we annotate the coption as complete, the code no longer compiles.

let get ~default (o : (_,complete) coption) =
   match o with
@@ -971,8 +684,7 @@ 
A Completion-Sensitive Option Type

 >       Type incomplete is not compatible with type complete
-

We can make this compile by deleting the Absent branch -(and the now useless default argument).

+

We can make this compile by deleting the Absent branch (and the now useless default argument).

let get (o : (_,complete) coption) =
   match o with
@@ -986,14 +698,11 @@ 
A Completion-Sensitive Option Type

 >val get : ('a, complete) coption -> 'a = <fun>
-

As we can see, when the coption is known to be -complete, the pattern matching is narrowed to just the -Present case.

+

As we can see, when the coption is known to be complete, the pattern matching is narrowed to just the Present case.

A Completion-Sensitive Request Type

-

We can use coption to define a completion-sensitive -version of logon_request.

+

We can use coption to define a completion-sensitive version of logon_request.

type 'c logon_request =
   { user_name : User_name.t
@@ -1001,12 +710,8 @@ 
A Completion-Sensitive Request Type

   ; permissions : (Permissions.t, 'c) coption
   }
-

There’s a single type parameter for the logon_request -that marks whether it’s complete, at which point, both the -user_id and permissions fields will be -complete as well.

-

As before, it’s easy to fill in the user_id and -permissions fields.

+

There’s a single type parameter for the logon_request that marks whether it’s complete, at which point, both the user_id and permissions fields will be complete as well.

+

As before, it’s easy to fill in the user_id and permissions fields.

let set_user_id request x = { request with user_id = Present x };;
 >val set_user_id : 'a logon_request -> User_id.t -> 'a logon_request = <fun>
@@ -1015,10 +720,7 @@ 
A Completion-Sensitive Request Type

 >  <fun>
-

Note that filling in the fields doesn’t automatically mark a request -as complete. To do that, we need to explicitly test for -completeness, and then construct a version of the record with just the -completed fields filled in.

+

Note that filling in the fields doesn’t automatically mark a request as complete. To do that, we need to explicitly test for completeness, and then construct a version of the record with just the completed fields filled in.

let check_completeness request =
   match request.user_id, request.permissions with
@@ -1029,11 +731,7 @@ 
A Completion-Sensitive Request Type

 >  <fun>
-

The result is polymorphic, meaning it can return a logon request of -any kind, which includes the possibility of returning a complete -request. In practice, the function type is easier to understand if we -constrain the return value to explicitly return a complete -request.

+

The result is polymorphic, meaning it can return a logon request of any kind, which includes the possibility of returning a complete request. In practice, the function type is easier to understand if we constrain the return value to explicitly return a complete request.

let check_completeness request : complete logon_request option =
   match request.user_id, request.permissions with
@@ -1044,8 +742,7 @@ 
A Completion-Sensitive Request Type

 >  incomplete logon_request -> complete logon_request option = <fun>
-

Finally, we can write an authorization checker that works -unconditionally on a complete login request.

+

Finally, we can write an authorization checker that works unconditionally on a complete login request.

let authorized (request : complete logon_request) =
   let { user_id = Present user_id; permissions = Present permissions; _ } = request in
@@ -1053,33 +750,21 @@ 
A Completion-Sensitive Request Type

 >val authorized : complete logon_request -> bool = <fun>
-

After all that work, the result may seem a bit underwhelming, and -indeed, most of the time, this kind of narrowing isn’t worth the -complexity of setting it up. But for a sufficiently complex state -machine, cutting down on the possibilities that your code needs to -contemplate can make a big difference to the comprehensibility and -correctness of the result.

+

After all that work, the result may seem a bit underwhelming, and indeed, most of the time, this kind of narrowing isn’t worth the complexity of setting it up. But for a sufficiently complex state machine, cutting down on the possibilities that your code needs to contemplate can make a big difference to the comprehensibility and correctness of the result.

Type Distinctness and Abstraction

-

In the example in this section, we used two types, -complete and incomplete to mark different -states, and we defined those types so as to be in some sense obviously -different.

+

In the example in this section, we used two types, complete and incomplete to mark different states, and we defined those types so as to be in some sense obviously different.

type incomplete = Incomplete
 type complete = Complete
-

This isn’t strictly necessary. Here’s another way of defining these -types that makes them less obviously distinct.

+

This isn’t strictly necessary. Here’s another way of defining these types that makes them less obviously distinct.

type incomplete = Z
 type complete = Z
-

OCaml’s variant types are nominal, so complete and -incomplete are distinct types, despite having variants of -the same name, as you can see when we try to put instances of each type -in the same list.

+

OCaml’s variant types are nominal, so complete and incomplete are distinct types, despite having variants of the same name, as you can see when we try to put instances of each type in the same list.

let i = (Z : incomplete) and c = (Z : complete);;
 >val i : incomplete = Z
@@ -1090,17 +775,13 @@ 
Type Distinctness and Abstraction

 >       but an expression was expected of type incomplete
-

As a result, we can narrow a pattern match using these types as -indices, much as we did earlier. First, we set up the -coption type:

+

As a result, we can narrow a pattern match using these types as indices, much as we did earlier. First, we set up the coption type:

type ('a, _) coption =
   | Absent : (_, incomplete) coption
   | Present : 'a -> ('a, _) coption
-

Then, we write a function that requires the coption to -be complete, and accordingly, need only contemplate the -Present case.

+

Then, we write a function that requires the coption to be complete, and accordingly, need only contemplate the Present case.

let assume_complete (coption : (_,complete) coption) =
   match coption with
@@ -1108,10 +789,7 @@ 
Type Distinctness and Abstraction

 >val assume_complete : ('a, complete) coption -> 'a = <fun>
-

An easy-to-miss issue here is that the way we expose these types -through an interface can cause OCaml to lose track of the distinctness -of the types in question. Consider this version, where we entirely hide -the definition of complete and incomplete.

+

An easy-to-miss issue here is that the way we expose these types through an interface can cause OCaml to lose track of the distinctness of the types in question. Consider this version, where we entirely hide the definition of complete and incomplete.

module M : sig
   type incomplete
@@ -1126,8 +804,7 @@ 
Type Distinctness and Abstraction

   | Absent : (_, incomplete) coption
   | Present : 'a -> ('a, _) coption
-

Now, the assume_complete function we wrote is no longer -found to be exhaustive.

+

Now, the assume_complete function we wrote is no longer found to be exhaustive.

let assume_complete (coption : (_,complete) coption) =
   match coption with
@@ -1139,11 +816,8 @@ 
Type Distinctness and Abstraction

 >val assume_complete : ('a, complete) coption -> 'a = <fun>
-

That’s because by leaving the types abstract, we’ve entirely hidden -the underlying types, leaving the type system with no evidence that the -types are distinct.

-

Let’s see what happens if we expose the implementation of these -types.

+

That’s because by leaving the types abstract, we’ve entirely hidden the underlying types, leaving the type system with no evidence that the types are distinct.

+

Let’s see what happens if we expose the implementation of these types.

module M : sig
   type incomplete = Z
@@ -1170,12 +844,8 @@ 
Type Distinctness and Abstraction

 >val assume_complete : ('a, complete) coption -> 'a = <fun>
-

In order to be exhaustive, we need the types that are exposed to be -definitively different, which would be the case if we defined them as -variants with differently named tags, as we did originally.

-

The reason for this is that types that appear to be different in an -interface may turn out to be the same in the implementation, as we can -see below.

+

In order to be exhaustive, we need the types that are exposed to be definitively different, which would be the case if we defined them as variants with differently named tags, as we did originally.

+

The reason for this is that types that appear to be different in an interface may turn out to be the same in the implementation, as we can see below.

module M : sig
   type incomplete = Z
@@ -1185,30 +855,17 @@ 
Type Distinctness and Abstraction

   type complete = incomplete = Z
 end
-

All of which is to say: when creating types to act as abstract -markers for the type parameter of a GADT, you should choose definitions -that make the distinctness of those types clear, and you should expose -those definitions in your mlis.

+

All of which is to say: when creating types to act as abstract markers for the type parameter of a GADT, you should choose definitions that make the distinctness of those types clear, and you should expose those definitions in your mlis.

Narrowing Without GADTs

-

Thus far, we’ve only seen narrowing in the context of GADTs, but -OCaml can eliminate impossible cases from ordinary variants too. As with -GADTs, to eliminate a case you need to demonstrate that the case in -question is impossible at the type level.

-

One way to do this is via an uninhabited type, which is a -type that has no associated values. You can declare such a value by -creating a variant with no tags.   

+

Thus far, we’ve only seen narrowing in the context of GADTs, but OCaml can eliminate impossible cases from ordinary variants too. As with GADTs, to eliminate a case you need to demonstrate that the case in question is impossible at the type level.

+

One way to do this is via an uninhabited type, which is a type that has no associated values. You can declare such a value by creating a variant with no tags.   

type nothing = |
-

This turns out to be useful enough that Base has a -standard uninhabited type, Nothing.t.  

-

So, how does an uninhabited type help? Well, consider the -Result.t type, discussed in as described in Chapter 7, Error Handling. Normally, to match a -Result.t, you need to handle both the Ok and -Error cases.

+

This turns out to be useful enough that Base has a standard uninhabited type, Nothing.t.  

+

So, how does an uninhabited type help? Well, consider the Result.t type, discussed in as described in Chapter 7, Error Handling. Normally, to match a Result.t, you need to handle both the Ok and Error cases.

open Stdio;;
 let print_result (x : (int,string) Result.t) =
@@ -1218,9 +875,7 @@ 
Narrowing Without GADTs

 >val print_result : (int, string) result -> unit = <fun>
-

But if the Error case contains an uninhabitable type, -well, that case can never be instantiated, and OCaml will tell you as -much.

+

But if the Error case contains an uninhabitable type, well, that case can never be instantiated, and OCaml will tell you as much.

let print_result (x : (int, Nothing.t) Result.t) =
   match x with
@@ -1232,8 +887,7 @@ 
Narrowing Without GADTs

 >val print_result : (int, Nothing.t) result -> unit = <fun>
-

We can follow the advice above, and add a so-called refutation -case.  

+

We can follow the advice above, and add a so-called refutation case.  

let print_result (x : (int, Nothing.t) Result.t) =
   match x with
@@ -1242,10 +896,7 @@ 
Narrowing Without GADTs

 >val print_result : (int, Nothing.t) result -> unit = <fun>
-

The period in the final case tells the compiler that we believe this -case can never be reached, and OCaml will verify that it’s true. In some -simple cases, however, the compiler can automatically add the refutation -case for you, so you don’t need to write it out explicitly.

+

The period in the final case tells the compiler that we believe this case can never be reached, and OCaml will verify that it’s true. In some simple cases, however, the compiler can automatically add the refutation case for you, so you don’t need to write it out explicitly.

let print_result (x : (int, Nothing.t) Result.t) =
   match x with
@@ -1253,25 +904,14 @@ 
Narrowing Without GADTs

 >val print_result : (int, Nothing.t) result -> unit = <fun>
-

Narrowing with uninhabitable types can be useful when using a highly -configurable library that supports multiple different modes of use, not -all of which are necessarily needed for a given application. One example -of this comes from Async’s RPC (remote procedure-call) -library. Async RPCs support a particular flavor of interaction called a -State_rpc. Such an RPC is parameterized by four types, for -four different kinds of data:    

+

Narrowing with uninhabitable types can be useful when using a highly configurable library that supports multiple different modes of use, not all of which are necessarily needed for a given application. One example of this comes from Async’s RPC (remote procedure-call) library. Async RPCs support a particular flavor of interaction called a State_rpc. Such an RPC is parameterized by four types, for four different kinds of data:    

  • query, for the initial client request,
    • -
  • state, for the initial snapshot returned by the -server,
    • -
  • update, for the sequence of updates to that snapshot, -and
    • +
  • state, for the initial snapshot returned by the server,
    • +
  • update, for the sequence of updates to that snapshot, and
  • error, for an error to terminate the stream.
-

Now, imagine you want to use a State_rpc in a context -where you don’t need to terminate the stream with a custom error. We -could just instantiate the State_rpc using the type -unit for the error type.

+

Now, imagine you want to use a State_rpc in a context where you don’t need to terminate the stream with a custom error. We could just instantiate the State_rpc using the type unit for the error type.

open Core
 open Async
@@ -1285,8 +925,7 @@ 
Narrowing Without GADTs

     ~bin_error:[%bin_type_class: unit]
     ()
-

But with this approach, you still have to handle the error case when -writing code to dispatch the RPC.

+

But with this approach, you still have to handle the error case when writing code to dispatch the RPC.

let dispatch conn =
   match%bind Rpc.State_rpc.dispatch rpc conn () >>| ok_exn with
@@ -1295,8 +934,7 @@ 
Narrowing Without GADTs

 >val dispatch : Rpc.Connection.t -> unit Deferred.t = <fun>
-

An alternative approach is to use an uninhabited type for the -error:

+

An alternative approach is to use an uninhabited type for the error:

let rpc =
   Rpc.State_rpc.create
@@ -1308,9 +946,7 @@ 
Narrowing Without GADTs

     ~bin_error:[%bin_type_class: Nothing.t]
     ()
-

Now, we’ve essentially banned the use of the error type, -and as a result, our dispatch function needs only deal with the -Ok case.

+

Now, we’ve essentially banned the use of the error type, and as a result, our dispatch function needs only deal with the Ok case.

let dispatch conn =
   match%bind Rpc.State_rpc.dispatch rpc conn () >>| ok_exn with
@@ -1318,22 +954,16 @@ 
Narrowing Without GADTs

 >val dispatch : Rpc.Connection.t -> unit Deferred.t = <fun>
-

What’s nice about this example is that it shows that narrowing can be -applied to code that isn’t designed with narrowing in mind.

+

What’s nice about this example is that it shows that narrowing can be applied to code that isn’t designed with narrowing in mind.

Limitations of GADTs

-

Hopefully, we’ve demonstrated the utility of GADTs, while at the same -time showing some of the attendant complexities. In this final section, -we’re going to highlight some remaining difficulties with using GADTs -that you may run into, as well as how to work around them.

+

Hopefully, we’ve demonstrated the utility of GADTs, while at the same time showing some of the attendant complexities. In this final section, we’re going to highlight some remaining difficulties with using GADTs that you may run into, as well as how to work around them.

Or-Patterns

-

GADTs don’t work well with or-patterns. Consider the following type -that represents various ways we might use for obtaining some piece of -data.  

+

GADTs don’t work well with or-patterns. Consider the following type that represents various ways we might use for obtaining some piece of data.  

open Core
 module Source_kind = struct
@@ -1343,8 +973,7 @@ 
Or-Patterns

     | Raw_data : string t
 end
-

We can write a function that takes a Source_kind.t and -the corresponding source, and prints it out.

+

We can write a function that takes a Source_kind.t and the corresponding source, and prints it out.

let source_to_sexp (type a) (kind : a Source_kind.t) (source : a) =
   match kind with
@@ -1354,9 +983,7 @@ 
Or-Patterns

 >val source_to_sexp : 'a Source_kind.t -> 'a -> Sexp.t = <fun>
-

But, observing that the right-hand side of Raw_data and -Filename are the same, you might try to merge those cases -together with an or-pattern. Unfortunately, that doesn’t work.

+

But, observing that the right-hand side of Raw_data and Filename are the same, you might try to merge those cases together with an or-pattern. Unfortunately, that doesn’t work.

let source_to_sexp (type a) (kind : a Source_kind.t) (source : a) =
   match kind with
@@ -1367,9 +994,7 @@ 
Or-Patterns

 >         string
-

Or-patterns do sometimes work, but only when you don’t make use of -the type information that is discovered during the pattern match. Here’s -an example of a function that uses or-patterns successfully.

+

Or-patterns do sometimes work, but only when you don’t make use of the type information that is discovered during the pattern match. Here’s an example of a function that uses or-patterns successfully.

let requires_io (type a) (kind : a Source_kind.t) =
   match kind with
@@ -1378,20 +1003,11 @@ 
Or-Patterns

 >val requires_io : 'a Source_kind.t -> bool = <fun>
-

In any case, the lack of or-patterns is annoying, but it’s not a big -deal, since you can reduce the code duplication by pulling out most of -the content of the duplicated right-hand sides into functions that can -be called in each of the duplicated cases.

+

In any case, the lack of or-patterns is annoying, but it’s not a big deal, since you can reduce the code duplication by pulling out most of the content of the duplicated right-hand sides into functions that can be called in each of the duplicated cases.

Deriving Serializers

-

As will be discussed in more detail in Chapter 20, Data Serialization With S-Expressions, -s-expressions are a convenient data format for representing structured -data. Rather than write the serializers and deserializers by hand, we -typically use ppx_sexp_value, which is a syntax extension -which auto-generates these functions for a given type, based on that -type’s definition.    

+

As will be discussed in more detail in Chapter 20, Data Serialization With S-Expressions, s-expressions are a convenient data format for representing structured data. Rather than write the serializers and deserializers by hand, we typically use ppx_sexp_value, which is a syntax extension which auto-generates these functions for a given type, based on that type’s definition.    

Here’s an example:

type position = { x: float; y: float } [@@deriving sexp];;
@@ -1404,8 +1020,7 @@ 
Deriving Serializers

 >- : position = {x = 72.; y = 1.2}
-

While [@@deriving sexp] works with most types, it -doesn’t always work with GADTs.

+

While [@@deriving sexp] works with most types, it doesn’t always work with GADTs.

type _ number_kind =
   | Int : int number_kind
@@ -1417,17 +1032,8 @@ 
Deriving Serializers

 >       Type int is not compatible with type a__007_
-

The error message is pretty awful, but if you stop and think about -it, it’s not too surprising that we ran into trouble here. What should -the type of number_kind_of_sexp be anyway? When parsing -"Int", the returned type would have to be -int number_kind, and when parsing "Float", the -type would have to be float number_kind. That kind of -dependency between the value of an argument and the type of the returned -value is just not expressible in OCaml’s type system.

-

This argument doesn’t stop us from serializing, and indeed, -[@@deriving sexp_of], which only creates the serializer, -works just fine.

+

The error message is pretty awful, but if you stop and think about it, it’s not too surprising that we ran into trouble here. What should the type of number_kind_of_sexp be anyway? When parsing "Int", the returned type would have to be int number_kind, and when parsing "Float", the type would have to be float number_kind. That kind of dependency between the value of an argument and the type of the returned value is just not expressible in OCaml’s type system.

+

This argument doesn’t stop us from serializing, and indeed, [@@deriving sexp_of], which only creates the serializer, works just fine.

type _ number_kind =
  | Int : int number_kind
@@ -1440,20 +1046,15 @@ 
Deriving Serializers

 >- : Sexp.t = Int
-

It is possible to build a deserializer for number_kind, -but it’s tricky. First, we’ll need a type that packs up a -number_kind while hiding its type parameter. This is going -to be the value we return from our parser.

+

It is possible to build a deserializer for number_kind, but it’s tricky. First, we’ll need a type that packs up a number_kind while hiding its type parameter. This is going to be the value we return from our parser.

type packed_number_kind = P : _ number_kind -> packed_number_kind
-

Next, we’ll need to create a non-GADT version of our type, for which -we’ll derive a deserializer.

+

Next, we’ll need to create a non-GADT version of our type, for which we’ll derive a deserializer.

type simple_number_kind = Int | Float [@@deriving of_sexp]
-

Then, we write a function for converting from our non-GADT type to -the packed variety.

+

Then, we write a function for converting from our non-GADT type to the packed variety.

let simple_number_kind_to_packed_number_kind kind :
   packed_number_kind
@@ -1465,8 +1066,7 @@ 
Deriving Serializers

 >  simple_number_kind -> packed_number_kind = <fun>
-

Finally, we combine our generated sexp-converter with our conversion -type to produce the full deserialization function.

+

Finally, we combine our generated sexp-converter with our conversion type to produce the full deserialization function.

let number_kind_of_sexp sexp =
   simple_number_kind_of_sexp sexp
@@ -1481,9 +1081,8 @@ 
Deriving Serializers

 >- : packed_number_kind list = [P Float; P Int]
-

While all of this is doable, it’s definitely awkward, and requires -some unpleasant code duplication.

+

While all of this is doable, it’s definitely awkward, and requires some unpleasant code duplication.

-

Next: Chapter 10Functors

 \ No newline at end of file +

Next: Chapter 10Functors

 \ No newline at end of file diff --git a/garbage-collector.html b/garbage-collector.html index a039f5270..b3668795a 100644 --- a/garbage-collector.html +++ b/garbage-collector.html @@ -1,162 +1,54 @@ -Understanding the Garbage Collector - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+Understanding the Garbage Collector - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

Understanding the Garbage Collector

-

This chapter includes contributions from Stephen Weeks and Sadiq -Jaffer.

+

This chapter includes contributions from Stephen Weeks and Sadiq Jaffer.

-

We’ve described the runtime format of individual OCaml variables -earlier, in Chapter 23, Memory Representation Of Values. When you execute -your program, OCaml manages the lifecycle of these variables by -regularly scanning allocated values and freeing them when they’re no -longer needed. This in turn means that your applications don’t need to -manually implement memory management, and it greatly reduces the -likelihood of memory leaks creeping into your code.  

-

The OCaml runtime is a C library that provides routines that can be -called from running OCaml programs. The runtime manages a heap, -which is a collection of memory regions that it obtains from the -operating system. The runtime uses this memory to hold heap -blocks that it fills up with OCaml values in response to allocation -requests by the OCaml program.      

+

We’ve described the runtime format of individual OCaml variables earlier, in Chapter 23, Memory Representation Of Values. When you execute your program, OCaml manages the lifecycle of these variables by regularly scanning allocated values and freeing them when they’re no longer needed. This in turn means that your applications don’t need to manually implement memory management, and it greatly reduces the likelihood of memory leaks creeping into your code.  

+

The OCaml runtime is a C library that provides routines that can be called from running OCaml programs. The runtime manages a heap, which is a collection of memory regions that it obtains from the operating system. The runtime uses this memory to hold heap blocks that it fills up with OCaml values in response to allocation requests by the OCaml program.      

Mark and Sweep Garbage Collection

-

When there isn’t enough memory available to satisfy an allocation -request from the pool of allocated heap blocks, the runtime system -invokes the garbage collector (GC). An OCaml program can’t explicitly -free a value when it is done with it. Instead, the GC regularly -determines which values are live and which values are -dead, i.e., no longer in use. Dead values are collected and -their memory made available for reuse by the application.   

-

The GC doesn’t keep constant track of values as they are allocated -and used. Instead, it regularly scans them by starting from a set of -root values that the application always has access to (such as -the stack). The GC maintains a directed graph in which heap blocks are -nodes, and there is an edge from heap block b1 to heap -block b2 if some field of b1 is a pointer to -b2.

-

All blocks reachable from the roots by following edges in the graph -must be retained, and unreachable blocks can be reused by the -application. The algorithm used by OCaml to perform this heap traversal -is commonly known as mark and sweep garbage collection, and -we’ll explain it further now.

+

When there isn’t enough memory available to satisfy an allocation request from the pool of allocated heap blocks, the runtime system invokes the garbage collector (GC). An OCaml program can’t explicitly free a value when it is done with it. Instead, the GC regularly determines which values are live and which values are dead, i.e., no longer in use. Dead values are collected and their memory made available for reuse by the application.   

+

The GC doesn’t keep constant track of values as they are allocated and used. Instead, it regularly scans them by starting from a set of root values that the application always has access to (such as the stack). The GC maintains a directed graph in which heap blocks are nodes, and there is an edge from heap block b1 to heap block b2 if some field of b1 is a pointer to b2.

+

All blocks reachable from the roots by following edges in the graph must be retained, and unreachable blocks can be reused by the application. The algorithm used by OCaml to perform this heap traversal is commonly known as mark and sweep garbage collection, and we’ll explain it further now.

Generational Garbage Collection

-

The usual OCaml programming style involves allocating many small -values that are used for a short period of time and then never accessed -again. OCaml takes advantage of this fact to improve performance by -using a generational GC.   

-

A generational GC maintains separate memory regions to hold blocks -based on how long the blocks have been live. OCaml’s heap is split into -two such regions:  

+

The usual OCaml programming style involves allocating many small values that are used for a short period of time and then never accessed again. OCaml takes advantage of this fact to improve performance by using a generational GC.   

+

A generational GC maintains separate memory regions to hold blocks based on how long the blocks have been live. OCaml’s heap is split into two such regions:  

    -

  • A small, fixed-size minor heap where most blocks are -initially allocated

    • -

  • A larger, variable-size major heap for blocks that have -been live longer

    • +

  • A small, fixed-size minor heap where most blocks are initially allocated

    • +

  • A larger, variable-size major heap for blocks that have been live longer

-

A typical functional programming style means that young blocks tend -to die young and old blocks tend to stay around for longer than young -ones. This is often referred to as the generational hypothesis. - 

-

OCaml uses different memory layouts and garbage-collection algorithms -for the major and minor heaps to account for this generational -difference. We’ll explain how they differ in more detail next.

-
-

The Gc Module and OCAMLRUNPARAM

-

OCaml provides several mechanisms to query and alter the behavior of -the runtime system. The Gc module provides this -functionality from within OCaml code, and we’ll frequently refer to it -in the rest of the chapter. As with several other standard library -modules, Core alters the Gc interface from the standard -OCaml library. We’ll assume that you’ve opened Core in our -explanations.   

-

You can also control the behavior of OCaml programs by setting the -OCAMLRUNPARAM environment variable before launching your -application. This lets you set GC parameters without recompiling, for -example to benchmark the effects of different settings. The format of -OCAMLRUNPARAM is documented in the OCaml manual.

-
+

A typical functional programming style means that young blocks tend to die young and old blocks tend to stay around for longer than young ones. This is often referred to as the generational hypothesis.  

+

OCaml uses different memory layouts and garbage-collection algorithms for the major and minor heaps to account for this generational difference. We’ll explain how they differ in more detail next.

+
+

The Gc Module and OCAMLRUNPARAM

+

OCaml provides several mechanisms to query and alter the behavior of the runtime system. The Gc module provides this functionality from within OCaml code, and we’ll frequently refer to it in the rest of the chapter. As with several other standard library modules, Core alters the Gc interface from the standard OCaml library. We’ll assume that you’ve opened Core in our explanations.   

+

You can also control the behavior of OCaml programs by setting the OCAMLRUNPARAM environment variable before launching your application. This lets you set GC parameters without recompiling, for example to benchmark the effects of different settings. The format of OCAMLRUNPARAM is documented in the OCaml manual.

+

The Fast Minor Heap

-

The minor heap is where most of your short-lived values are held. It -consists of one contiguous chunk of virtual memory containing a sequence -of OCaml blocks. If there is space, allocating a new block is a fast, -constant-time operation that requires just a couple of CPU instructions. -    

-

To garbage-collect the minor heap, OCaml uses copying -collection to move all live blocks in the minor heap to the major -heap. This takes work proportional to the number of live blocks in the -minor heap, which is typically small according to the generational -hypothesis. In general, the garbage collector stops the world -(that is, halts the application) while it runs, which is why it’s so -important that it complete quickly to let the application resume running -with minimal interruption.

+

The minor heap is where most of your short-lived values are held. It consists of one contiguous chunk of virtual memory containing a sequence of OCaml blocks. If there is space, allocating a new block is a fast, constant-time operation that requires just a couple of CPU instructions.     

+

To garbage-collect the minor heap, OCaml uses copying collection to move all live blocks in the minor heap to the major heap. This takes work proportional to the number of live blocks in the minor heap, which is typically small according to the generational hypothesis. In general, the garbage collector stops the world (that is, halts the application) while it runs, which is why it’s so important that it complete quickly to let the application resume running with minimal interruption.

Allocating on the Minor Heap

-

The minor heap is a contiguous chunk of virtual memory that is -usually a few megabytes in size so that it can be scanned quickly.  

+

The minor heap is a contiguous chunk of virtual memory that is usually a few megabytes in size so that it can be scanned quickly.  



-

The runtime stores the boundaries of the minor heap in two pointers -that delimit the start and end of the heap region -(caml_young_start and caml_young_end, but we -will drop the caml_young prefix for brevity). The -base is the memory address returned by the system -malloc, and start is aligned against the next -nearest word boundary from base to make it easier to store -OCaml values.

-

In a fresh minor heap, the limit equals the -start, and the current ptr will equal the -end. ptr decreases as blocks are allocated -until it reaches limit, at which point a minor garbage -collection is triggered.

-

Allocating a block in the minor heap just requires ptr -to be decremented by the size of the block (including the header) and a -check that it’s not less than limit. If there isn’t enough -space left for the block without decrementing past limit, a -minor garbage collection is triggered. This is a very fast check (with -no branching) on most CPU architectures.

+

The runtime stores the boundaries of the minor heap in two pointers that delimit the start and end of the heap region (caml_young_start and caml_young_end, but we will drop the caml_young prefix for brevity). The base is the memory address returned by the system malloc, and start is aligned against the next nearest word boundary from base to make it easier to store OCaml values.

+

In a fresh minor heap, the limit equals the start, and the current ptr will equal the end. ptr decreases as blocks are allocated until it reaches limit, at which point a minor garbage collection is triggered.

+

Allocating a block in the minor heap just requires ptr to be decremented by the size of the block (including the header) and a check that it’s not less than limit. If there isn’t enough space left for the block without decrementing past limit, a minor garbage collection is triggered. This is a very fast check (with no branching) on most CPU architectures.

Understanding Allocation

-

You may wonder why limit is required at all, since it -always seems to equal start. It’s because the easiest way -for the runtime to schedule a minor heap collection is by setting -limit to equal end. The next allocation will -never have enough space after this is done and will always trigger a -garbage collection. There are various internal reasons for such early -collections, such as handling pending UNIX signals, but they don’t -ordinarily matter for application code.

-

It is possible to write loops or recurse in a way that may take a -long time to do an allocation - if at all. To ensure that UNIX signals -and other internal bookkeeping that require interrupting the running -OCaml program still happen the compiler introduces poll points -into generated native code.

-

These poll points check ptr against limit -and developers should expect them to be placed at the start of every -function and the back edge of loops. The compiler includes a dataflow -pass that removes all but the minimum set of points necessary to ensure -these checks happen in a bounded amount of time.

+

You may wonder why limit is required at all, since it always seems to equal start. It’s because the easiest way for the runtime to schedule a minor heap collection is by setting limit to equal end. The next allocation will never have enough space after this is done and will always trigger a garbage collection. There are various internal reasons for such early collections, such as handling pending UNIX signals, but they don’t ordinarily matter for application code.

+

It is possible to write loops or recurse in a way that may take a long time to do an allocation - if at all. To ensure that UNIX signals and other internal bookkeeping that require interrupting the running OCaml program still happen the compiler introduces poll points into generated native code.

+

These poll points check ptr against limit and developers should expect them to be placed at the start of every function and the back edge of loops. The compiler includes a dataflow pass that removes all but the minimum set of points necessary to ensure these checks happen in a bounded amount of time.

 

-
-
-

Setting the Size of the Minor Heap

-

The default minor heap size in OCaml is normally 2 MB on 64-bit -platforms, but this is increased to 8 MB if you use Core (which -generally prefers default settings that improve performance, but at the -cost of a bigger memory profile). This setting can be overridden via the -s=<words> argument to OCAMLRUNPARAM. You -can change it after the program has started by calling the -Gc.set function:

+
+

Setting the Size of the Minor Heap

+

The default minor heap size in OCaml is normally 2 MB on 64-bit platforms, but this is increased to 8 MB if you use Core (which generally prefers default settings that improve performance, but at the cost of a bigger memory profile). This setting can be overridden via the s=<words> argument to OCAMLRUNPARAM. You can change it after the program has started by calling the Gc.set function:

open Core;;
 let c = Gc.get ();;
@@ -170,230 +62,89 @@ 
Setting the Size of the Minor Heap

 >- : unit = ()
-

Changing the GC size dynamically will trigger an immediate minor heap -collection. Note that Core increases the default minor heap size from -the standard OCaml installation quite significantly, and you’ll want to -reduce this if running in very memory-constrained environments.

+

Changing the GC size dynamically will trigger an immediate minor heap collection. Note that Core increases the default minor heap size from the standard OCaml installation quite significantly, and you’ll want to reduce this if running in very memory-constrained environments.

+

The Long-Lived Major Heap

-

The major heap is where the bulk of the longer-lived and larger -values in your program are stored. It consists of any number of -noncontiguous chunks of virtual memory, each containing live blocks -interspersed with regions of free memory. The runtime system maintains a -free-list data structure that indexes all the free memory that it has -allocated, and uses it to satisfy allocation requests for OCaml blocks. -     

-

The major heap is typically much larger than the minor heap and can -scale to gigabytes in size. It is cleaned via a mark-and-sweep garbage -collection algorithm that operates in several phases:

+

The major heap is where the bulk of the longer-lived and larger values in your program are stored. It consists of any number of noncontiguous chunks of virtual memory, each containing live blocks interspersed with regions of free memory. The runtime system maintains a free-list data structure that indexes all the free memory that it has allocated, and uses it to satisfy allocation requests for OCaml blocks.      

+

The major heap is typically much larger than the minor heap and can scale to gigabytes in size. It is cleaned via a mark-and-sweep garbage collection algorithm that operates in several phases:

    -

  • The mark phase scans the block graph and marks all live -blocks by setting a bit in the tag of the block header (known as the -color tag).

    • -

  • The sweep phase sequentially scans the heap chunks and -identifies dead blocks that weren’t marked earlier.

    • -

  • The compact phase relocates live blocks into a freshly -allocated heap to eliminate gaps in the free list. This prevents the -fragmentation of heap blocks in long-running programs and normally -occurs much less frequently than the mark and sweep phases.

    • +

  • The mark phase scans the block graph and marks all live blocks by setting a bit in the tag of the block header (known as the color tag).

    • +

  • The sweep phase sequentially scans the heap chunks and identifies dead blocks that weren’t marked earlier.

    • +

  • The compact phase relocates live blocks into a freshly allocated heap to eliminate gaps in the free list. This prevents the fragmentation of heap blocks in long-running programs and normally occurs much less frequently than the mark and sweep phases.

-

A major garbage collection must also stop the world to ensure that -blocks can be moved around without this being observed by the live -application. The mark-and-sweep phases run incrementally over slices of -the heap to avoid pausing the application for long periods of time, and -also precede each slice with a fast minor collection. Only the -compaction phase touches all the memory in one go, and is a relatively -rare operation.

+

A major garbage collection must also stop the world to ensure that blocks can be moved around without this being observed by the live application. The mark-and-sweep phases run incrementally over slices of the heap to avoid pausing the application for long periods of time, and also precede each slice with a fast minor collection. Only the compaction phase touches all the memory in one go, and is a relatively rare operation.

Allocating on the Major Heap

-

The major heap consists of a singly linked list of contiguous memory -chunks sorted in increasing order of virtual address. Each chunk is a -single memory region allocated via malloc(3) and consists of a -header and data area which contains OCaml heap chunks. A heap chunk -header contains:   

+

The major heap consists of a singly linked list of contiguous memory chunks sorted in increasing order of virtual address. Each chunk is a single memory region allocated via malloc(3) and consists of a header and data area which contains OCaml heap chunks. A heap chunk header contains:   

    -

  • The malloced virtual address of the memory region -containing the chunk

    • +

  • The malloced virtual address of the memory region containing the chunk

  • The size in bytes of the data area

    • -

  • An allocation size in bytes used during heap compaction to merge -small blocks to defragment the heap

    • +

  • An allocation size in bytes used during heap compaction to merge small blocks to defragment the heap

  • A link to the next heap chunk in the list

    • -

  • A pointer to the start and end of the range of blocks that may -contain unexamined fields and need to be scanned later. Only used after -mark stack overflow.

    • +

  • A pointer to the start and end of the range of blocks that may contain unexamined fields and need to be scanned later. Only used after mark stack overflow.

-

Each chunk’s data area starts on a page boundary, and its size is a -multiple of the page size (4 KB). It contains a contiguous sequence of -heap blocks that can be as small as one or two 4 KB pages, but are -usually allocated in 1 MB chunks (or 512 KB on 32-bit architectures). - 

+

Each chunk’s data area starts on a page boundary, and its size is a multiple of the page size (4 KB). It contains a contiguous sequence of heap blocks that can be as small as one or two 4 KB pages, but are usually allocated in 1 MB chunks (or 512 KB on 32-bit architectures).  

Controlling the Major Heap Increment

-

The Gc module uses the major_heap_increment -value to control the major heap growth. This defines the number of words -to add to the major heap per expansion and is the only memory allocation -operation that the operating system observes from the OCaml runtime -after initial startup (since the minor is fixed in size).

-

Allocating an OCaml value on the major heap first checks the free -list of blocks for a suitable region to place it. If there isn’t enough -room on the free list, the runtime expands the major heap by allocating -a fresh heap chunk that will be large enough. That chunk is then added -to the free list, and the free list is checked again (and this time will -definitely succeed).

-

Older versions of OCaml required setting a fixed number of bytes for -the major heap increment. That was a value that was tricky to get right: -too small of a value could lead to lots of smaller heap chunks spread -across different regions of virtual memory that require more -housekeeping in the OCaml runtime to keep track of them; too large of a -value can waste memory for programs with small heaps.

-

You can use Gc.tune to set that value, but the values -are a little counter-intuitive, for backwards-compatibility reasons. -Values under 1000 are interpreted as percentages, and the default is -15%. Values 1000 and over are treated as a raw number of bytes. But most -of the time, you won’t set the value at all.

+

The Gc module uses the major_heap_increment value to control the major heap growth. This defines the number of words to add to the major heap per expansion and is the only memory allocation operation that the operating system observes from the OCaml runtime after initial startup (since the minor is fixed in size).

+

Allocating an OCaml value on the major heap first checks the free list of blocks for a suitable region to place it. If there isn’t enough room on the free list, the runtime expands the major heap by allocating a fresh heap chunk that will be large enough. That chunk is then added to the free list, and the free list is checked again (and this time will definitely succeed).

+

Older versions of OCaml required setting a fixed number of bytes for the major heap increment. That was a value that was tricky to get right: too small of a value could lead to lots of smaller heap chunks spread across different regions of virtual memory that require more housekeeping in the OCaml runtime to keep track of them; too large of a value can waste memory for programs with small heaps.

+

You can use Gc.tune to set that value, but the values are a little counter-intuitive, for backwards-compatibility reasons. Values under 1000 are interpreted as percentages, and the default is 15%. Values 1000 and over are treated as a raw number of bytes. But most of the time, you won’t set the value at all.

Memory Allocation Strategies

-

The major heap does its best to manage memory allocation as -efficiently as possible and relies on heap compaction to ensure that -memory stays contiguous and unfragmented. The default allocation policy -normally works fine for most applications, but it’s worth bearing in -mind that there are other options, too.   

-

The free list of blocks is always checked first when allocating a new -block in the major heap. The default free list search is called -best-fit allocation, with alternatives next-fit and -first-fit algorithms also available.    

+

The major heap does its best to manage memory allocation as efficiently as possible and relies on heap compaction to ensure that memory stays contiguous and unfragmented. The default allocation policy normally works fine for most applications, but it’s worth bearing in mind that there are other options, too.   

+

The free list of blocks is always checked first when allocating a new block in the major heap. The default free list search is called best-fit allocation, with alternatives next-fit and first-fit algorithms also available.    

Best-Fit Allocation

-

The best-fit allocator is a combination of two strategies. The first, -size-segregated free lists, is based on the observation that nearly all -major heap allocations in OCaml are small (consider list elements and -tuples which are only a couple of machine words). Best fit keeps -separate free lists for sizes up to and including 16 words which gives a -fast path for most allocations. Allocations for these sizes can be -serviced from their segregated free lists or, if they are empty, from -the next size with a space.

-

The second strategy, for larger allocations, is the use of a -specialized data structure known as a splay tree for the free -list. This is a type of search tree that adapts to recent access -patterns. For our use this means that the most commonly requested -allocation sizes are the quickest to access.

-

Small allocations, when there are no larger sizes available in the -segregated free lists, and large allocations greater than sixteen words -are serviced from the main free list. The free list is queried for the -smallest block that is at least as large as the allocation -requested.

-

Best-fit allocation is the default allocation mechanism. It -represents a good trade-off between the allocation cost (in terms of CPU -work) and heap fragmentation.

+

The best-fit allocator is a combination of two strategies. The first, size-segregated free lists, is based on the observation that nearly all major heap allocations in OCaml are small (consider list elements and tuples which are only a couple of machine words). Best fit keeps separate free lists for sizes up to and including 16 words which gives a fast path for most allocations. Allocations for these sizes can be serviced from their segregated free lists or, if they are empty, from the next size with a space.

+

The second strategy, for larger allocations, is the use of a specialized data structure known as a splay tree for the free list. This is a type of search tree that adapts to recent access patterns. For our use this means that the most commonly requested allocation sizes are the quickest to access.

+

Small allocations, when there are no larger sizes available in the segregated free lists, and large allocations greater than sixteen words are serviced from the main free list. The free list is queried for the smallest block that is at least as large as the allocation requested.

+

Best-fit allocation is the default allocation mechanism. It represents a good trade-off between the allocation cost (in terms of CPU work) and heap fragmentation.

Next-Fit Allocation

-

Next-fit allocation keeps a pointer to the block in the free list -that was most recently used to satisfy a request. When a new request -comes in, the allocator searches from the next block to the end of the -free list, and then from the beginning of the free list up to that -block.

-

Next-fit allocation is quite a cheap allocation mechanism, since the -same heap chunk can be reused across allocation requests until it runs -out. This in turn means that there is good memory locality to use CPU -caches better. The big downside of next-fit is that since most -allocations are small, large blocks at the start of the free list become -heavily fragmented.

+

Next-fit allocation keeps a pointer to the block in the free list that was most recently used to satisfy a request. When a new request comes in, the allocator searches from the next block to the end of the free list, and then from the beginning of the free list up to that block.

+

Next-fit allocation is quite a cheap allocation mechanism, since the same heap chunk can be reused across allocation requests until it runs out. This in turn means that there is good memory locality to use CPU caches better. The big downside of next-fit is that since most allocations are small, large blocks at the start of the free list become heavily fragmented.

First-Fit Allocation

-

If your program allocates values of many varied sizes, you may -sometimes find that your free list becomes fragmented. In this -situation, the GC is forced to perform an expensive compaction despite -there being free chunks, since none of the chunks alone are big enough -to satisfy the request.

-

First-fit allocation focuses on reducing memory fragmentation (and -hence the number of compactions), but at the expense of slower memory -allocation. Every allocation scans the free list from the beginning for -a suitable free chunk, instead of reusing the most recent heap chunk as -the next-fit allocator does.  

-

For some workloads that need more real-time behavior under load, the -reduction in the frequency of heap compaction will outweigh the extra -allocation cost.

+

If your program allocates values of many varied sizes, you may sometimes find that your free list becomes fragmented. In this situation, the GC is forced to perform an expensive compaction despite there being free chunks, since none of the chunks alone are big enough to satisfy the request.

+

First-fit allocation focuses on reducing memory fragmentation (and hence the number of compactions), but at the expense of slower memory allocation. Every allocation scans the free list from the beginning for a suitable free chunk, instead of reusing the most recent heap chunk as the next-fit allocator does.  

+

For some workloads that need more real-time behavior under load, the reduction in the frequency of heap compaction will outweigh the extra allocation cost.

Controlling the Heap Allocation Policy

-

You can set the heap allocation policy by calling -Gc.tune:

+

You can set the heap allocation policy by calling Gc.tune:

Gc.tune ~allocation_policy:First_fit ();;
 >- : unit = ()
-

The same behavior can be controlled via an environment variable by -setting OCAMLRUNPARAM to a=0 for next-fit, -a=1 for first-fit, or a=2 for best-fit.

+

The same behavior can be controlled via an environment variable by setting OCAMLRUNPARAM to a=0 for next-fit, a=1 for first-fit, or a=2 for best-fit.

Marking and Scanning the Heap

-

The marking process can take a long time to run over the complete -major heap and has to pause the main application while it’s active. It -therefore runs incrementally by marking the heap in slices. -Each value in the heap has a 2-bit color field in its header -that is used to store information about whether the value has been -marked so that the GC can resume easily between slices.  

+

The marking process can take a long time to run over the complete major heap and has to pause the main application while it’s active. It therefore runs incrementally by marking the heap in slices. Each value in the heap has a 2-bit color field in its header that is used to store information about whether the value has been marked so that the GC can resume easily between slices.  

  • Blue: On the free list and not currently in use
  • White (during marking): Not reached yet, but possibly reachable
  • White (during sweeping): Unreachable and can be freed
  • Black: Reachable, and its fields have been scanned
-

The color tags in the value headers store most of the state of the -marking process, allowing it to be paused and resumed later. On -allocation, all heap values are initially given the color white -indicating they are possibly reachable but haven’t been scanned yet. The -GC and application alternate between marking a slice of the major heap -and actually getting on with executing the program logic. The OCaml -runtime calculates a sensible value for the size of each major heap -slice based on the rate of allocation and available memory.

-

The marking process starts with a set of root values that -are always live (such as the application stack and globals). These root -values have their color set to black and are pushed on to a specialized -data structure known as the mark stack. Marking proceeds by -popping a value from the stack and examining its fields. Any fields -containing white-colored blocks are changed to black and pushed onto the -mark stack.

-

This process is repeated until the mark stack is empty and there are -no further values to mark. There’s one important edge case in this -process, though. The mark stack can only grow to a certain size, after -which the GC can no longer recurse into intermediate values since it has -nowhere to store them while it follows their fields. This is known as -mark stack overflow and a process called pruning -begins. Pruning empties the mark stack entirely, summarizing the -addresses of each block as start and end ranges in each heap chunk -header.

-

Later in the marking process when the mark stack is empty it is -replenished by redarkening the heap. This starts at the first -heap chunk (by address) that has blocks needing redarkening (i.e were -removed from the mark stack during a prune) and entries from the -redarkening range are added to the mark stack until it is a quarter -full. The emptying and replenishing cycle continues until there are no -heap chunks with ranges left to redarken.

+

The color tags in the value headers store most of the state of the marking process, allowing it to be paused and resumed later. On allocation, all heap values are initially given the color white indicating they are possibly reachable but haven’t been scanned yet. The GC and application alternate between marking a slice of the major heap and actually getting on with executing the program logic. The OCaml runtime calculates a sensible value for the size of each major heap slice based on the rate of allocation and available memory.

+

The marking process starts with a set of root values that are always live (such as the application stack and globals). These root values have their color set to black and are pushed on to a specialized data structure known as the mark stack. Marking proceeds by popping a value from the stack and examining its fields. Any fields containing white-colored blocks are changed to black and pushed onto the mark stack.

+

This process is repeated until the mark stack is empty and there are no further values to mark. There’s one important edge case in this process, though. The mark stack can only grow to a certain size, after which the GC can no longer recurse into intermediate values since it has nowhere to store them while it follows their fields. This is known as mark stack overflow and a process called pruning begins. Pruning empties the mark stack entirely, summarizing the addresses of each block as start and end ranges in each heap chunk header.

+

Later in the marking process when the mark stack is empty it is replenished by redarkening the heap. This starts at the first heap chunk (by address) that has blocks needing redarkening (i.e were removed from the mark stack during a prune) and entries from the redarkening range are added to the mark stack until it is a quarter full. The emptying and replenishing cycle continues until there are no heap chunks with ranges left to redarken.

Controlling Major Heap Collections

-

You can trigger a single slice of the major GC via the -major_slice call. This performs a minor collection first, -and then a single slice. The size of the slice is normally automatically -computed by the GC to an appropriate value and returns this value so -that you can modify it in future calls if necessary:

+

You can trigger a single slice of the major GC via the major_slice call. This performs a minor collection first, and then a single slice. The size of the slice is normally automatically computed by the GC to an appropriate value and returns this value so that you can modify it in future calls if necessary:

Gc.major_slice 0;;
 >- : int = 0
@@ -401,40 +152,18 @@ 
Controlling Major Heap Collections

 >- : unit = ()
-

The space_overhead setting controls how aggressive the -GC is about setting the slice size to a large size. This represents the -proportion of memory used for live data that will be “wasted” because -the GC doesn’t immediately collect unreachable blocks. Core defaults -this to 100 to reflect a typical system that isn’t overly -memory-constrained. Set this even higher if you have lots of memory, or -lower to cause the GC to work harder and collect blocks faster at the -expense of using more CPU time.

+

The space_overhead setting controls how aggressive the GC is about setting the slice size to a large size. This represents the proportion of memory used for live data that will be “wasted” because the GC doesn’t immediately collect unreachable blocks. Core defaults this to 100 to reflect a typical system that isn’t overly memory-constrained. Set this even higher if you have lots of memory, or lower to cause the GC to work harder and collect blocks faster at the expense of using more CPU time.

Heap Compaction

-

After a certain number of major GC cycles have completed, the heap -may begin to be fragmented due to values being deallocated out of order -from how they were allocated. This makes it harder for the GC to find a -contiguous block of memory for fresh allocations, which in turn would -require the heap to be grown unnecessarily.    

-

The heap compaction cycle avoids this by relocating all the values in -the major heap into a fresh heap that places them all contiguously in -memory again. A naive implementation of the algorithm would require -extra memory to store the new heap, but OCaml performs the compaction in -place within the existing heap.

-
-

Controlling Frequency of Compactions

-

The max_overhead setting in the Gc module -defines the connection between free memory and allocated memory after -which compaction is activated.

-

A value of 0 triggers a compaction after every major -garbage collection cycle, whereas the maximum value of -1000000 disables heap compaction completely. The default -settings should be fine unless you have unusual allocation patterns that -are causing a higher-than-usual rate of compactions:

-
+

After a certain number of major GC cycles have completed, the heap may begin to be fragmented due to values being deallocated out of order from how they were allocated. This makes it harder for the GC to find a contiguous block of memory for fresh allocations, which in turn would require the heap to be grown unnecessarily.    

+

The heap compaction cycle avoids this by relocating all the values in the major heap into a fresh heap that places them all contiguously in memory again. A naive implementation of the algorithm would require extra memory to store the new heap, but OCaml performs the compaction in place within the existing heap.

+
+

Controlling Frequency of Compactions

+

The max_overhead setting in the Gc module defines the connection between free memory and allocated memory after which compaction is activated.

+

A value of 0 triggers a compaction after every major garbage collection cycle, whereas the maximum value of 1000000 disables heap compaction completely. The default settings should be fine unless you have unusual allocation patterns that are causing a higher-than-usual rate of compactions:

+
Gc.tune ~max_overhead:0 ();;
 >- : unit = ()
@@ -443,34 +172,13 @@ 
Controlling Frequency of Compactions

Intergenerational Pointers

-

One complexity of generational collection arises from the fact that -minor heap sweeps are much more frequent than major heap collections. In -order to know which blocks in the minor heap are live, the collector -must track which minor-heap blocks are directly pointed to by major-heap -blocks. Without this information, each minor collection would also -require scanning the much larger major heap.    

-

OCaml maintains a set of such intergenerational pointers to -avoid this dependency between a major and minor heap collection. The -compiler introduces a write barrier to update this so-called -remembered set whenever a major-heap block is modified to point -at a minor-heap block.   

+

One complexity of generational collection arises from the fact that minor heap sweeps are much more frequent than major heap collections. In order to know which blocks in the minor heap are live, the collector must track which minor-heap blocks are directly pointed to by major-heap blocks. Without this information, each minor collection would also require scanning the much larger major heap.    

+

OCaml maintains a set of such intergenerational pointers to avoid this dependency between a major and minor heap collection. The compiler introduces a write barrier to update this so-called remembered set whenever a major-heap block is modified to point at a minor-heap block.   

The Mutable Write Barrier

-

The write barrier can have profound implications for the structure of -your code. It’s one of the reasons using immutable data structures and -allocating a fresh copy with changes can sometimes be faster than -mutating a record in place.

-

The OCaml compiler keeps track of any mutable types and adds a call -to the runtime caml_modify function before making the -change. This checks the location of the target write and the value it’s -being changed to, and ensures that the remembered set is consistent. -Although the write barrier is reasonably efficient, it can sometimes be -slower than simply allocating a fresh value on the fast minor heap and -doing some extra minor collections.

-

Let’s see this for ourselves with a simple test program. You’ll need -to install the Core benchmarking suite via -opam install core_bench before you compile this code:

+

The write barrier can have profound implications for the structure of your code. It’s one of the reasons using immutable data structures and allocating a fresh copy with changes can sometimes be faster than mutating a record in place.

+

The OCaml compiler keeps track of any mutable types and adds a call to the runtime caml_modify function before making the change. This checks the location of the target write and the value it’s being changed to, and ensures that the remembered set is consistent. Although the write barrier is reasonably efficient, it can sometimes be slower than simply allocating a fresh value on the fast minor heap and doing some extra minor collections.

+

Let’s see this for ourselves with a simple test program. You’ll need to install the Core benchmarking suite via opam install core_bench before you compile this code:

open Core
 open Core_bench
@@ -514,10 +222,7 @@ 
The Mutable Write Barrier

   in
   Bench.make_command tests |> Command_unix.run
-

This program defines a type t1 that is mutable and -t2 that is immutable. The benchmark loop iterates over both -fields and increments a counter. Compile and execute this with some -extra options to show the amount of garbage collection occurring:

+

This program defines a type t1 that is mutable and t2 that is immutable. The benchmark loop iterates over both fields and increments a counter. Compile and execute this with some extra options to show the amount of garbage collection occurring:

dune exec -- ./barrier_bench.exe -ascii alloc -quota 1
 >Estimated testing time 2s (2 benchmarks x 1s). Change using '-quota'.
@@ -528,18 +233,8 @@ 
The Mutable Write Barrier

 >  immutable     3.95ms    5.00Mw     95.64w     95.64w       77.98%
-

There is a space/time trade-off here. The mutable version takes -longer to complete than the immutable one but allocates many fewer -minor-heap words than the immutable version. Minor allocation in OCaml -is very fast, and so it is often better to use immutable data structures -in preference to the more conventional mutable versions. On the other -hand, if you only rarely mutate a value, it can be faster to take the -write-barrier hit and not allocate at all.

-

The only way to know for sure is to benchmark your program under -real-world scenarios using Core_bench and experiment with -the trade-offs. The command-line benchmark binaries have a number of -useful options that affect garbage collection behavior and the output -format:

+

There is a space/time trade-off here. The mutable version takes longer to complete than the immutable one but allocates many fewer minor-heap words than the immutable version. Minor allocation in OCaml is very fast, and so it is often better to use immutable data structures in preference to the more conventional mutable versions. On the other hand, if you only rarely mutate a value, it can be faster to take the write-barrier hit and not allocate at all.

+

The only way to know for sure is to benchmark your program under real-world scenarios using Core_bench and experiment with the trade-offs. The command-line benchmark binaries have a number of useful options that affect garbage collection behavior and the output format:

dune exec -- ./barrier_bench.exe -help
 >Benchmark for mutable, immutable
@@ -562,43 +257,14 @@ 
The Mutable Write Barrier

Attaching Finalizer Functions to Values

-

OCaml’s automatic memory management guarantees that a value will -eventually be freed when it’s no longer in use, either via the GC -sweeping it or the program terminating. It’s sometimes useful to run -extra code just before a value is freed by the GC, for example, to check -that a file descriptor has been closed, or that a log message is -recorded.    

-
-

What Values Can Be Finalized?

-

Various values cannot have finalizers attached since they aren’t -heap-allocated. Some examples of values that are not heap-allocated are -integers, constant constructors, Booleans, the empty array, the empty -list, and the unit value. The exact list of what is heap-allocated or -not is implementation-dependent, which is why Core provides the -Heap_block module to explicitly check before attaching the -finalizer.

-

Some constant values can be heap-allocated but never deallocated -during the lifetime of the program, for example, a list of integer -constants. Heap_block explicitly checks to see if the value -is in the major or minor heap, and rejects most constant values. -Compiler optimizations may also duplicate some immutable values such as -floating-point values in arrays. These may be finalized while another -duplicate copy is being used by the program.

-
-

Core provides a Heap_block module that dynamically -checks if a given value is suitable for finalizing. Core keeps the -functions for registering finalizers in the Core.Gc.Expert -module. Finalizers can run at any time in any thread, so they can be -pretty hard to reason about in multi-threaded contexts.   Async, which we discussed in -Chapter 16, Concurrent Programming with Async, shadows the -Gc module with its own module that contains a function, -Gc.add_finalizer, which is concurrency-safe. In particular, -finalizers are scheduled in their own Async job, and care is taken by -Async to capture exceptions and raise them to the appropriate monitor -for error-handling.  

-

Let’s explore this with a small example that finalizes values of -different types, all of which are heap-allocated.

+

OCaml’s automatic memory management guarantees that a value will eventually be freed when it’s no longer in use, either via the GC sweeping it or the program terminating. It’s sometimes useful to run extra code just before a value is freed by the GC, for example, to check that a file descriptor has been closed, or that a log message is recorded.    

+
+

What Values Can Be Finalized?

+

Various values cannot have finalizers attached since they aren’t heap-allocated. Some examples of values that are not heap-allocated are integers, constant constructors, Booleans, the empty array, the empty list, and the unit value. The exact list of what is heap-allocated or not is implementation-dependent, which is why Core provides the Heap_block module to explicitly check before attaching the finalizer.

+

Some constant values can be heap-allocated but never deallocated during the lifetime of the program, for example, a list of integer constants. Heap_block explicitly checks to see if the value is in the major or minor heap, and rejects most constant values. Compiler optimizations may also duplicate some immutable values such as floating-point values in arrays. These may be finalized while another duplicate copy is being used by the program.

+
+

Core provides a Heap_block module that dynamically checks if a given value is suitable for finalizing. Core keeps the functions for registering finalizers in the Core.Gc.Expert module. Finalizers can run at any time in any thread, so they can be pretty hard to reason about in multi-threaded contexts.   Async, which we discussed in Chapter 16, Concurrent Programming with Async, shadows the Gc module with its own module that contains a function, Gc.add_finalizer, which is concurrency-safe. In particular, finalizers are scheduled in their own Async job, and care is taken by Async to capture exceptions and raise them to the appropriate monitor for error-handling.  

+

Let’s explore this with a small example that finalizes values of different types, all of which are heap-allocated.

open Core
 open Async
@@ -633,24 +299,9 @@ 
What Values Can Be Finalized?

 >   allocated variant: OK
-

The GC calls the finalization functions in the order of the -deallocation. If several values become unreachable during the same GC -cycle, the finalization functions will be called in the reverse order of -the corresponding calls to add_finalizer. Each call to -add_finalizer adds to the set of functions, which are run -when the value becomes unreachable. You can have many finalizers all -pointing to the same heap block if you wish.

-

After a garbage collection determines that a heap block -b is unreachable, it removes from the set of finalizers all -the functions associated with b, and serially applies each -of those functions to b. Thus, every finalizer function -attached to b will run at most once. However, program -termination will not cause all the finalizers to be run before the -runtime exits.

-

The finalizer can use all features of OCaml, including assignments -that make the value reachable again and thus prevent it from being -garbage-collected. It can also loop forever, which will cause other -finalizers to be interleaved with it.

+

The GC calls the finalization functions in the order of the deallocation. If several values become unreachable during the same GC cycle, the finalization functions will be called in the reverse order of the corresponding calls to add_finalizer. Each call to add_finalizer adds to the set of functions, which are run when the value becomes unreachable. You can have many finalizers all pointing to the same heap block if you wish.

+

After a garbage collection determines that a heap block b is unreachable, it removes from the set of finalizers all the functions associated with b, and serially applies each of those functions to b. Thus, every finalizer function attached to b will run at most once. However, program termination will not cause all the finalizers to be run before the runtime exits.

+

The finalizer can use all features of OCaml, including assignments that make the value reachable again and thus prevent it from being garbage-collected. It can also loop forever, which will cause other finalizers to be interleaved with it.

-

Next: Chapter 25The Compiler Frontend: Parsing and Type Checking

 \ No newline at end of file +

Next: Chapter 25The Compiler Frontend: Parsing and Type Checking

 \ No newline at end of file diff --git a/guided-tour.html b/guided-tour.html index eb3232c28..ccbbe8695 100644 --- a/guided-tour.html +++ b/guided-tour.html @@ -1,53 +1,19 @@ -A Guided Tour - Real World OCaml

Real World OCaml

2nd Edition (published in Q4 2022)
Back
Table of Contents
+A Guided Tour - Real World OCaml

Real World OCaml

2nd Edition (Oct 2022)
Back
Table of Contents

A Guided Tour

-

This chapter gives an overview of OCaml by walking through a series -of small examples that cover most of the major features of the language. -This should provide a sense of what OCaml can do, without getting too -deep into any one topic.

-

Throughout the book we’re going to use Base, a more -full-featured and capable replacement for OCaml’s standard library. -We’ll also use utop, a shell that lets you type in -expressions and evaluate them interactively. utop is an -easier-to-use version of OCaml’s standard toplevel (which you can start -by typing ocaml at the command line). These instructions will -assume you’re using utop, but the ordinary toplevel should -mostly work fine.

-

Before going any further, make sure you’ve followed the steps in the installation -page.

-
-

Base and Core

-

Base comes along with another, yet more extensive -standard library replacement, called Core. We’re going to -mostly stick to Base, but it’s worth understanding the -differences between these libraries.    

+

This chapter gives an overview of OCaml by walking through a series of small examples that cover most of the major features of the language. This should provide a sense of what OCaml can do, without getting too deep into any one topic.

+

Throughout the book we’re going to use Base, a more full-featured and capable replacement for OCaml’s standard library. We’ll also use utop, a shell that lets you type in expressions and evaluate them interactively. utop is an easier-to-use version of OCaml’s standard toplevel (which you can start by typing ocaml at the command line). These instructions will assume you’re using utop, but the ordinary toplevel should mostly work fine.

+

Before going any further, make sure you’ve followed the steps in the installation page.

+
+

Base and Core

+

Base comes along with another, yet more extensive standard library replacement, called Core. We’re going to mostly stick to Base, but it’s worth understanding the differences between these libraries.    

    -

  • Base is designed to be lightweight, -portable, and stable, while providing all of the fundamentals you need -from a standard library. It comes with a minimum of external -dependencies, so Base just takes seconds to build and -install.

    • -

  • Core extends Base in a number -of ways: it adds new data structures, like heaps, hash-sets, and -functional queues; it provides types to represent times and time-zones; -well-integrated support for efficient binary serializers; and much more. -At the same time, it has many more dependencies, and so takes longer to -build, and will add more to the size of your executables.

    • +

  • Base is designed to be lightweight, portable, and stable, while providing all of the fundamentals you need from a standard library. It comes with a minimum of external dependencies, so Base just takes seconds to build and install.

    • +

  • Core extends Base in a number of ways: it adds new data structures, like heaps, hash-sets, and functional queues; it provides types to represent times and time-zones; well-integrated support for efficient binary serializers; and much more. At the same time, it has many more dependencies, and so takes longer to build, and will add more to the size of your executables.

-

As of the version of Base and Core used in -this book (version v0.14), Core is less -portable than Base, running only on UNIX-like systems. For -that reason, there is another package, Core_kernel, which -is the portable subset of Core. That said, in the latest -stable release, v0.15 (which was released too late to be -adopted for this edition of the book) Core is portable, and -Core_kernel has been deprecated. Given that, we don’t use -Core_kernel in this text.

-
-

Before getting started, make sure you have a working OCaml -installation so you can try out the examples as you read through the -chapter.

+

As of the version of Base and Core used in this book (version v0.14), Core is less portable than Base, running only on UNIX-like systems. For that reason, there is another package, Core_kernel, which is the portable subset of Core. That said, in the latest stable release, v0.15 (which was released too late to be adopted for this edition of the book) Core is portable, and Core_kernel has been deprecated. Given that, we don’t use Core_kernel in this text.

+
+

Before getting started, make sure you have a working OCaml installation so you can try out the examples as you read through the chapter.

OCaml as a Calculator

Our first step is to open Base:

@@ -55,10 +21,7 @@

OCaml as a Calculator

open Base;;
-

By opening Base, we make the definitions it contains -available without having to reference Base explicitly. This -is required for many of the examples in the tour and in the remainder of -the book.

+

By opening Base, we make the definitions it contains available without having to reference Base explicitly. This is required for many of the examples in the tour and in the remainder of the book.

Now let’s try a few simple numerical calculations:

3 + 4;;
@@ -73,35 +36,14 @@ 
OCaml as a Calculator

 >- : bool = true
-

By and large, this is pretty similar to what you’d find in any -programming language, but a few things jump right out at you:

+

By and large, this is pretty similar to what you’d find in any programming language, but a few things jump right out at you:

    -

  • We needed to type ;; in order to tell the toplevel -that it should evaluate an expression. This is a peculiarity of the -toplevel that is not required in standalone programs (though it is -sometimes helpful to include ;; to improve OCaml’s error -reporting, by making it more explicit where a given top-level -declaration was intended to end).

    • -

  • After evaluating an expression, the toplevel first prints the -type of the result, and then prints the result itself.

    • -

  • OCaml allows you to place underscores in the middle of numeric -literals to improve readability. Note that underscores can be placed -anywhere within a number, not just every three digits.

    • -

  • OCaml carefully distinguishes between float, the -type for floating-point numbers, and int, the type for -integers. The types have different literals (6. instead of -6) and different infix operators (+. instead -of +), and OCaml doesn’t automatically cast between these -types. This can be a bit of a nuisance, but it has its benefits, since -it prevents some kinds of bugs that arise in other languages due to -unexpected differences between the behavior of int and -float. For example, in many languages, 1 / 3 -is zero, but 1.0 /. 3.0 is a third. OCaml requires you to -be explicit about which operation you’re using.

    • +

  • We needed to type ;; in order to tell the toplevel that it should evaluate an expression. This is a peculiarity of the toplevel that is not required in standalone programs (though it is sometimes helpful to include ;; to improve OCaml’s error reporting, by making it more explicit where a given top-level declaration was intended to end).

    • +

  • After evaluating an expression, the toplevel first prints the type of the result, and then prints the result itself.

    • +

  • OCaml allows you to place underscores in the middle of numeric literals to improve readability. Note that underscores can be placed anywhere within a number, not just every three digits.

    • +

  • OCaml carefully distinguishes between float, the type for floating-point numbers, and int, the type for integers. The types have different literals (6. instead of 6) and different infix operators (+. instead of +), and OCaml doesn’t automatically cast between these types. This can be a bit of a nuisance, but it has its benefits, since it prevents some kinds of bugs that arise in other languages due to unexpected differences between the behavior of int and float. For example, in many languages, 1 / 3 is zero, but 1.0 /. 3.0 is a third. OCaml requires you to be explicit about which operation you’re using.

-

We can also create a variable to name the value of a given -expression, using the let keyword. This is known as a -let binding:  

+

We can also create a variable to name the value of a given expression, using the let keyword. This is known as a let binding:  

let x = 3 + 4;;
 >val x : int = 7
@@ -109,13 +51,8 @@ 
OCaml as a Calculator

 >val y : int = 14
-

After a new variable is created, the toplevel tells us the name of -the variable (x or y), in addition to its type -(int) and value (7 or 14).

-

Note that there are some constraints on what identifiers can be used -for variable names. Punctuation is excluded, except for _ -and ', and variables must start with a lowercase letter or -an underscore. Thus, these are legal:

+

After a new variable is created, the toplevel tells us the name of the variable (x or y), in addition to its type (int) and value (7 or 14).

+

Note that there are some constraints on what identifiers can be used for variable names. Punctuation is excluded, except for _ and ', and variables must start with a lowercase letter or an underscore. Thus, these are legal:

let x7 = 3 + 4;;
 >val x7 : int = 7
@@ -138,14 +75,11 @@ 
OCaml as a Calculator

 >Error: Syntax error
-

This highlights that variables can’t be capitalized, can’t begin with -numbers, and can’t contain dashes.

+

This highlights that variables can’t be capitalized, can’t begin with numbers, and can’t contain dashes.

Functions and Type Inference

-

The let syntax can also be used to define a -function:  

+

The let syntax can also be used to define a function:  

let square x = x * x;;
 >val square : int -> int = <fun>
@@ -155,19 +89,8 @@ 
Functions and Type Inference

 >- : int = 16
-

Functions in OCaml are values like any other, which is why we use the -let keyword to bind a function to a variable name, just as -we use let to bind a simple value like an integer to a -variable name. When using let to define a function, the -first identifier after the let is the function name, and -each subsequent identifier is a different argument to the function. -Thus, square is a function with a single argument.

-

Now that we’re creating more interesting values like functions, the -types have gotten more interesting too. int -> int is a -function type, in this case indicating a function that takes an -int and returns an int. We can also write -functions that take multiple arguments. (Reminder: Don’t forget -open Base, or these examples won’t work!)   

+

Functions in OCaml are values like any other, which is why we use the let keyword to bind a function to a variable name, just as we use let to bind a simple value like an integer to a variable name. When using let to define a function, the first identifier after the let is the function name, and each subsequent identifier is a different argument to the function. Thus, square is a function with a single argument.

+

Now that we’re creating more interesting values like functions, the types have gotten more interesting too. int -> int is a function type, in this case indicating a function that takes an int and returns an int. We can also write functions that take multiple arguments. (Reminder: Don’t forget open Base, or these examples won’t work!)   

let ratio x y =
   Float.of_int x /. Float.of_int y;;
@@ -176,26 +99,9 @@ 
Functions and Type Inference

 >- : float = 0.571428571428571397
-

Note that in OCaml, function arguments are separated by spaces -instead of by parentheses and commas, which is more like the UNIX shell -than it is like traditional programming languages such as Python or -Java.

-

The preceding example also happens to be our first use of modules. -Here, Float.of_int refers to the of_int -function contained in the Float module. This is different -from what you might expect from an object-oriented language, where -dot-notation is typically used for accessing a method of an object. Note -that module names always start with a capital letter.

-

Modules can also be opened to make their contents available without -explicitly qualifying by the module name. We did that once already, when -we opened Base earlier. We can use that to make this code a -little easier to read, both avoiding the repetition of -Float above, and avoiding use of the slightly awkward -/. operator. In the following example, we open the -Float.O module, which has a bunch of useful operators and -functions that are designed to be used in this kind of context. Note -that this causes the standard int-only arithmetic operators to be -shadowed locally.  

+

Note that in OCaml, function arguments are separated by spaces instead of by parentheses and commas, which is more like the UNIX shell than it is like traditional programming languages such as Python or Java.

+

The preceding example also happens to be our first use of modules. Here, Float.of_int refers to the of_int function contained in the Float module. This is different from what you might expect from an object-oriented language, where dot-notation is typically used for accessing a method of an object. Note that module names always start with a capital letter.

+

Modules can also be opened to make their contents available without explicitly qualifying by the module name. We did that once already, when we opened Base earlier. We can use that to make this code a little easier to read, both avoiding the repetition of Float above, and avoiding use of the slightly awkward /. operator. In the following example, we open the Float.O module, which has a bunch of useful operators and functions that are designed to be used in this kind of context. Note that this causes the standard int-only arithmetic operators to be shadowed locally.  

let ratio x y =
   let open Float.O in
@@ -203,27 +109,15 @@ 
Functions and Type Inference

 >val ratio : int -> int -> float = <fun>
-

We used a slightly different syntax for opening the module, since we -were only opening it in the local scope inside the definition of -ratio. There’s also a more concise syntax for local opens, -as you can see here.

+

We used a slightly different syntax for opening the module, since we were only opening it in the local scope inside the definition of ratio. There’s also a more concise syntax for local opens, as you can see here.

let ratio x y =
   Float.O.(of_int x / of_int y);;
 >val ratio : int -> int -> float = <fun>
-

The notation for the type-signature of a multiargument function may -be a little surprising at first, but we’ll explain where it comes from -when we get to function currying in Chapter 2, Multi Argument Functions. For the moment, think of -the arrows as separating different arguments of the function, with the -type after the final arrow being the return value. Thus, -int -> int -> float describes a function that takes -two int arguments and returns a float.

-

We can also write functions that take other functions as arguments. -Here’s an example of a function that takes three arguments: a test -function and two integer arguments. The function returns the sum of the -integers that pass the test:

+

The notation for the type-signature of a multiargument function may be a little surprising at first, but we’ll explain where it comes from when we get to function currying in Chapter 2, Multi Argument Functions. For the moment, think of the arrows as separating different arguments of the function, with the type after the final arrow being the return value. Thus, int -> int -> float describes a function that takes two int arguments and returns a float.

+

We can also write functions that take other functions as arguments. Here’s an example of a function that takes three arguments: a test function and two integer arguments. The function returns the sum of the integers that pass the test:

let sum_if_true test first second =
   (if test first then first else 0)
@@ -231,10 +125,7 @@ 
Functions and Type Inference

 >val sum_if_true : (int -> bool) -> int -> int -> int = <fun>
-

If we look at the inferred type signature in detail, we see that the -first argument is a function that takes an integer and returns a -boolean, and that the remaining two arguments are integers. Here’s an -example of this function in action:

+

If we look at the inferred type signature in detail, we see that the first argument is a function that takes an integer and returns a boolean, and that the remaining two arguments are integers. Here’s an example of this function in action:

let even x =
   x % 2 = 0;;
@@ -245,53 +136,24 @@ 
Functions and Type Inference

 >- : int = 6
-

Note that in the definition of even, we used -= in two different ways: once as part of the -let binding that separates the thing being defined from its -definition; and once as an equality test, when comparing -x % 2 to 0. These are very different -operations despite the fact that they share some syntax.

+

Note that in the definition of even, we used = in two different ways: once as part of the let binding that separates the thing being defined from its definition; and once as an equality test, when comparing x % 2 to 0. These are very different operations despite the fact that they share some syntax.

Type Inference

-

As the types we encounter get more complicated, you might ask -yourself how OCaml is able to figure them out, given that we didn’t -write down any explicit type information. 

-

OCaml determines the type of an expression using a technique called -type inference, by which the type of an expression is inferred -from the available type information about the components of that -expression.

-

As an example, let’s walk through the process of inferring the type -of sum_if_true:

+

As the types we encounter get more complicated, you might ask yourself how OCaml is able to figure them out, given that we didn’t write down any explicit type information. 

+

OCaml determines the type of an expression using a technique called type inference, by which the type of an expression is inferred from the available type information about the components of that expression.

+

As an example, let’s walk through the process of inferring the type of sum_if_true:

    -

  1. OCaml requires that both branches of an if -expression have the same type, so the expression

    +

  2. OCaml requires that both branches of an if expression have the same type, so the expression

    if test first then first else 0

    -

    requires that first must be the same type as -0, and so first must be of type -int. Similarly, from

    +

    requires that first must be the same type as 0, and so first must be of type int. Similarly, from

    if test second then second else 0

    -

    we can infer that second has type -int.

    3. -

  3. test is passed first as an argument. -Since first has type int, the input type of -test must be int.

    4. -

  4. test first is used as the condition in an -if expression, so the return type of test must -be bool.

    5. -

  5. The fact that + returns int implies -that the return value of sum_if_true must be int.

    6. +

    we can infer that second has type int.

    +

  6. test is passed first as an argument. Since first has type int, the input type of test must be int.

    7. +

  7. test first is used as the condition in an if expression, so the return type of test must be bool.

    8. +

  8. The fact that + returns int implies that the return value of sum_if_true must be int.

-

Together, that nails down the types of all the variables, which -determines the overall type of sum_if_true.

-

Over time, you’ll build a rough intuition for how the OCaml inference -engine works, which makes it easier to reason through your programs. You -can also make it easier to understand the types of a given expression by -adding explicit type annotations. These annotations don’t change the -behavior of an OCaml program, but they can serve as useful -documentation, as well as catch unintended type changes. They can also -be helpful in figuring out why a given piece of code fails to -compile.

+

Together, that nails down the types of all the variables, which determines the overall type of sum_if_true.

+

Over time, you’ll build a rough intuition for how the OCaml inference engine works, which makes it easier to reason through your programs. You can also make it easier to understand the types of a given expression by adding explicit type annotations. These annotations don’t change the behavior of an OCaml program, but they can serve as useful documentation, as well as catch unintended type changes. They can also be helpful in figuring out why a given piece of code fails to compile.

Here’s an annotated version of sum_if_true:

let sum_if_true (test : int -> bool) (x : int) (y : int) : int =
@@ -300,45 +162,20 @@ 
Type Inference

 >val sum_if_true : (int -> bool) -> int -> int -> int = <fun>
-

In the above, we’ve marked every argument to the function with its -type, with the final annotation indicating the type of the return value. -Such type annotations can be placed on any expression in an OCaml -program.

+

In the above, we’ve marked every argument to the function with its type, with the final annotation indicating the type of the return value. Such type annotations can be placed on any expression in an OCaml program.

Inferring Generic Types

-

Sometimes, there isn’t enough information to fully determine the -concrete type of a given value. Consider this function.. 

+

Sometimes, there isn’t enough information to fully determine the concrete type of a given value. Consider this function.. 

let first_if_true test x y =
   if test x then x else y;;
 >val first_if_true : ('a -> bool) -> 'a -> 'a -> 'a = <fun>
-

first_if_true takes as its arguments a function -test, and two values, x and y, -where x is to be returned if test x evaluates -to true, and y otherwise. So what’s the type -of the x argument to first_if_true? There are -no obvious clues such as arithmetic operators or literals to narrow it -down. That makes it seem like first_if_true would work on -values of any type.

-

Indeed, if we look at the type returned by the toplevel, we see that -rather than choose a single concrete type, OCaml has introduced a -type variable 'a to express that the type is -generic. (You can tell it’s a type variable by the leading single quote -mark.) In particular, the type of the test argument is -('a -> bool), which means that test is a -one-argument function whose return value is bool and whose -argument could be of any type 'a. But, whatever type -'a is, it has to be the same as the type of the other two -arguments, x and y, and of the return value of -first_if_true. This kind of genericity is called -parametric polymorphism because it works by parameterizing the -type in question with a type variable. It is very similar to generics in -C# and Java.   

-

Because the type of first_if_true is generic, we can -write this:

+

first_if_true takes as its arguments a function test, and two values, x and y, where x is to be returned if test x evaluates to true, and y otherwise. So what’s the type of the x argument to first_if_true? There are no obvious clues such as arithmetic operators or literals to narrow it down. That makes it seem like first_if_true would work on values of any type.

+

Indeed, if we look at the type returned by the toplevel, we see that rather than choose a single concrete type, OCaml has introduced a type variable 'a to express that the type is generic. (You can tell it’s a type variable by the leading single quote mark.) In particular, the type of the test argument is ('a -> bool), which means that test is a one-argument function whose return value is bool and whose argument could be of any type 'a. But, whatever type 'a is, it has to be the same as the type of the other two arguments, x and y, and of the return value of first_if_true. This kind of genericity is called parametric polymorphism because it works by parameterizing the type in question with a type variable. It is very similar to generics in C# and Java.   

+

Because the type of first_if_true is generic, we can write this:

let long_string s = String.length s > 6;;
 >val long_string : string -> bool = <fun>
@@ -354,12 +191,7 @@ 
Inferring Generic Types

 >- : int = 4
-

Both long_string and big_number are -functions, and each is passed to first_if_true with two -other arguments of the appropriate type (strings in the first example, -and integers in the second). But we can’t mix and match two different -concrete types for 'a in the same use of -first_if_true:

+

Both long_string and big_number are functions, and each is passed to first_if_true with two other arguments of the appropriate type (strings in the first example, and integers in the second). But we can’t mix and match two different concrete types for 'a in the same use of first_if_true:

first_if_true big_number "short" "loooooong";;
 >Line 1, characters 26-33:
@@ -367,22 +199,11 @@ 
Inferring Generic Types

 >         int
-

In this example, big_number requires that -'a be instantiated as int, whereas -"short" and "loooooong" require that -'a be instantiated as string, and they can’t -both be right at the same time.

-
-

Type Errors Versus Exceptions

-

There’s a big difference in OCaml between errors that are caught at -compile time and those that are caught at runtime. It’s better to catch -errors as early as possible in the development process, and compilation -time is best of all.    

-

Working in the toplevel somewhat obscures the difference between -runtime and compile-time errors, but that difference is still there. -Generally, type errors like this one:

+

In this example, big_number requires that 'a be instantiated as int, whereas "short" and "loooooong" require that 'a be instantiated as string, and they can’t both be right at the same time.

+
+

Type Errors Versus Exceptions

+

There’s a big difference in OCaml between errors that are caught at compile time and those that are caught at runtime. It’s better to catch errors as early as possible in the development process, and compilation time is best of all.    

+

Working in the toplevel somewhat obscures the difference between runtime and compile-time errors, but that difference is still there. Generally, type errors like this one:

let add_potato x =
   x + "potato";;
@@ -391,10 +212,7 @@ 
Type Errors Versus Exceptions

 >         int
-

are compile-time errors (because + requires that both -its arguments be of type int), whereas errors that can’t be -caught by the type system, like division by zero, lead to runtime -exceptions:

+

are compile-time errors (because + requires that both its arguments be of type int), whereas errors that can’t be caught by the type system, like division by zero, lead to runtime exceptions:

let is_a_multiple x y =
   x % y = 0;;
@@ -406,26 +224,15 @@ 
Type Errors Versus Exceptions

 >Invalid_argument "8 % 0 in core_int.ml: modulus should be positive".
-

The distinction here is that type errors will stop you whether or not -the offending code is ever actually executed. Merely defining -add_potato is an error, whereas is_a_multiple -only fails when it’s called, and then, only when it’s called with an -input that triggers the exception.

-
+

The distinction here is that type errors will stop you whether or not the offending code is ever actually executed. Merely defining add_potato is an error, whereas is_a_multiple only fails when it’s called, and then, only when it’s called with an input that triggers the exception.

+

Tuples, Lists, Options, and Pattern Matching

Tuples

-

So far we’ve encountered a handful of basic types like -int, float, and string, as well -as function types like string -> int. But we haven’t yet -talked about any data structures. We’ll start by looking at a -particularly simple data structure, the tuple. A tuple is an ordered -collection of values that can each be of a different type. You can -create a tuple by joining values together with a comma.   

+

So far we’ve encountered a handful of basic types like int, float, and string, as well as function types like string -> int. But we haven’t yet talked about any data structures. We’ll start by looking at a particularly simple data structure, the tuple. A tuple is an ordered collection of values that can each be of a different type. You can create a tuple by joining values together with a comma.   

let a_tuple = (3,"three");;
 >val a_tuple : int * string = (3, "three")
@@ -433,76 +240,46 @@ 
Tuples

 >val another_tuple : int * string * float = (3, "four", 5.)
-

For the mathematically inclined, * is used in the type -t * s because that type corresponds to the set of all pairs -containing one value of type t and one of type -s. In other words, it’s the Cartesian product of -the two types, which is why we use *, the symbol for -product.

-

You can extract the components of a tuple using OCaml’s -pattern-matching syntax, as shown below:

+

For the mathematically inclined, * is used in the type t * s because that type corresponds to the set of all pairs containing one value of type t and one of type s. In other words, it’s the Cartesian product of the two types, which is why we use *, the symbol for product.

+

You can extract the components of a tuple using OCaml’s pattern-matching syntax, as shown below:

let (x,y) = a_tuple;;
 >val x : int = 3
 >val y : string = "three"
-

Here, the (x,y) on the left-hand side of the -let binding is the pattern. This pattern lets us mint the -new variables x and y, each bound to different -components of the value being matched. These can now be used in -subsequent expressions:

+

Here, the (x,y) on the left-hand side of the let binding is the pattern. This pattern lets us mint the new variables x and y, each bound to different components of the value being matched. These can now be used in subsequent expressions:

x + String.length y;;
 >- : int = 8
-

Note that the same syntax is used both for constructing and for -pattern matching on tuples.

-

Pattern matching can also show up in function arguments. Here’s a -function for computing the distance between two points on the plane, -where each point is represented as a pair of floats. The -pattern-matching syntax lets us get at the values we need with a minimum -of fuss:

+

Note that the same syntax is used both for constructing and for pattern matching on tuples.

+

Pattern matching can also show up in function arguments. Here’s a function for computing the distance between two points on the plane, where each point is represented as a pair of floats. The pattern-matching syntax lets us get at the values we need with a minimum of fuss:

let distance (x1,y1) (x2,y2) =
   Float.sqrt ((x1 -. x2) **. 2. +. (y1 -. y2) **. 2.);;
 >val distance : float * float -> float * float -> float = <fun>
-

The **. operator used above is for raising a -floating-point number to a power.

-

This is just a first taste of pattern matching. Pattern matching is a -pervasive tool in OCaml, and as you’ll see, it has surprising power.

-
-

Operators in Base and the Stdlib

-

OCaml’s standard library and Base mostly use the same -operators for the same things, but there are some differences. For -example, in Base, **. is float exponentiation, -and ** is integer exponentiation, whereas in the standard -library, ** is float exponentiation, and integer -exponentiation isn’t exposed as an operator.

-

Base does what it does to be consistent with other -numerical operators like *. and *, where the -period at the end is used to mark the floating-point versions.

-

In general, Base is not shy about presenting different -APIs than OCaml’s standard library when it’s done in the service of -consistency and clarity.

-
+

The **. operator used above is for raising a floating-point number to a power.

+

This is just a first taste of pattern matching. Pattern matching is a pervasive tool in OCaml, and as you’ll see, it has surprising power.

+
+

Operators in Base and the Stdlib

+

OCaml’s standard library and Base mostly use the same operators for the same things, but there are some differences. For example, in Base, **. is float exponentiation, and ** is integer exponentiation, whereas in the standard library, ** is float exponentiation, and integer exponentiation isn’t exposed as an operator.

+

Base does what it does to be consistent with other numerical operators like *. and *, where the period at the end is used to mark the floating-point versions.

+

In general, Base is not shy about presenting different APIs than OCaml’s standard library when it’s done in the service of consistency and clarity.

+

Lists

-

Where tuples let you combine a fixed number of items, potentially of -different types, lists let you hold any number of items of the same -type. Consider the following example: 

+

Where tuples let you combine a fixed number of items, potentially of different types, lists let you hold any number of items of the same type. Consider the following example: 

let languages = ["OCaml";"Perl";"C"];;
 >val languages : string list = ["OCaml"; "Perl"; "C"]
-

Note that you can’t mix elements of different types in the same list, -unlike tuples:

+

Note that you can’t mix elements of different types in the same list, unlike tuples:

let numbers = [3;"four";5];;
 >Line 1, characters 18-24:
@@ -512,85 +289,59 @@ 
Lists

The List Module

-

Base comes with a List module that has a -rich collection of functions for working with lists. We can access -values from within a module by using dot notation. For example, this is -how we compute the length of a list:

+

Base comes with a List module that has a rich collection of functions for working with lists. We can access values from within a module by using dot notation. For example, this is how we compute the length of a list:

List.length languages;;
 >- : int = 3
-

Here’s something a little more complicated. We can compute the list -of the lengths of each language as follows:

+

Here’s something a little more complicated. We can compute the list of the lengths of each language as follows:

List.map languages ~f:String.length;;
 >- : int list = [5; 4; 1]
-

List.map takes two arguments: a list and a function for -transforming the elements of that list. It returns a new list with the -transformed elements and does not modify the original list.

-

Notably, the function passed to List.map is passed under -a labeled argument ~f. Labeled arguments are -specified by name rather than by position, and thus allow you to change -the order in which arguments are presented to a function without -changing its behavior, as you can see here:  

+

List.map takes two arguments: a list and a function for transforming the elements of that list. It returns a new list with the transformed elements and does not modify the original list.

+

Notably, the function passed to List.map is passed under a labeled argument ~f. Labeled arguments are specified by name rather than by position, and thus allow you to change the order in which arguments are presented to a function without changing its behavior, as you can see here:  

List.map ~f:String.length languages;;
 >- : int list = [5; 4; 1]
-

We’ll learn more about labeled arguments and why they’re important in -Chapter 2, Variables And Functions.

+

We’ll learn more about labeled arguments and why they’re important in Chapter 2, Variables And Functions.

Constructing Lists with ::

-

In addition to constructing lists using brackets, we can use the list -constructor :: for adding elements to the front of a -list:  

+

In addition to constructing lists using brackets, we can use the list constructor :: for adding elements to the front of a list:  

"French" :: "Spanish" :: languages;;
 >- : string list = ["French"; "Spanish"; "OCaml"; "Perl"; "C"]
-

Here, we’re creating a new and extended list, not changing the list -we started with, as you can see below:

+

Here, we’re creating a new and extended list, not changing the list we started with, as you can see below:

languages;;
 >- : string list = ["OCaml"; "Perl"; "C"]
-
-
-

Semicolons Versus Commas

-

Unlike many other languages, OCaml uses semicolons to separate list -elements in lists rather than commas. Commas, instead, are used for -separating elements in a tuple. If you try to use commas in a list, -you’ll see that your code compiles but doesn’t do quite what you might -expect:  

+
+

Semicolons Versus Commas

+

Unlike many other languages, OCaml uses semicolons to separate list elements in lists rather than commas. Commas, instead, are used for separating elements in a tuple. If you try to use commas in a list, you’ll see that your code compiles but doesn’t do quite what you might expect:  

["OCaml", "Perl", "C"];;
 >- : (string * string * string) list = [("OCaml", "Perl", "C")]
-

In particular, rather than a list of three strings, what we have is a -singleton list containing a three-tuple of strings.

-

This example uncovers the fact that commas create a tuple, even if -there are no surrounding parens. So, we can write:

+

In particular, rather than a list of three strings, what we have is a singleton list containing a three-tuple of strings.

+

This example uncovers the fact that commas create a tuple, even if there are no surrounding parens. So, we can write:

1,2,3;;
 >- : int * int * int = (1, 2, 3)
-

to allocate a tuple of integers. This is generally considered poor -style and should be avoided.

-
-

The bracket notation for lists is really just syntactic sugar for -::. Thus, the following declarations are all equivalent. -Note that [] is used to represent the empty list and that -:: is right-associative:

+

to allocate a tuple of integers. This is generally considered poor style and should be avoided.

+ +

The bracket notation for lists is really just syntactic sugar for ::. Thus, the following declarations are all equivalent. Note that [] is used to represent the empty list and that :: is right-associative:

[1; 2; 3];;
 >- : int list = [1; 2; 3]
@@ -600,25 +351,17 @@ 
Semicolons Versus Commas

 >- : int list = [1; 2; 3]
-

The :: constructor can only be used for adding one -element to the front of the list, with the list terminating at -[], the empty list. There’s also a list concatenation -operator, @, which can concatenate two lists:

+

The :: constructor can only be used for adding one element to the front of the list, with the list terminating at [], the empty list. There’s also a list concatenation operator, @, which can concatenate two lists:

[1;2;3] @ [4;5;6];;
 >- : int list = [1; 2; 3; 4; 5; 6]
-

It’s important to remember that, unlike ::, this is not -a constant-time operation. Concatenating two lists takes time -proportional to the length of the first list.

+

It’s important to remember that, unlike ::, this is not a constant-time operation. Concatenating two lists takes time proportional to the length of the first list.

+

List Patterns Using Match

-

The elements of a list can be accessed through pattern matching. List -patterns are based on the two list constructors, [] and -::. Here’s a simple example:  

+

The elements of a list can be accessed through pattern matching. List patterns are based on the two list constructors, [] and ::. Here’s a simple example:  

let my_favorite_language (my_favorite :: the_rest) =
   my_favorite;;
@@ -629,19 +372,8 @@ 
List Patterns Using Match

 >val my_favorite_language : 'a list -> 'a = <fun>
-

By pattern matching using ::, we’ve isolated and named -the first element of the list (my_favorite) and the -remainder of the list (the_rest). If you know Lisp or -Scheme, what we’ve done is the equivalent of using the functions -car and cdr to isolate the first element of a -list and the remainder of that list.

-

As you can see, however, the toplevel did not like this definition -and spit out a warning indicating that the pattern is not exhaustive. -This means that there are values of the type in question that won’t be -captured by the pattern. The warning even gives an example of a value -that doesn’t match the provided pattern, in particular, [], -the empty list. If we try to run my_favorite_language, -we’ll see that it works on nonempty lists and fails on empty ones:

+

By pattern matching using ::, we’ve isolated and named the first element of the list (my_favorite) and the remainder of the list (the_rest). If you know Lisp or Scheme, what we’ve done is the equivalent of using the functions car and cdr to isolate the first element of a list and the remainder of that list.

+

As you can see, however, the toplevel did not like this definition and spit out a warning indicating that the pattern is not exhaustive. This means that there are values of the type in question that won’t be captured by the pattern. The warning even gives an example of a value that doesn’t match the provided pattern, in particular, [], the empty list. If we try to run my_favorite_language, we’ll see that it works on nonempty lists and fails on empty ones:

my_favorite_language ["English";"Spanish";"French"];;
 >- : string = "English"
@@ -649,18 +381,9 @@ 
List Patterns Using Match

 >Exception: Match_failure ("//toplevel//", 1, 26).
-

You can avoid these warnings, and more importantly make sure that -your code actually handles all of the possible cases, by using a -match expression instead.

-

A match expression is a kind of juiced-up version of the -switch statement found in C and Java. It essentially lets -you list a sequence of patterns, separated by pipe characters. (The one -before the first case is optional.) The compiler then dispatches to the -code following the first matching pattern. As we’ve already seen, the -pattern can mint new variables that correspond to parts of the value -being matched.

-

Here’s a new version of my_favorite_language that uses -match and doesn’t trigger a compiler warning:

+

You can avoid these warnings, and more importantly make sure that your code actually handles all of the possible cases, by using a match expression instead.

+

A match expression is a kind of juiced-up version of the switch statement found in C and Java. It essentially lets you list a sequence of patterns, separated by pipe characters. (The one before the first case is optional.) The compiler then dispatches to the code following the first matching pattern. As we’ve already seen, the pattern can mint new variables that correspond to parts of the value being matched.

+

Here’s a new version of my_favorite_language that uses match and doesn’t trigger a compiler warning:

let my_favorite_language languages =
   match languages with
@@ -673,33 +396,13 @@ 
List Patterns Using Match

 >- : string = "OCaml"
-

The preceding code also includes our first comment. OCaml comments -are bounded by (* and *) and can be nested -arbitrarily and cover multiple lines. There’s no equivalent of C++-style -single-line comments that are prefixed by //.

-

The first pattern, first :: the_rest, covers the case -where languages has at least one element, since every list -except for the empty list can be written down with one or more -::’s. The second pattern, [], matches only the -empty list. These cases are exhaustive, since every list is either empty -or has at least one element, a fact that is verified by the -compiler.

+

The preceding code also includes our first comment. OCaml comments are bounded by (* and *) and can be nested arbitrarily and cover multiple lines. There’s no equivalent of C++-style single-line comments that are prefixed by //.

+

The first pattern, first :: the_rest, covers the case where languages has at least one element, since every list except for the empty list can be written down with one or more ::’s. The second pattern, [], matches only the empty list. These cases are exhaustive, since every list is either empty or has at least one element, a fact that is verified by the compiler.

Recursive List Functions

-

Recursive functions, or functions that call themselves, are an -important part of working in OCaml or really any functional language. -The typical approach to designing a recursive function is to separate -the logic into a set of base cases that can be solved directly -and a set of inductive cases, where the function breaks the -problem down into smaller pieces and then calls itself to solve those -smaller problems.  

-

When writing recursive list functions, this separation between the -base cases and the inductive cases is often done using pattern matching. -Here’s a simple example of a function that sums the elements of a -list:

+

Recursive functions, or functions that call themselves, are an important part of working in OCaml or really any functional language. The typical approach to designing a recursive function is to separate the logic into a set of base cases that can be solved directly and a set of inductive cases, where the function breaks the problem down into smaller pieces and then calls itself to solve those smaller problems.  

+

When writing recursive list functions, this separation between the base cases and the inductive cases is often done using pattern matching. Here’s a simple example of a function that sums the elements of a list:

let rec sum l =
   match l with
@@ -710,14 +413,8 @@ 
Recursive List Functions

 >- : int = 6
-

Following the common OCaml idiom, we use hd to refer to -the head of the list and tl to refer to the tail. Note that -we had to use the rec keyword to allow sum to -refer to itself. As you might imagine, the base case and inductive case -are different arms of the match.

-

Logically, you can think of the evaluation of a simple recursive -function like sum almost as if it were a mathematical -equation whose meaning you were unfolding step by step:

+

Following the common OCaml idiom, we use hd to refer to the head of the list and tl to refer to the tail. Note that we had to use the rec keyword to allow sum to refer to itself. As you might imagine, the base case and inductive case are different arms of the match.

+

Logically, you can think of the evaluation of a simple recursive function like sum almost as if it were a mathematical equation whose meaning you were unfolding step by step:

sum [1;2;3]
 = 1 + sum [2;3]
@@ -728,10 +425,8 @@ 
Recursive List Functions

 = 1 + 5
 = 6
-

This suggests a reasonable if not entirely accurate mental model for -what OCaml is actually doing to evaluate a recursive function.

-

We can introduce more complicated list patterns as well. Here’s a -function for removing sequential duplicates:

+

This suggests a reasonable if not entirely accurate mental model for what OCaml is actually doing to evaluate a recursive function.

+

We can introduce more complicated list patterns as well. Here’s a function for removing sequential duplicates:

let rec remove_sequential_duplicates list =
   match list with
@@ -748,10 +443,7 @@ 
Recursive List Functions

 >val remove_sequential_duplicates : int list -> int list = <fun>
-

Again, the first arm of the match is the base case, and the second is -the inductive case. Unfortunately, this code has a problem, as indicated -by the warning message. In particular, it doesn’t handle one-element -lists. We can fix this warning by adding another case to the match:

+

Again, the first arm of the match is the base case, and the second is the inductive case. Unfortunately, this code has a problem, as indicated by the warning message. In particular, it doesn’t handle one-element lists. We can fix this warning by adding another case to the match:

let rec remove_sequential_duplicates list =
   match list with
@@ -767,46 +459,21 @@ 
Recursive List Functions

 >- : int list = [1; 2; 3; 4; 1]
-

Note that this code used another variant of the list pattern, -[hd], to match a list with a single element. We can do this -to match a list with any fixed number of elements; for example, -[x;y;z] will match any list with exactly three elements and -will bind those elements to the variables x, -y, and z.

-

In the last few examples, our list processing code involved a lot of -recursive functions. In practice, this isn’t usually necessary. Most of -the time, you’ll find yourself happy to use the iteration functions -found in the List module. But it’s good to know how to use -recursion for when you need to iterate in a new way. 

+

Note that this code used another variant of the list pattern, [hd], to match a list with a single element. We can do this to match a list with any fixed number of elements; for example, [x;y;z] will match any list with exactly three elements and will bind those elements to the variables x, y, and z.

+

In the last few examples, our list processing code involved a lot of recursive functions. In practice, this isn’t usually necessary. Most of the time, you’ll find yourself happy to use the iteration functions found in the List module. But it’s good to know how to use recursion for when you need to iterate in a new way. 

Options

-

Another common data structure in OCaml is the option. An -option is used to express that a value might or might not be present. -For example:  

+

Another common data structure in OCaml is the option. An option is used to express that a value might or might not be present. For example:  

let divide x y =
   if y = 0 then None else Some (x / y);;
 >val divide : int -> int -> int option = <fun>
-

The function divide either returns None if -the divisor is zero, or Some of the result of the division -otherwise. Some and None are constructors that -let you build optional values, just as :: and -[] let you build lists. You can think of an option as a -specialized list that can only have zero or one elements.

-

To examine the contents of an option, we use pattern matching, as we -did with tuples and lists. Let’s see how this plays out in a small -example. We’ll write a function that takes a filename, and returns a -version of that filename with the file extension (the part after the -dot) downcased. We’ll base this on the function -String.rsplit2 to split the string based on the rightmost -period found in the string. Note that String.rsplit2 has -return type (string * string) option, returning -None when no character was found to split on.

+

The function divide either returns None if the divisor is zero, or Some of the result of the division otherwise. Some and None are constructors that let you build optional values, just as :: and [] let you build lists. You can think of an option as a specialized list that can only have zero or one elements.

+

To examine the contents of an option, we use pattern matching, as we did with tuples and lists. Let’s see how this plays out in a small example. We’ll write a function that takes a filename, and returns a version of that filename with the file extension (the part after the dot) downcased. We’ll base this on the function String.rsplit2 to split the string based on the rightmost period found in the string. Note that String.rsplit2 has return type (string * string) option, returning None when no character was found to split on.

let downcase_extension filename =
   match String.rsplit2 filename ~on:'.' with
@@ -819,103 +486,59 @@ 
Options

 >- : string list = ["Hello_World.txt"; "Hello_World.txt"; "Hello_World"]
-

Note that we used the ^ operator for concatenating -strings. The concatenation operator is provided as part of the -Stdlib module, which is automatically opened in every OCaml -program.

-

Options are important because they are the standard way in OCaml to -encode a value that might not be there; there’s no such thing as a -NullPointerException in OCaml. This is different from most -other languages, including Java and C#, where most if not all data types -are nullable, meaning that, whatever their type is, any given -value also contains the possibility of being a null value. In such -languages, null is lurking everywhere. 

-

In OCaml, however, missing values are explicit. A value of type -string * string always contains two well-defined values of -type string. If you want to allow, say, the first of those -to be absent, then you need to change the type to -string option * string. As we’ll see in Chapter 7, Error -Handling, this explicitness allows the compiler to provide a great -deal of help in making sure you’re correctly handling the possibility of -missing data.

+

Note that we used the ^ operator for concatenating strings. The concatenation operator is provided as part of the Stdlib module, which is automatically opened in every OCaml program.

+

Options are important because they are the standard way in OCaml to encode a value that might not be there; there’s no such thing as a NullPointerException in OCaml. This is different from most other languages, including Java and C#, where most if not all data types are nullable, meaning that, whatever their type is, any given value also contains the possibility of being a null value. In such languages, null is lurking everywhere. 

+

In OCaml, however, missing values are explicit. A value of type string * string always contains two well-defined values of type string. If you want to allow, say, the first of those to be absent, then you need to change the type to string option * string. As we’ll see in Chapter 7, Error Handling, this explicitness allows the compiler to provide a great deal of help in making sure you’re correctly handling the possibility of missing data.

Records and Variants

-

So far, we’ve only looked at data structures that were predefined in -the language, like lists and tuples. But OCaml also allows us to define -new data types. Here’s a toy example of a data type representing a point -in two-dimensional space: 

+

So far, we’ve only looked at data structures that were predefined in the language, like lists and tuples. But OCaml also allows us to define new data types. Here’s a toy example of a data type representing a point in two-dimensional space: 

type point2d = { x : float; y : float }
-

point2d is a record type, which you can think -of as a tuple where the individual fields are named, rather than being -defined positionally. Record types are easy enough to construct:  

+

point2d is a record type, which you can think of as a tuple where the individual fields are named, rather than being defined positionally. Record types are easy enough to construct:  

let p = { x = 3.; y = -4. };;
 >val p : point2d = {x = 3.; y = -4.}
-

And we can get access to the contents of these types using pattern -matching:

+

And we can get access to the contents of these types using pattern matching:

let magnitude { x = x_pos; y = y_pos } =
   Float.sqrt (x_pos **. 2. +. y_pos **. 2.);;
 >val magnitude : point2d -> float = <fun>
-

The pattern match here binds the variable x_pos to the -value contained in the x field, and the variable -y_pos to the value in the y field.

-

We can write this more tersely using what’s called field -punning. In particular, when the name of the field and the name of -the variable it is bound to coincide, we don’t have to write them both -down. Using this, our magnitude function can be rewritten as -follows: 

+

The pattern match here binds the variable x_pos to the value contained in the x field, and the variable y_pos to the value in the y field.

+

We can write this more tersely using what’s called field punning. In particular, when the name of the field and the name of the variable it is bound to coincide, we don’t have to write them both down. Using this, our magnitude function can be rewritten as follows: 

let magnitude { x; y } = Float.sqrt (x **. 2. +. y **. 2.);;
 >val magnitude : point2d -> float = <fun>
-

Alternatively, we can use dot notation for accessing record -fields:

+

Alternatively, we can use dot notation for accessing record fields:

let distance v1 v2 =
   magnitude { x = v1.x -. v2.x; y = v1.y -. v2.y };;
 >val distance : point2d -> point2d -> float = <fun>
-

And we can of course include our newly defined types as components in -larger types. Here, for example, are some types for modeling different -geometric objects that contain values of type point2d:

+

And we can of course include our newly defined types as components in larger types. Here, for example, are some types for modeling different geometric objects that contain values of type point2d:

type circle_desc  = { center: point2d; radius: float }
 type rect_desc    = { lower_left: point2d; width: float; height: float }
 type segment_desc = { endpoint1: point2d; endpoint2: point2d }
-

Now, imagine that you want to combine multiple objects of these types -together as a description of a multi-object scene. You need some unified -way of representing these objects together in a single type. -Variant types let you do just that:    

+

Now, imagine that you want to combine multiple objects of these types together as a description of a multi-object scene. You need some unified way of representing these objects together in a single type. Variant types let you do just that:    

type scene_element =
   | Circle  of circle_desc
   | Rect    of rect_desc
   | Segment of segment_desc
-

The | character separates the different cases of the -variant (the first | is optional), and each case has a -capitalized tag, like Circle, Rect or -Segment, to distinguish that case from the others.

-

Here’s how we might write a function for testing whether a point is -in the interior of some element of a list of -scene_elements. Note that there are two let -bindings in a row without a double semicolon between them. That’s -because the double semicolon is required only to tell utop to -process the input, not to separate two declarations

+

The | character separates the different cases of the variant (the first | is optional), and each case has a capitalized tag, like Circle, Rect or Segment, to distinguish that case from the others.

+

Here’s how we might write a function for testing whether a point is in the interior of some element of a list of scene_elements. Note that there are two let bindings in a row without a double semicolon between them. That’s because the double semicolon is required only to tell utop to process the input, not to separate two declarations

let is_inside_scene_element point scene_element =
   let open Float.O in
@@ -940,64 +563,21 @@ 
Records and Variants

 >- : bool = true
-

You might at this point notice that the use of match -here is reminiscent of how we used match with -option and list. This is no accident: -option and list are just examples of variant -types that are important enough to be defined in the standard library -(and in the case of lists, to have some special syntax).

-

We also made our first use of an anonymous function in the -call to List.exists. Anonymous functions are declared using -the fun keyword, and don’t need to be explicitly named. -Such functions are common in OCaml, particularly when using iteration -functions like List.exists.  

-

The purpose of List.exists is to check if there are any -elements of the list in question for which the provided function -evaluates to true. In this case, we’re using -List.exists to check if there is a scene element within -which our point resides.

-
-

Base and Polymorphic Comparison

-

One other thing to notice was the fact that we opened -Float.O in the definition of -is_inside_scene_element. That allowed us to use the simple, -un-dotted infix operators, but more importantly it brought the float -comparison operators into scope. When using Base, the -default comparison operators work only on integers, and you need to -explicitly choose other comparison operators when you want them. OCaml -also offers a special set of polymorphic comparison operators -that can work on almost any type, but those are considered to be -problematic, and so are hidden by default by Base. We’ll -learn more about polymorphic compare in Chapter 3, Terser and Faster Patterns.

-
+

You might at this point notice that the use of match here is reminiscent of how we used match with option and list. This is no accident: option and list are just examples of variant types that are important enough to be defined in the standard library (and in the case of lists, to have some special syntax).

+

We also made our first use of an anonymous function in the call to List.exists. Anonymous functions are declared using the fun keyword, and don’t need to be explicitly named. Such functions are common in OCaml, particularly when using iteration functions like List.exists.  

+

The purpose of List.exists is to check if there are any elements of the list in question for which the provided function evaluates to true. In this case, we’re using List.exists to check if there is a scene element within which our point resides.

+
+

Base and Polymorphic Comparison

+

One other thing to notice was the fact that we opened Float.O in the definition of is_inside_scene_element. That allowed us to use the simple, un-dotted infix operators, but more importantly it brought the float comparison operators into scope. When using Base, the default comparison operators work only on integers, and you need to explicitly choose other comparison operators when you want them. OCaml also offers a special set of polymorphic comparison operators that can work on almost any type, but those are considered to be problematic, and so are hidden by default by Base. We’ll learn more about polymorphic compare in Chapter 3, Terser and Faster Patterns.

+

Imperative Programming

-

The code we’ve written so far has been almost entirely pure -or functional, which roughly speaking means that the code in -question doesn’t modify variables or values as part of its execution. -Indeed, almost all of the data structures we’ve encountered are -immutable, meaning there’s no way in the language to modify -them at all. This is a quite different style from imperative -programming, where computations are structured as sequences of -instructions that operate by making modifications to the state of the -program.    

-

Functional code is the default in OCaml, with variable bindings and -most data structures being immutable. But OCaml also has excellent -support for imperative programming, including mutable data structures -like arrays and hash tables, and control-flow constructs like -for and while loops.

+

The code we’ve written so far has been almost entirely pure or functional, which roughly speaking means that the code in question doesn’t modify variables or values as part of its execution. Indeed, almost all of the data structures we’ve encountered are immutable, meaning there’s no way in the language to modify them at all. This is a quite different style from imperative programming, where computations are structured as sequences of instructions that operate by making modifications to the state of the program.    

+

Functional code is the default in OCaml, with variable bindings and most data structures being immutable. But OCaml also has excellent support for imperative programming, including mutable data structures like arrays and hash tables, and control-flow constructs like for and while loops.

Arrays

-

Perhaps the simplest mutable data structure in OCaml is the array. -Arrays in OCaml are very similar to arrays in other languages like C: -indexing starts at 0, and accessing or modifying an array element is a -constant-time operation. Arrays are more compact in terms of memory -utilization than most other data structures in OCaml, including lists. -Here’s an example:   

+

Perhaps the simplest mutable data structure in OCaml is the array. Arrays in OCaml are very similar to arrays in other languages like C: indexing starts at 0, and accessing or modifying an array element is a constant-time operation. Arrays are more compact in terms of memory utilization than most other data structures in OCaml, including lists. Here’s an example:   

let numbers = [| 1; 2; 3; 4 |];;
 >val numbers : int array = [|1; 2; 3; 4|]
@@ -1007,28 +587,12 @@ 
Arrays

 >- : int array = [|1; 2; 4; 4|]
-

The .(i) syntax is used to refer to an element of an -array, and the <- syntax is for modification. Because -the elements of the array are counted starting at zero, element -numbers.(2) is the third element.

-

The unit type that we see in the preceding code is -interesting in that it has only one possible value, written -(). This means that a value of type unit -doesn’t convey any information, and so is generally used as a -placeholder. Thus, we use unit for the return value of an -operation like setting a mutable field that communicates by side effect -rather than by returning a value. It’s also used as the argument to -functions that don’t require an input value. This is similar to the role -that void plays in languages like C and Java.

+

The .(i) syntax is used to refer to an element of an array, and the <- syntax is for modification. Because the elements of the array are counted starting at zero, element numbers.(2) is the third element.

+

The unit type that we see in the preceding code is interesting in that it has only one possible value, written (). This means that a value of type unit doesn’t convey any information, and so is generally used as a placeholder. Thus, we use unit for the return value of an operation like setting a mutable field that communicates by side effect rather than by returning a value. It’s also used as the argument to functions that don’t require an input value. This is similar to the role that void plays in languages like C and Java.

Mutable Record Fields

-

The array is an important mutable data structure, but it’s not the -only one. Records, which are immutable by default, can have some of -their fields explicitly declared as mutable. Here’s an example of a -mutable data structure for storing a running statistical summary of a -collection of numbers.   

+

The array is an important mutable data structure, but it’s not the only one. Records, which are immutable by default, can have some of their fields explicitly declared as mutable. Here’s an example of a mutable data structure for storing a running statistical summary of a collection of numbers.   

type running_sum =
   { mutable sum: float;
@@ -1036,9 +600,7 @@ 
Mutable Record Fields

     mutable samples: int;
   }
-

The fields in running_sum are designed to be easy to -extend incrementally, and sufficient to compute means and standard -deviations, as shown in the following example.

+

The fields in running_sum are designed to be easy to extend incrementally, and sufficient to compute means and standard deviations, as shown in the following example.

let mean rsum = rsum.sum /. Float.of_int rsum.samples;;
 >val mean : running_sum -> float = <fun>
@@ -1048,8 +610,7 @@ 
Mutable Record Fields

 >val stdev : running_sum -> float = <fun>
-

We also need functions to create and update -running_sums:

+

We also need functions to create and update running_sums:

let create () = { sum = 0.; sum_sq = 0.; samples = 0 };;
 >val create : unit -> running_sum = <fun>
@@ -1060,17 +621,9 @@ 
Mutable Record Fields

 >val update : running_sum -> float -> unit = <fun>
-

create returns a running_sum corresponding -to the empty set, and update rsum x changes -rsum to reflect the addition of x to its set -of samples by updating the number of samples, the sum, and the sum of -squares.

-

Note the use of single semicolons to sequence operations. When we -were working purely functionally, this wasn’t necessary, but you start -needing it when you’re writing imperative code.

-

Here’s an example of create and update in -action. Note that this code uses List.iter, which calls the -function ~f on each element of the provided list:

+

create returns a running_sum corresponding to the empty set, and update rsum x changes rsum to reflect the addition of x to its set of samples by updating the number of samples, the sum, and the sum of squares.

+

Note the use of single semicolons to sequence operations. When we were working purely functionally, this wasn’t necessary, but you start needing it when you’re writing imperative code.

+

Here’s an example of create and update in action. Note that this code uses List.iter, which calls the function ~f on each element of the provided list:

let rsum = create ();;
 >val rsum : running_sum = {sum = 0.; sum_sq = 0.; samples = 0}
@@ -1082,18 +635,11 @@ 
Mutable Record Fields

 >- : float = 3.94405318873307698
-

Warning: the preceding algorithm is numerically naive and has poor -precision in the presence of many values that cancel each other out. -This Wikipedia article -on algorithms for calculating variance provides more details.

+

Warning: the preceding algorithm is numerically naive and has poor precision in the presence of many values that cancel each other out. This Wikipedia article on algorithms for calculating variance provides more details.

Refs

-

We can create a single mutable value by using a ref. The -ref type comes predefined in the standard library, but -there’s nothing really special about it. It’s just a record type with a -single mutable field called contents:  

+

We can create a single mutable value by using a ref. The ref type comes predefined in the standard library, but there’s nothing really special about it. It’s just a record type with a single mutable field called contents:  

let x = { contents = 0 };;
 >val x : int ref = {contents = 0}
@@ -1103,8 +649,7 @@ 
Refs

 >- : int ref = {contents = 1}
-

There are a handful of useful functions and operators defined for -refs to make them more convenient to work with:

+

There are a handful of useful functions and operators defined for refs to make them more convenient to work with:

let x = ref 0  (* create a ref, i.e., { contents = 0 } *);;
 >val x : int ref = {Base.Ref.contents = 0}
@@ -1116,9 +661,7 @@ 
Refs

 >- : int = 1
-

There’s nothing magical with these operators either. You can -completely reimplement the ref type and all of these -operators in just a few lines of code:

+

There’s nothing magical with these operators either. You can completely reimplement the ref type and all of these operators in just a few lines of code:

type 'a ref = { mutable contents : 'a };;
 >type 'a ref = { mutable contents : 'a; }
@@ -1130,18 +673,8 @@ 
Refs

 >val ( := ) : 'a ref -> 'a -> unit = <fun>
-

The 'a before the ref indicates that the -ref type is polymorphic, in the same way that lists are -polymorphic, meaning it can contain values of any type. The parentheses -around ! and := are needed because these are -operators, rather than ordinary functions. 

-

Even though a ref is just another record type, it’s -important because it is the standard way of simulating the traditional -mutable variables you’ll find in most languages. For example, we can sum -over the elements of a list imperatively by calling -List.iter to call a simple function on every element of a -list, using a ref to accumulate the results:

+

The 'a before the ref indicates that the ref type is polymorphic, in the same way that lists are polymorphic, meaning it can contain values of any type. The parentheses around ! and := are needed because these are operators, rather than ordinary functions. 

+

Even though a ref is just another record type, it’s important because it is the standard way of simulating the traditional mutable variables you’ll find in most languages. For example, we can sum over the elements of a list imperatively by calling List.iter to call a simple function on every element of a list, using a ref to accumulate the results:

let sum list =
   let sum = ref 0 in
@@ -1150,33 +683,24 @@ 
Refs

 >val sum : int list -> int = <fun>
-

This isn’t the most idiomatic way to sum up a list, but it shows how -you can use a ref in place of a mutable variable.

-
-

Nesting lets with let and in

-

The definition of sum in the above examples was our -first use of let to define a new variable within the body -of a function. A let paired with an in can be -used to introduce a new binding within any local scope, including a -function body. The in marks the beginning of the scope -within which the new variable can be used. Thus, we could write: 

+

This isn’t the most idiomatic way to sum up a list, but it shows how you can use a ref in place of a mutable variable.

+
+

Nesting lets with let and in

+

The definition of sum in the above examples was our first use of let to define a new variable within the body of a function. A let paired with an in can be used to introduce a new binding within any local scope, including a function body. The in marks the beginning of the scope within which the new variable can be used. Thus, we could write: 

let z = 7 in
 z + z;;
 >- : int = 14
-

Note that the scope of the let binding is terminated by -the double-semicolon, so the value of z is no longer -available:

+

Note that the scope of the let binding is terminated by the double-semicolon, so the value of z is no longer available:

z;;
 >Line 1, characters 1-2:
 >Error: Unbound value z
-

We can also have multiple let bindings in a row, each -one adding a new variable binding to what came before:

+

We can also have multiple let bindings in a row, each one adding a new variable binding to what came before:

let x = 7 in
 let y = x * x in
@@ -1184,18 +708,12 @@ 
Nesting lets with let and in

 >- : int = 56
-

This kind of nested let binding is a common way of -building up a complex expression, with each let naming some -component, before combining them in one final expression.

-
+

This kind of nested let binding is a common way of building up a complex expression, with each let naming some component, before combining them in one final expression.

+

For and While Loops

-

OCaml also supports traditional imperative control-flow constructs -like for and while loops. Here, for example, -is some code for permuting an array that uses a for loop: -    - 

+

OCaml also supports traditional imperative control-flow constructs like for and while loops. Here, for example, is some code for permuting an array that uses a for loop:      

let permute array =
   let length = Array.length array in
@@ -1210,12 +728,8 @@ 
For and While Loops

 >val permute : 'a array -> unit = <fun>
-

This is our first use of the Random module. Note that -Random starts with a fixed seed, but you can call -Random.self_init to choose a new seed at random.  

-

From a syntactic perspective, you should note the keywords that -distinguish a for loop: for, to, -do, and done.

+

This is our first use of the Random module. Note that Random starts with a fixed seed, but you can call Random.self_init to choose a new seed at random.  

+

From a syntactic perspective, you should note the keywords that distinguish a for loop: for, to, do, and done.

Here’s an example run of this code:

let ar = Array.init 20 ~f:(fun i -> i);;
@@ -1228,10 +742,7 @@ 
For and While Loops

 >[|12; 16; 5; 13; 1; 6; 0; 7; 15; 19; 14; 4; 2; 11; 3; 8; 17; 9; 10; 18|]
-

OCaml also supports while loops, as shown in the -following function for finding the position of the first negative entry -in an array. Note that while (like for) is -also a keyword:

+

OCaml also supports while loops, as shown in the following function for finding the position of the first negative entry in an array. Note that while (like for) is also a keyword:

let find_first_negative_entry array =
   let pos = ref 0 in
@@ -1246,15 +757,7 @@ 
For and While Loops

 >- : int option = Some 1
-

As a side note, the preceding code takes advantage of the fact that -&&, OCaml’s “and” operator, short-circuits. In -particular, in an expression of the form -expr1&&expr2, -expr2 will only be evaluated if -expr1 evaluated to true. Were it not for that, -then the preceding function would result in an out-of-bounds error. -Indeed, we can trigger that out-of-bounds error by rewriting the -function to avoid the short-circuiting:

+

As a side note, the preceding code takes advantage of the fact that &&, OCaml’s “and” operator, short-circuits. In particular, in an expression of the form expr1&&expr2, expr2 will only be evaluated if expr1 evaluated to true. Were it not for that, then the preceding function would result in an out-of-bounds error. Indeed, we can trigger that out-of-bounds error by rewriting the function to avoid the short-circuiting:

let find_first_negative_entry array =
   let pos = ref 0 in
@@ -1271,20 +774,13 @@ 
For and While Loops

 >Exception: Invalid_argument "index out of bounds".
-

The or operator, ||, short-circuits in a similar way to -&&.

+

The or operator, ||, short-circuits in a similar way to &&.

A Complete Program

-

So far, we’ve played with the basic features of the language via -utop. Now we’ll show how to create a simple standalone -program. In particular, we’ll create a program that sums up a list of -numbers read in from the standard input. 

-

Here’s the code, which you can save in a file called -sum.ml. Note that we don’t terminate -expressions with ;; here, since it’s not required outside -the toplevel.

+

So far, we’ve played with the basic features of the language via utop. Now we’ll show how to create a simple standalone program. In particular, we’ll create a program that sums up a list of numbers read in from the standard input. 

+

Here’s the code, which you can save in a file called sum.ml. Note that we don’t terminate expressions with ;; here, since it’s not required outside the toplevel.

open Base
 open Stdio
@@ -1298,55 +794,29 @@ 
A Complete Program

 let () =
   printf "Total: %F\n" (read_and_accumulate 0.)
-

This is our first use of OCaml’s input and output routines, and we -needed to open another library, Stdio, to get access to -them. The function read_and_accumulate is a recursive -function that uses In_channel.input_line to read in lines -one by one from the standard input, invoking itself at each iteration -with its updated accumulated sum. Note that input_line -returns an optional value, with None indicating the end of -the input stream.

-

After read_and_accumulate returns, the total needs to be -printed. This is done using the printf command, which -provides support for type-safe format strings. The format string is -parsed by the compiler and used to determine the number and type of the -remaining arguments that are required. In this case, there is a single -formatting directive, %F, so printf expects -one additional argument of type float.

+

This is our first use of OCaml’s input and output routines, and we needed to open another library, Stdio, to get access to them. The function read_and_accumulate is a recursive function that uses In_channel.input_line to read in lines one by one from the standard input, invoking itself at each iteration with its updated accumulated sum. Note that input_line returns an optional value, with None indicating the end of the input stream.

+

After read_and_accumulate returns, the total needs to be printed. This is done using the printf command, which provides support for type-safe format strings. The format string is parsed by the compiler and used to determine the number and type of the remaining arguments that are required. In this case, there is a single formatting directive, %F, so printf expects one additional argument of type float.

Compiling and Running

-

We’ll compile our program using dune, a build system -that’s designed for use with OCaml projects. First, we need to write a -dune-project file to specify the project’s root -directory.

+

We’ll compile our program using dune, a build system that’s designed for use with OCaml projects. First, we need to write a dune-project file to specify the project’s root directory.

(lang dune 2.9)
 (name rwo-example)
-

Then, we need to write a dune file to specify the -specific thing being built. Note that a single project will have just -one dune-project file, but potentially many sub-directories -with different dune files.

+

Then, we need to write a dune file to specify the specific thing being built. Note that a single project will have just one dune-project file, but potentially many sub-directories with different dune files.

In this case, however, we just have one:

(executable
  (name      sum)
  (libraries base stdio))
-

All we need to specify is the fact that we’re building an executable -(rather than a library), the name of the executable, and the name of the -libraries we depend on.

+

All we need to specify is the fact that we’re building an executable (rather than a library), the name of the executable, and the name of the libraries we depend on.

We can now invoke dune to build the executable.

dune build sum.exe
-

The .exe suffix indicates that we’re building a -native-code executable, which we’ll discuss more in Chapter 4, Files Modules And Programs. Once the build -completes, we can use the resulting program like any command-line -utility. We can feed input to sum.exe by typing in a -sequence of numbers, one per line, hitting -Ctrl-D when we’re done:

+

The .exe suffix indicates that we’re building a native-code executable, which we’ll discuss more in Chapter 4, Files Modules And Programs. Once the build completes, we can use the resulting program like any command-line utility. We can feed input to sum.exe by typing in a sequence of numbers, one per line, hitting Ctrl-D when we’re done:

$ ./_build/default/sum.exe
 1
@@ -1355,17 +825,12 @@ 
Compiling and Running

 94.5
 Total: 100.5
-

More work is needed to make a really usable command-line program, -including a proper command-line parsing interface and better error -handling, all of which is covered in Chapter 15, Command Line Parsing.

+

More work is needed to make a really usable command-line program, including a proper command-line parsing interface and better error handling, all of which is covered in Chapter 15, Command Line Parsing.

Where to Go from Here

-

That’s it for the guided tour! There are plenty of features left and -lots of details to explain, but we hope that you now have a sense of -what to expect from OCaml, and that you’ll be more comfortable reading -the rest of the book as a result.

+

That’s it for the guided tour! There are plenty of features left and lots of details to explain, but we hope that you now have a sense of what to expect from OCaml, and that you’ll be more comfortable reading the rest of the book as a result.

-

Next: Chapter 02Variables and Functions

 \ No newline at end of file +

Next: Chapter 02Variables and Functions

 \ No newline at end of file diff --git a/images/book-cover.jpg b/images/book-cover.jpg index a72a28886cda5d3dcb5a2a092c9089d1bd5c7848..db8ae3bd9ed7d25d8e24890cc8d260ae78c5f8d5 100644 GIT binary patch literal 76024 zcmeFac|4T+|2IC7B}pk$aw@W9%TA%Btd%v)kR=)G*!R>Csi-D#QYaN6#28YxsFX~S zHe@Nu7L_)8mSOJK`x-jS=X}23&*S&K@8A8$eRO8#nrnGq@8$J+zMik;n!{{kc5~g< z4A#IM;JV~Ws<*t?r`xVxr&&3_yzj~dHDH@Si6!$q?8qvm6a6b zl@;X`MM&zZDoP|3H5VsEX%Q)1k3c^ka)^&K4xOW^q^PN+E~2zqQ(0A0MG=RCK3`?^ zo5{rC*sikfy`2+|%x!z1&SHO@IQF^z!{MJ})Pw=gk5LD<|2W14Y^?Tn!+y58ZFh7r z{OyDnnydLwW*hS>ju#u9qpk^#jrk47hke7j96_ft+i}NnyxiQ}i@13gE#l>0%(Ivu zzm%7EDPC~J3jB%{g8aPL-`of5%U|CYELpsG3Ez@se0<9U`1tq)&>Nos>k|I|qygrA z96vX18}2Im0#O_r{{nXY1Jl4Ak(2OUO#Ftcn6IXJnv7x64!@YjP2 zaBS@J2bbX%EL_0Owvdg3i<6s!omT}8^0O~ozg&r9#~!B@q5)CLoC0Uli*&@q@fHtA zDyo4;8+5rO5-bP1hn$1dR%YBhwn6Wlm5Y9LX7Hv}jf5W8;akTaHih(#7)YMK7G0cJ z^XmO*RtdeiZ;a>`8k3Y=`lzLUe6yjo`+*b5Ic2r2AAaz|3PEodVtwM`Bp%a;_T=WT>f?a!l zcH!UmK=Su4FyG*K*|B@_};&`%8E_ef)yYmc(s2k|468eChKySx(#grKVg4 z7ui@jBqkpZ%6s;Cwcs;}f|A6Z#+P=tf@g33DD$H__Gff9-fA1#r>mJB6xVj^x-(JS zH!01mxt-DeDz)|C(!Tz>uj{&7LKJ42eu!wD=DOayziZR2i&1C&t3qEMx>`IzpChl_S4c}=~ymbxNw>KIwCC{CIVbJnDSDfJH{MI1l-%{qKanjc>;bwpO?cCEs z&pnT&!pXZJK^m6g+%9N{Jaa}Q8N6ppO{}$U7qqA%O zMGfPlY0Q_Oc5bf|rgkuKc=s9I#1kT}Oq~8>ChpX&+4LvoHA76Cm8OXUF~8#5-BI39 ztA3t~mi1TM#_u^C4y}6g=I*7H4O2O>{VU?n1-3Ro=b1HR}J~#1kZyf*XS2Y=#!5EYGK5ptv zp*qbj{a7j?%FQPJ`c?ePA%!brU+ffb?LGOyM%T+Hs?_^8`+YC6jAVT~N(zppFCCVb z99zr85yz6pTE+W99t0in?rl*`eO~!vHvZdJ+Ya8yvqjq1BL^-Y@1vjQ|0d_=aLo0= zk_W3F`|c@P@2~4`5co@i&Z>pV{Fh;V_AHC(ihZw~@LTzMb+Oxv>Ll1-?%wip_oDq) zRuszaNM^&QU4&Z>f(N%**)egc@A8GNOg|t#r*%3#`LVQLj){BWXSsm7tfO!C+8yih z4{KkNX~$J2BdCjJ>xuocie9&qo|%qbwtw`ByX0M0=!|efqyiJiZeP`$&%_mJ{?JI; zYe{t~wyK`+==gTay}`X|q~*Dp!+no8E$zErYKQpIdaKG_qz&65r3t1JaFlp20(p*x<|{rr4q-RJybjTgeM`CMCd@1^SAeb_NwZ=I1gY#=$d zbjC7SW9*pZDp{@Mp0;+uFNc1t+_fb9^}QTl##A3Q^J8=QXsiWa^oiNrCrli*?O~0K z{kHU8lXax1^pd2y^-Jjd&0A`uggSZv{L0hIGhl03hus~W-3C1|yTzynPU?pE4lR;i zE#;C?Q5JT5Gx7Vfggxr%>KV7=hPdU)L)_9Up7861ZIjXySYC5I7cY>#H`J-zB; zJ^W9g>VAc!Mdl&^R^%YC$IHlb{@*_Vn}*1=3%eu*Bh z-4Xt;H{aB~5)N%`?0n{+Qkt&0McQ~cWL5ua_2JN}%$okuxJsXK$57eJA1-cLw{iM( zen{E}xqB(ka}CWhPBav^v?&k0O%8<#+WS@_`{kts3y}aGTGDe1wHj%5EuGFr?gL4= zmwBY4xpdBWu40Sn+H-~5ch^pPp4A+yU&eggwevF1FHr`;>noO7a2-7|NfhR}drE@+ zx3?!pB{yV68DGxcrNDg6{K{AQyYW@ry>f#LNs?EMdK*s(2W2b&zcq2tJA z;c(0S1IfO@W+B1;A;D(;!DK(*AlSPK9n$p=_9glGd82Kdt~V+0=NAk2Ad{b8bOU{L zfB)he{QDPUl6TP0Bc|@bp+CRq`*`d9{Ibh8`1huDoW0!5NS@AcOI8PQ7KS>y@Qoey z!oo#F#KkX!wB4@@eZ~F^1AYHxpN{vxo~IM&Vny%`HW0J$4xYajed>6-i2U7tiy-e{ zY`=M!_jZe)SMY?oItTj&>Ny2Fp%Gx6GoUyALtC_#t)(R-k z-`&>I*_*ZBJkWXjUep5ix6C=v&)?d^)z=)Y7unZ+e){;(0SoApj$d%FpO3enulw93 zyz@s;7yht+$^3o`vb)D04=kQP0Db#;K9U^vudF`IahwJ7pO{o_z>e|F-4s5wkKkCc zxCGm`5Z>2eXRiM9clHtN`y%We-WFh2NwQ>@XMXu{3$PxHVQ1~cDY&`85pQ^B(XRz_ zd(tna{M@6AHD(L%^XI5xdywSM?ZMidfbTe*KMu$I+duY?=}qao&zu&sZHL|Pn!9!#+5#pAf7rP~Xp4Op zys^fPzT((C;OlnS;K7k_e(=u=Cx>%}&rqBz#^mRe51sq-?+^RXwWy6h|NY^Fp9khT ziN5=Wcq1@DgXJXoh4{Jz{YBHAgO%VI)^(&FKV=Yh{zZmyIzMHd)eVcEqKplTGsv6l z>>5P$Hb%<0;7|AAz>dN0gyWl6zu8(J8)J9{axoy=3$+T_MjuXeB4FgN!GCi)aenuoPhU)O*Y)%E3;e_6^0V64G5mP| zF2R}l`J%aA6zt~@hAqhTj~K_}4Pf{$dwEG1c>c@&#jwO4e?1-3Dk*d$951}sv zKknl{?&Cl1g+n zzd1t$r;JmCmkMfh?37?HEm15SD`I0}E~4uf=qjQi zuPEoNsHiBiSwmiFvy!v&W;Z9&MiCTeR!~$|P}(F10cRC8O;sh4xj#|3VW5kfriGrt z-0k3-mgwBL4jecje?UdvFVI~LwkPABK8|)M&=Nlw8--4cNkaHjz zbG$`Ri{Mm-1Z#;xXIZ20@&DQE-%i*3i7w7R&+`ZFhuz%8S;5uY)yLI0I0&Xz3C#f3 zsQ-M-{pTfrYWnA4Y(3^Lg4s1O`TGmzm(IuM=PiPQ^+RD8|J6bVAs(k-;Tq%@66oxz z9}3+Ro4;95u!ZaI^ZBRV&EM?Lm&57U#r~SQfv!&ApId=%KEKevhe`g^90TgdTvM*# z>APx)B4LnIQj}9tw}PaDlDekKCRs%#O+`i4sV06dWVeHVcPf^j_|vH%j4lv8_-{{k zan^M63-obKf`Qaw;S>4LMa8MP)g4lCq+lldFrfsr(J=1q#5Z&9~q$QZ(Y3?Kk!Kxmd z(>qJofVnMivIr(?noiCrc%>!kjMm@PMRaaA`ETaqKf3eW^#>s2@?Wk#>+V5*Zovnf z0$q2w!`%H3KcMspJ2(VLJQyBH#D-74>%tbOI~o8W^M{>IR8Uh-k9( zP$AUUW!_5q2f4_)IERS@`-%Jk5dQ64y0`|C{~$Dfyv5J^i~R4p*?%ja{}~9eR^;#G z^Z)mSQ$XgJ1vCnO#UR#qJ=O#pp!LEmmge7~&3`JfV8S%#SA8D6|6`n-yFV*T^*_K{ zBuv=A|G-P;cLCW!n?H%xws|)6&*wig@XrkVGXww3z&|ta&kX!OGy}hfYFvGR*B^jD z%pa=VAOtj5?Z&xqAr}YNVouJ*d_3GdeEg_bjeq3|^tbXqDOOv+wP?`_-W8%NR*3!& zs@>Q;IJdJefby{4tKA+$wOa;slk1PgZn%XU>};G1aC6meY;5d*JPM)tKUBN1Enr90 zZk$}~@WvLjfQ_AhA&vve-FEEZT;T-eZUV|@)4A}hYO`*Ym83!Oz@tMFx~i5588@AS zs@3!)&t=})w93jQ_+euY_l9GOcm!7;Kd*0at(e$EkP;g19YGaut|8Hhtg5$vQ}m|3 z)5at0@KX@l%u?-8NgMS;?AIKzE>2;zN{(Mb?(+E_wTjxv&Ga$bwX>Y8pRcRIPbO&Q%iWok-Y>4il3r*QY zJKW>2-K9>oHI1~CwxVDDyif-FWDUPtAR`C+rCwdBCQno4?SqV~ep|jN&s1BNN zrr}AtZTeSA`MFNKojsw3L_I*aOQ+jT!OXzG=(cKe-Lo|;m1#95+#zHd6Y_o~(flZt z0+r$Ba8fRX_?r?jH_9Q}mhLdxL$hQnDW}V=A!NW#$}e=;0P8j8&CumMQo$BWJqBs?|8%R+d8RrIa6qh5X)x58%Rw$WlsbNVH|)N<>PUH^CQw;DtBO z_awAZe!|{_w`59*nK$7TnvHVyZV&?4fVUg6dQNBv_h1D1ETJjGBn8P0ED-T5`iAaF zw-v+NVPnpvfZQ}I5wAlR>2@#~H1(`JG@ZIf=yG_isx{`NC-A4|I&wIm5sKtAHN0a#q2Bk(Pj-mU&{h*ui;h+YoG6QRfDI&0H zNS1vksAVt{_|P`04Yde_*nvVk3-dsttsxX9$$Tv`ay-#QDUN&*K6gV-sRNyq%5bz) zXJECA(0XA~57*~L+4gLKhTHK68YtzA8`Kz?77nCtD_{dMjG-zL!GoCzWX_>AaO0 zXe~t@IVniB67E-&&7}yXSJUw7R72l#Dz!^#catNp`4?E&-AyBcgopS5{8vh`6^XXC zh`ppgwseS+E$nw?D*SjF&(gBIb^g3UUBCV32}TJwP^0fUTyG z3|gC$lCCT1URT=Pcuz1L#4HoAX zCT?3f_17+)Mt^Fa5|R3qQc^1I5k)D{rBocld&Gkr@xp&RU$9wH0&K+~HUuG8c`Rxg z#;Xc);y{6sfJ{+}$x*gf;VWQo6W-Q9=IfA=p}-a-_(>4fe!0xmvlD z_8W4=+p7z3FuHqfc^yN5@Rc70sRZa=!Bu2&deQKr>|y<2iLtkQnBD@A34lu^u{@<0 zpt~&5_u?r5h@jfw^8uCu=)}ul6G5?Hj!J6L@N4n5WHdRppFs#NfPybM3=54EZHQ0~ z3UWq?Py*nf4gh->h6oenV=B`TKs-5!`lC*@3#Hq(fhb3TcEXebQbW@Yr7~adC=s$# z1@oFlpvzJroJdc~=zMf}I?)pE4+dASr{I3lnQ%}e7=t_#MMKn|yfx;at5P1Z0Jj3L zVzeS_A4+)_p~?Us$`LA42IIqw9sp%K>aZS26Hdu`8Q=@_*O1K)?jZ|(0wkM54d_p? zsSHd>9DoEiGX`j2XnhgFpk+;j6(Gf6>kAWC%RnKdb1zUX#R?VA`$BwRgH$gT0K%wPKHVJ5!Q%|vj zaydcsfc2p=y-`G;Xoua?Jq6k_B$~zDfdC?9x(V-fiteyoiFgXkBHi{n9VQr7oE_;y zE~U(vkcol#eXy2b8gip;?_(Izg!S12q*|s-pu8u#Jj<8EqZ=?0Q4BP6Oh^DkJp!yF z(|(%N@?#X5$ON1sNDYIDij!%12V{ao6PiHF54QqJw1!YKJu66c1U3R|=@kI}-WK1h5TF1%$S35Rw>D5PQL&>Ffm>WAWI6&<)80sE9;eH9 zg4)i2c)^61&mc)8Hr?;ig_CGEvNA_WG~pyfG(hS>*I_R0?0LRewAO>;Yz67jSPqJS(a0`q(pv#Ow@V(}wl?I4mOvvk@fuQvyAxj1S z;$S9?Y(OSMrJ!w?ze;qEIV=EC!?(fIU@H$p`4>4rfbnOsc$K!G`=Y$8O?s^WObr-x zV5XSr9i~#s*?xsJB4n1?6YhY$Ak&E{0J-&eTQ7iG0OrEXLBs+}N&=9a19|+S{j?|P zlyJY2E|TNo09bMd-6+6E&kOiA469(Tzby0st#dY*!s4D7SV#B=oDZEYM~c2SCj`hq z+M*e^tp;<<1_t_js!Vf$wN+|c-Wy;RTyfF|2Ix!pfxPhpivi|~tW$ts-W%&1bX$fJ z@e_rnCj6B|TS1|v!@6L@gNX*I&X$3tL6Zi?TZBXly$_mnbsWy2p+2F*!n}lx^1Qd^ zUyFz*e+Xu^rMCgl$pL*!tVGzOw=66wEER1TUH3Sn1lLo}*>SXusMy(i-I1heG*$&h_Bdh9y zb^%qx7|$JP}XrHP&MmdAV<5vsx%>32VnU^XhvGX;=9nfWDo&Dj)3L~fr=AIN-?Pd zfsiRa77Ypy7o7<}8VPa;gDpk)26Po*1V=V# zVD=Fn!gPXL27=eXBCg2fgJ=~awvmJkNEv7)wz6dcC8bi(0^AY2qIqBm1)lkZ;0uY%JzoP3k2rmY$-oBp!EeJ-=iJ2bHR?R0Np4-YTQCFO0S3CwgNFhmB;R!jhz z1D%ipo&@9$jh$tKq`_0&NncEcX`HhZ$SJTG*bm4Gu1_(@V#_Q*GbnPcK#2~LG>|@w zja4ExzcK|lJU(FE1nX*ta=NV~&>&Dypa|JA1MEly&LHy7vR@@TC;-9s2AFt9*|zn- zG=M(X9|lYS9|#j1D87{SIC3z_x?K?1oGsqFXr1(Q8VrQk+TVP-sHrGPUx0gQj3l*-NdJRC{$MVlbt4d@Q&9ilg=am2r1K|s@J zmSFAdjKO`+$~XUlRu)*Fk`W3FpeTl^lot=u67ZK(rNJ2lw+9+hL@FoN&O^(V4%h=r zRuW}fjYbGi|4mFlJ&Ho30R-zbg8vDeEvZNsKmd>d9EN;Crzt=*GK$$UJ$y2y#~>5} z0Cxewx+?N-z%2+xgojS#XTgEBB^!9upkqSm$ln0~UWB9=HGukCAOaE$BSam8fq}@V zQ}xZfEy_NC#Xyii1MN5fBnj+3+!quEB>WhX#_P~Wu!mj(a$?}b*a1A!KyVPNf<|C8 z5NmM&v_le5pay&FA@-6oCnxuS$&4U^>%)>*X*=jF#vYJOm&4W@6XiJ`i{KCh0aWwf ziNY5G^W{gYDFc`Cf=OPOClp;F6#z{H_0t2>SqnV`Fa#Ngc}CrV4@sW26ePecy=*1m zBa91Dtw8O>?jte=PG9YJm{a-^xFd|&pKgaHEE5d~Gh|r23R)?l5LbX`A%DTVxF?xJ z-KTHT7B!;YNQ<*NA9&^%j%@CAoT4v(9}exr;uPvsLmn(H106&f*D0V5Yk={V;s7In z*}Qda_($cScEBW&NdohcJ4m-TEad31t|9mWP#~QF z>)!zK155>6P{1^#LLemI6+wW+Pk_c+3*bDKD3(D0&+ z!#xs{74g02x+F8gb_+SGde>?Tg#4eY7gP-CYFD5 z?cTLmvR^*-iiF&-0TXxTXt&To$?AlaX3y42AKLhAcn`zgg^80>*syPruF)ioq+JI}&ou#U(X_rNAC(@=h&lIcD*uYF<408{?&SLz!5_+l)vFl~uGfthxM`0xR2A%WKKXG)^|D4s zd!w26ah;naC8}5K&`WurF#6)j$ZV9N*23CZ(c9GK!;bNrmu}WRrl@hU7tjKrCa;M? ztEQ1+fQ|sS?8}FLZ7>UIm)7IWBmJjWr{8wGsx)JCZcyTFeRaY* z$ESx^ZacS%i6i_nyJ>L(CgA)abis+Lite|dl8o00jKWI_?9Y}c^naSE7WZZ~}@^3K`z{+jp4x0`URbFX=y z0KOjx$4~zSbIOY5cO5e8I~HAZ{(y4Y1f~C$g~%Av zX)g9-UJ;1i3Vqp zT)^5ipARdrr1pj^Qqeh0K*k*D0BFx&$s)7q<0XhQ!a!_ACcw(7rx#Ix$z*UMf8^d} z6!%63H;h#9sTC}OT`f=fQALTuDxS4w&XDp6qtOVS7szU# zFPeP&_xGF^0D($~r((UzYaG!0ZQEsnU!m}b!aJqi8AqA8{VONgB8gJg@un5Y)8!ld z&YMQ|+ONc~UX?gTj;y?STYIGL1Y@U5SnZ+qEyMB`4`kcbEg3ai7~6OBvDD7>h6)cA zgLbaCb?njrD9SGdo{t`R2vx4c|G;Mx?9}^yV-w_D%U#%c;makutSR!zGNXvy#3Ehj zTVe@)NeEp&Tgnr2>&`%S1R&A6KpHIkeG(0eX#zQHnV>p@R~_$d4w91tQLA+m;OYX* z6#ywjEL5f$QaQ$eA$JgfA|4OUE?Nd|6;*5#qSlX=MjA_5EKI(x~xnJ4I+>L z*GL}0gaFoJu?i&QfRgF(0w3ju49cC^u!QX2N*(jn$4WONO7D*znmLJT;!h33Lt+s0 z7BU2J(D5(kupaH>NR4_aC8L10)u4@`GDGptRA&3_W|gviR6C8XoH)1aY4!(D5SDNc zkdlMxi+QhW-VipG4t#QE+(!5;oj_bI{6(y+&Sp0KXfyxHbH6J52yH5RBFV(1_y;T$ z?Hx|KCD~R~u08TJej-sZ?Cf_t{fCpk_OHJ8C{4SmCZg=|MA3IU1Nx*%KUs4xo{8K0 zD)LdSSa0p!m6Z5l=ck30y33aiXU(2IHp0J8+{N*1%ICwk<<89>TEWB#y^mB8R_n2L ze0*l7W%J$A-P20IU6Q%_4FiWndl@kSj&>HY0V3;^)}H#f#POct$iAID204sghS>Bo z9o^Y{PwgJU<5{n|u(STNI-k5|zIN1yA6+`p%GGZM4P-K6Us`|Fs%N~X`6r0*x2{!tQo&^r%zwH^nB~!hsz@CuFi6aA5>lGaMe@#uxEZu7T3yg5x)x` zQVhlK$6_X&mW_fAh}y3;wg905Tk5`ZKj)ed*yv1-7 z@hGrCV1keufB%hPh{i|x2siONZLb1 zEJ;<0*#0K|yI-I7z@xNUOi;x{87CT7eSQ13jKMvlD>j)BCvFvacGl6c*jM<2wD8ax z;R^|GlojeaByzYEO;Q`Ixx1_B@i4_-T`)&D6uY zvTQ{Ke0LlaJX+}3rd@Wkr}*>=qtD!WSB%n*<2hh#C1ou+`qnBZOeu8j~iE> z63Y}0+-u=C;dpw94Ao>ko32>U^1SN>k1_*l?{%(}sR+_;EvMgq#J@O*Bt0a2W7XG9 z3$GVDC#x1x2X-#%{9@U|H9^WtjoPrOw>&!f@TZt&vj&;fuCb3@VpMnReigm7iE(C7 zB3Jd-qPk3rZ^QH#jI4da(24zoqGHKFg9oQXqAXvLqBopf^;J3k(gH>RVUgM|HnVRB z3!m!&c-!q4pUvib`_Jw>T<<}S)Vx!k=K3M>y!J?_ zaJ^BTc|ydm9;e5hZu^akM6OBh%X+t>Dp1Jm#WPiycK^=Jdl-tdRzGM0;%R9^OTUf& z(Ehwz`{?^B*$?A#c#e_u5#38I$L$7ZI+wkrnw?IJeBBpVd!I~y_AKt;lkJW7wc8s1y!YCJiW5&d-zBI;5A2|GJn&ByiViEA6_oI}9EDGR(RYs|8u;|==c-tMo&d)*$?7)4h*$aWnIH^!3Zlgu( zi!o9R01BZR#e;zC@PU@2Up(yolp1AFQO<|rP!=0ZkiGgAfq{4-5Lx zj78d@Oa*cs2p2H}JfCfxGs7GZfC7+1bPJq&7VY{MGT8EQbcz!qL|3r{@g#|69uLkT zAeTJ>!khrukk=oGyaM=0ya(Y9NC^-iIdZrSA!#$+@`~~%j~(VaJ4<(J*i4GXdTSot zc5p*;baOx99qw?vTLr{LNfwwfP@Uv#o$DDPW1o%HhiWMy9>6BmE8bKp_!(!LLG zOQSE1Q0q^P?>jK56IlSVTi<_PcJ|OmuW=FxFD~K^65dt894|h^Bz&#&k+%^F+~#@xZdy%O<+qz*k`@ropRqz3ympgqa-zDzQ2IX@`8G7o$k3E^OJieZs0+(^9s%o})1`XKa^t zpQw%x92Dsrc~_;-{z~2?vOZ@J`B{taFH1O>OgJ}PUmr91*ncpt>&}A38`480+f#{? zk2^<*yTWu%Z`u6DO?#8%L@&8}F^}|s?8rl}9fkVZ=MO*R$nrPWTtISmLY@_+GLFLoI5Wi#7&ilpxn*;BK++8u zZ#jy|qOcjwC)yT319Rw5zgWdZpeEq+eL?|>P6(ku%7ZQkaW-`@e5h1G9q9lht%3T3 z47Z^GF6dhaD*b2&qo|}h)@niFf8j2)Eck^f_Jnv;0M45v<=!1 zJ{tIzVA_EygSB29U{s`QgH1Q=fg@e;uFld^m`IFG{KcjhGFv~b{;hY5w-mS-=&pIc zLi7@{L!~ke>eSn)6b8bB0l*}HBExM*_ggQ8KphvFv{&?fT9apJo$Uj|4>)RyYS?sS1J0c|+FIyxDQ&;2`?kiIfZH?9#?jFDQT(|a2bLL6WQ8V~WKoze|N&`tZYUJF^ z_N>k_-#u0+{Q7adrC^F|&GNcPxssj5oFkTk_pWB2?;C1O&Ki$wshfCs`ozRJdc$eg zIv6iX%M!M{%WvPPpvFNmvf?P=r=kbNJ8mBkJI8Bnxa!r03u`j3G1R>Uq zoFwG>0UZO69RuA49LNP|iF!K7)%9=#V0^G-kOl;1hR$yz_;5nt_FFkhiP=J3)JYU6 z05PONYSXNnQf`g+n8rje9FDW%=e?@{dtsVY6ABZ67mL6=72oB z`Me!69bOa&+tI*`LY0>-3f&{K0y!-{x|~Hh%GCpp22XUEk~Q-FFQZtqAjm6bzpT)Z z;GPfx=0n~gmi#&!Mf{Ge6qe~g3IN)a1!4>LLgq5GjT(k%F=TR(ilFNaL7&0VL~#lm z5GIyl%#mCc@m<+vG^ap_3jb{c1AYlo;A!&eb_h_U)0dd{pvJ*oA^+giIAoO|VuM0I z7A^18(;><^r6>y-J20&5koT4uWa6G?I`~E=MAjJi3?7ybRmeKsuj^H0R7509Mo#)% zpB&iR)%Z^8O`EL7`QN0PbJne|N|vra@7#a?$+;@k2z{?qJ1!zA$Ago34gE7BBjTp{_aDhhUJc8V zWmJ`2NL5;`Znj~KUDKwPD$!AsEl-8EU0imN=BL;1KWsJom0e-^s_u!7pmmxT1!%R^xtM>?841dg%tq1W}`{vGw%!uX|Q8vLEg3+r$1jWs%&$wBfS) zy2Dq!gjClA-`(vJXBR4Gz?W^)+2iXqI8$|cG9ZR*6*ZM{HOW;tfpI31+A(P$r{-K4 z!*TX98_s%KLcDLhRcc5XC*dJ}XMCi7<;uv!ADj0bH~CfI`O)4ZW88ZumcBb&cx$lV z{G!HoMxgtznN=Uw%H=;gLv`$E*Y|IEY9FNUtgpi$TwbS^txGgpwqS)d)`G#ra z7wa#56?ObVYxkOb)EqB<7d}k9El*u8Yg6xEB=GekW6!-A;mKPol?Fcu&C2ZO!0%!_ z36D-c==rWL<7nfj?p5+V$4*O9Ws_!7O~%vKEnRMLFKu`8`6C7K{o2oDw|o_{DOf*x zn7nT}&qcANHD$u>(_3%8uu@vKH<0`Li|Mqc2$O_So)X&V*>hd|Stn-IjN{JxKd@)!0@y&&e+aKa%o8OUAlM5jhx=egp{fg@1EtXLer!Zi0eIV`0G3cd zgJNDnnC*ocK4jeuQPq_x(&sD>gm1+%AsXcmWa$f3(r{O+YT!hW8oJJ2sQX; zYY)HAy6{&Bi9v}IV|MpiTDpN8>ktX^uC^T|-PYJ`a!?%9%jXT(0` zME1+}t9H8&7p$Aj`D#5gcCN86?@Z+7C(0iwjNG%ujtU0@Y_I$I_K8PsRy!9zV843C zB6(Ct;gv^Hwzzq3TV=Ef9bB5hakKIs5svaM`CwvA(V1dMobueKIsW=b-m8!rV-NR@Z1v z^jj|3QT5u*qjPibWPYuh*<@|!r2bfsc9PkoYMYv)J<}BHma+x@tOS3AQ z2cHWR_rInmkOvFNZU0Ds6#~%s^5e(&pS}| zVqi=ty9AmCwGdDN0gV;N6pzx0-+@u!L32^G9wlko@~ZhzeiJ2+P=S0o-5!xR#8T|; zL7o+WLmE65#NG3n5GDak*jv0qF?btj1%`v@zaf^<$6_0Zhylz&2?O*WAQL5DDKv1O z5EimfusA>mvPLZk|I$+74hU`#i~$JcoA&}pVl6-qAU6PL2c-?Ljk*He1-1ov|5xmR zA1~Ge*nwd~L|_73A{tiXb{C5B;4vqttII^i??C2SG4=|DIbnI#zmf`hkfg`;y?CbG zv-nc1@Xny2nT>u5KA|x1@xsYRchA`~aTi=?EZjH?Ck}tL?5|T@r4r%zQM|}`to=}e zXXM@R499myQ(0!4m^e*s@%E<&enfcKJ=a;_pDC;%fi;s<6y>z*BO~l8PKqgLhQ+eR( zku`Ex+chAr={NEyvG3#9FFu|FzwGSu=+jk+=JnWfba>i6tiX9-=ptdUlahDx6YWcH z*2X1L7higu&~HSolY5zVswOc(H0^WM>nH9<_0Cl2b4oQx zySnb3_dH#D`LdzBcP_okeLe5;+!bRcY!acDCY!JBJy^S^^>wz_FO^}WGf5lz>Ntdo zZtzuoxv_R@?7~CG2UGS8$A1t`%vW9dQnRDD)x$OMwU)?Z-JOkD2XA)QBva;vXY^X1jUn<@Drc)2&S0<41=M?k*eCH{Ru(b}{l&X{t-( zSH|7+y=rk67g!J`TDxYy@{`D`j?NU=Pr8=-3$Bx1C#}q%tu`6(p*>A(;Fwj}4jE;^ zzSxvN-J!#UuU-941%4@y*;2k#&}cT0@4H@OcVnS9>F%Z@hxZwH;$@4&&r=Tr_JyW@m^IZ< z%`kY-U3@{J!umsLNzOaR0XYGcx1Rb3llyiyvv2gTUNyezQsm7#Ba`L0SKpL?$NEAoPh~dS7Yxw`E7gk?00D` z9)2;?lQyeTHu$6>?a*Qcb!rhjUjzwhGV%#i_7kpwrwLCmL4_V9Q=0JbfYA*Z@b#P^ z1qeCkFN=UCL4dUo9$@=e4J7L_PctXfVn*?)qiSYV1j?v52-Tn;cXI(4e|JA62UW)7x(Z$w&2cs5Hrh zx(|ecz{{GUW)p&pui&|&asW!i$jZ@Uj{s@X;JZO~#TWt+kdVZJz7Sjjs6||{*nq=} z2T~>l0C(W=siMdadX@`lA1YQunNaM>1yl_N+fcHCtOT}+Dp%%$s2G2nd-e}|aD&Af zkxBV8b%S;tC{@g+Pr*arhx%|L!UzwiyoG_It|5~HmdG0$bHO&jtO7~&cmd=zw+VtB zK<^KjI3?}g+d(6a6Gt%u$8a=SQ{TQ8Mru52r$zewN=Vt^F>dC2v)(n!S^24|FTkLT zkH;EnGI1K_!nT)=oPHH)a!5g}M%bkgUuQuh{F-b-cRBl*M>D}JW@{qxvti03rHw<} zjl^JluSH!k<;gaQ-dWoo2``U5zDK-|9&2+dG2S?8bozp?ujqEMTW4I8u1WLM9{nxn zYjgaU_r^yGKgs0~Ba>}u{mD~qOZC&0b}7>b;`e5rxvSx_lU(tr)2rvnbMqx)CmqD4 zvpY2FM0MBOSZzHnwnrd!!L|0TeUVG)(|g^Qx$dsN>dCu|R(x%@hQu<)c7@!!(1{aE z<7e#GINv4POj;7HVl2suM#)OmyM?yIT~F)VKP1*!$9=j`)T{Jl!rQhNLPnx`(MgF1 zIh@VL9fuT-ePyU~+NA3H;DaW83feN>daew7=(|tcU1YhI?gNjKx2@eXsS1PIyeFTn zcuXsNr7f@KA$I#tS#;`?H-@Hubo(KvUm`x7E0YKs${X%ww8h?kIZ%46HT+zzhspz= z>v;yPa&&J0>Joyleng#Vwgdyu|H_~ z?aiBS`WJl{;dwq87+9rlQ;v4im%p>YtfPKM_m?dXBq|5GTOF0(*~#@?h1=~cdqmZj!yDC4bKGQP#e7?<99GzzFzaVvz^xP0 zJup-fn(?-5I%nLvbs|XogXs8F^kiqp9<@iP>tOfTld)@$=hY7Q{me| z2mhH5wd2a{<|eIXwbWV7J7!uVN%5R#sP^RXdxcpma0@?G>tD}})1TgG`DN2SGC|OlJ*b2=-rq=MnC~Lsmf6+R-%z7vSj{j3>bpGEgb@sT3Y3K+gu)?x4%x zhfH)q4#G?9DTLx!ROSPf?ds4Kpp-zj5W1Oicp*0xs9G7`GZSmH9HBR%0znkS{m^4< zkahx=2Eq3@06{1dN604Di2xD+5ycxj1)v}n!5%yi=m!tIpyyg3L~IGzl?1LfD|`=4 zp&Z~wN7LiDRVyNInT#-m4ac%!>qK4?1C^6<%2@sH}R8uzjtWA6M_%EBik_jgv#2G8U_y0C#nEqql| zV|3}I<&)CTb%}LFapzO?KCJHQ_DLdaVdBD4bLg|Bq@fihv+8GStiG#`6@{jmmX_Gu zyOZ#?GUtbTtg`M{XwJ7j`LCb!IxE9PI=4(6mMZC7Za?G6aVq4n%v|=$8JyboezkYz^|zm(b}!AeB%x6@aJYi4Xqn(XJu%P2rB?B2f|LH; zLjk?V*+cz?IEm&=+(p}x^9`N;sVh@WR2o&2Hn_W1OmIzUbUphRF|2MmeL+3vi_^p9 zvQwSk_pEK}o$MkX;!MnLo3Vc(Zc!Vc#%{*zr+(mAvclVIkMDG*1iIFT%bt~--7Vc-q%Y6Rx&N+e`uh(@;ca1Ji`BmMFma}Pp2;+A9m#zC zY05t6-PBEex$`%N_onKNHtz8+c$b#*N$1fsH&0PuEH~ z74O{;>S1u@#SdTd$6Xtfg>O}HoQ>?2{Ni#)%$>o+jT^_Bx2}t5G+r!J99^xKeKwUE z+-lZ3E3G}~$j5(jR-0P=8$Lw%=^6PRL&ZflRV~#7o$p)gRJUGC>)N*neESVz{K8)% z+hVrLYHPU1XpGKoO{{+wJXFT^@#a?VJ9igJo*e0&&Kc7_$e_ku4r1bV-`Mm?TTQh; zW#FT(zc|??>FpjV&CZ) zx!s8FjcPQoKKt7Knj_J4(csKp9Y!Reb!hlrrI?D$=wK%9(gEeVAnjMQyLK01*QJjw z&UaIw`hOT$>evktfX2g)@4{_j+3(D13s>Kf+VbLQd;0_K^3vk7ndxU!H#w`Vc)6~e ztPTxs6W*~O1VX4=aQpXt0why8Zp9{kl}>&WXGkAv3mNP z+btZkV&RXlE{_h(uC})P?%bSw8372& z>cMj=;5f6Mr-G=URX2bhIOk9U1Jx`Q7nU_32N_Dz;OPLA?HFY}$7{(0Fxd|EJ_ws7 zc>q!SAd-e~7!njvGK6R)qPl>>G{~kbL*dGk_uoO$MzIZQ zXry1^`O8!6=rLlwoe{MQY;Ew7bK@nHZ6Y0a9j6_LC^9^t^5ljj z_rs#~t0tFMx%%U{#0l#aQOXn+7`Qx^gkyb!&GdUIyP-eY6Zb>CH(I9_V*k0RGv9#|qUXXw#CHi+4x;4er(=5RGEQiwFg zR(uV&2lOvn))dMw!odRiHDEU1nD90Ll}C#4fNx<}i-+e$!TiGxF#9sat#~_K6hvz8 zL3S8bU_z*60>!Ygr#`SJ;Z%`@hKGQ#ihVd2N<`qnx^lb^pS0%*0nH1LYAeWwP#0#t zu{MA|oP-*41LWsG=vNSn-+)qJiEhYoLFd6k697sAz6GcVk~^3w);*z=YyvL8GQqa$ z%gcDy7;tz%`0H>H{2B!G5ji~20dBr70Ih<{*tI=fB~AH)jg{36zmpRfw)S#@#f zLRfy}{DIiP=ifw@(1VDG34l=tF2zd#2?o`Pu*rK9KBNnEd=4f>z)T=i2g!W6RWl#d z0zm0KOVb}zfjd+gGJ6xEsMebWxBd3PdCI1LUmLm?4DtM%rYAzp@4d8W(XWcs^D1iS zJBNr*p6`rH=T2nY8xad`{8rV41NN=>pr`hZO!s6^za?OPd(_BiE#p}&J^oOhLSOR5 zOS6ehkq`92Xx|8H_)d5@{P5|urUChFfrhY4F>L7j?i@zm3#H&qc~u&gQ6OWB3jo8_CCZtr9aizW{ziK|C%~3ldR-SjI_5EzR@8ZM0 z9j|y^LB{TqOlq-iy8iVyvjm5yGOc^0nYdWv$o~CLs?Hk*FmWc+D;o@lk4CJ!de>kZ z+j4FD+mN3A$i%_C#nIBgR&A_FIAnH+{Xj21vNxQoSUYpJ&gU1&KA{DKyCg#fF>~Uw z@RR>V*_(h<*?w)~8Z=0WC>awnS7ss-nP-_rWS-|z*l93jCUYbq^N_is44INLi^y1} zGH2L!|8?DZe(&>s-}`;v@jp7A=QxtR_r3SN*IMT~*SXG%ZKKw9(1oBSr8Q%$a7m04 zR$3S*_Y6t&l-*yd>vKJ8b?T0+MsV3CSN<>et9r1=~y%?Eu9v0__62 z{}8T*@eUBZm^H~mly?eYLv}AuKCdpUf{OqW1zk1BlLskFU8G^F8nuok_&R*N)h231lWF@{r}Q-d^7k*=sjC0O0_D6Yg*T-#dLj8e*41aPaEFA;%&Y>oNaHQrI+}p?@0&GeOvmhftu<9Xo z2`U6&ErOV06ptWDNk)PHHntf31AzUVb#mxGP+o_9SrG^&l+@9CgNZeS7sm0Tq=nL! z8B%mX^uZsL`hY`#pZ(udB%mP4;xn?+0y+moT4;I%YYk3VIn%~8UELUU65fF-%B99@ zzsGVF(6!5b_aobfuSX1TU;F0lkDKX`6pS!haT&a2A`VfPay-jLqB7wVoKTD7mmB)Ia32oIizNEqTmFl=&l1-8RL1L2M@l72z^6hpxX~oxC#frpv;3seLNb+qe?e*#{0@Nw9T=eTa+sYAixlA zH^ZUd=nH27s!BT#3HPADdvd-csgBP`hoK_;5;S(OSORNaaBJu|`0==dwo4s=#*WAj zLJCwGfR<2&%7fbsI>XJsHzw*1ckWUURbwmAblT~#ie-Z&$2&FRZ7wYK!Cx@s1Zy_% zbjT|RIzp%z*aSllMOJ>X5)4g&1cZw65Ggx$mpz1=9WAQD%cHq23}E$;W(o~~|Hje| zN~wT1`hOWwKq2`w-Na{PQO3v%MpColSgeO;{|KekE(8@^RU4FI`$oR>Qw6Tx@4B zc~QjiwJyT8%HSx!9J8af%^*t*?;uGUYiR1XA6bV>G*>9?Z`RvRJd_%^`M&4Qp2PG% zf^V;zX|%iF{FE(tU20)u6;7QJXH}am&fN?=v~SRK+a^4a%Sr1j)kWE9uY(lNyWH-H zia&qN&hay9M#%qrBq@i1J3(_FUqt>pV_GvI-C4>^4$7dTpC1Z?7CPk*k=^+aJ#yuy z9)&bg!wAzzvuKm1gI5C2G0>BY#grKH-fevw{JT-zWrj2AXk8AM%357^LEbB^PF$+Za@6y|>GlCT*LH6Y7k z4`-wB9uD7dP_B+YWFt`G=et+=>*wUeSJ;?Imix{8x zGjh2MdcUi%(VMTnbH8Mx`vLCK>nMZizzWTa^I$)Kj13zVtoNYU-I)*q9}YtZ;laURRNv(cxdV(E@tPLRF zd5)|}oq3ug=^b{zA+)5HqUs*`JMZLZmLz5lma9&+Ut~Lh7+~>J;pAOrCfCllkVBpd3R*>q|;fVVGdvZ72Y%mL)plp5?tro^d(siTjjL;3w3?9+IX=6FtiJrbSn`zf9X~pXY z)sL_B>dcE(8nb!cFu>*Cb0eR6N@30G;JfrizjLWyGA~!|8O6yv!snLU4xQ19mL`Z1 z5)v9Nm3pK<&rmkkXuSQ2?)hAI{>8{NIy>W-Mp7k{&Y!1Q3?o|tvo97a9=UjS{)fiF ziOb#L0VG+TFC4TwZ^+VK*G=@b5ava?y!2(nRZ*5;uY@Xp^-J@+Y`U*`d`CVB%KZ z!T=ksJZU#xt2M%URD>pW(}+OHaq99jvOyzQ$h2^?K_$@6S$i8>4#k7ra~*+$>AHLn zi_0Sf-VI&L?&<&%SewFD0KsTv2dkg}0ud}=9GK;hd=`OfB6}bD9c-r-_93y&8F+al zT6uPHQ3>|N=&{=qVi;DvXHd${(*|2EY_A@gg%E9<{Zu!>0q!R3OfD*+h6aiWu#h9L z8;R6^&IFE&D25PBOG@MA zxbSlM#9-N@@Bc1oO_x2^`hK12d~yA0?!$2{F9Zj)9B7GW4h}l4cNZ=RQBSZ%NQJuT zyO35KFilardwaOvNXEfLub%qb{Kkg2F7Nm7DxFZ%ugCAW%$(^FgwE^i zjU0=8@fEYPwbGT$m2Y1RlfO+#z7a^R9%*~r`qzi;^r7oYO*p>s(VVCGb}%SuND}^v z02m*sp^);Eo!khsy@x3&PYVN9+jVkbz^Nkf+5hb%@5E^P4;;i9<;XwGeoq|Otto-$ z8Wa;ylaoLRLYuOPTxVFPf*TG5YjC@QMbk9q5&}aq#c-#?y2mh; zo*lYJ9M^Mi!MZ1s&>Cc-Mjs@&U(gH_O(Xkrq09jB1E|G7mQoCfZ$X2;qjt^JUW3Id z#L;6;F)Rq-1DJz-GHB8Pr7R)gFFZCA6TAD003L!q19Z1}+E7$N&}E6B#U4m(N# zU%Y1T2sgwMR8C~4$g6~D>fi~!Bdsu2Me;FKoms9Ys)4|(WBX(%0dPT?ftN>o0_H@R zhmr(DCo&$wsI`AhI{ymOkpl*--9XVsPk<`a|1jz_`O`JCAK+MKI+{h=V3aUIgicb3l_!@_J9lRQ2S@=J+b4=acwH;rt7-sbz$6w}g<4)6CN&v*53A<-ku|XQrF9Rpxmw zF9j|SOTinee~2dF9P$=vYT7W|N5YwXvW5a-6~~^xkWzF|8;^B0%f&5L?D@pwn`lNs z(R?(9Ym0@Xk42wJ`HJ)*&y~a{HX5}XC&QdSSEi6b2pWp~O8LR@x{as$anw28P*DY1 z?>z)QhU~}5@5P%oK5t5XB-rtWFw{=hJm-1cIa5*;Y-}s>d4Dk3>ZR?uky-D9ca5bw zw&x_rz7rbHq^+;w8X9?CFAYU0HJ*H+8QF1Nn4x~7cGLy`K>0z-CN5y(ql|~SOk+Zt z%)wm(w|6CUf8$UNNM*eiLvBLHq<`cIXX!zec=i{f3Zx`kpY|GsO?)Fc8mFs1-8-V6 z>@7$X!Ff|$R#r39C7k?u`S<39ul#D(?|a>9S+>}=-KQgCq|9PlGVLy0)48Sm5U_uuu5uwWMQbCgex=kmGH3f5 zZcHWV4G}~q3oqROE9B@M0F!1#SYdP;Pt4ctA=m-&wid{_|BbIcRKw`ojFXvbz`6Y8 zbEVJK0fd6SxbYAK1gfG?>!7+3V^XUMSa3kyH)5t>?xfufOl~5$X#N65fY=&s%R3oC zzyL9dil{wAAQUX9V)00QT0`to2fz-qaGGBP%mqqGNnbQs0_`V_nSTu)X}cZ*Ylao7 zvzf#I{eT!o8(M(;e5k2xsG&ZBtP->7g7^eV3JJg}V38C4{KNLjp*0g&O9c-B6s6Fx z_ak*Y^vht)h?rRPrcf0_fddw3$*3p6{6x{xbnjj$Vh_P?4s&I!K1k$<{pXp@^=66?TO5#HB2qAuo( zxU|vJ>gDE~hl-0BUPyIl9*B&s{w~OT2jA4h*_*h<=ylSeQ52O?QgQs)ca+oOTkjn{ z(|E4PJLT378eDjfS=QG?j%~T&PR-EUA9YguV!FwtzHtm5FsJQGuA|-}tEJ+2z;`=@ zzIs>9?guH?q|9RSAvP`KG;u9P6!GCg?`V~NvRcM}@ zNd6iKKesZn$i+Rc@^0V6K8`hJTG>}?CZOB2hz^tdj>{k%_(Rml%l{LYFTBEsM41Ch zWF3N{A|7J4LuPBW%ye5A)B?KN8U^4KB$dL6IQ)$4) z;wEl9NBK76vW9T1zxyha^GAsmEtA@_`>KkG$+ObisI^`mj$bP*-%}FvZk);X)%l-% z+&W1?CvPBkS9z9`cCW93W;0AQ~V)vAJ`?d~&tU1L zA66oP-ve+7Cc!|c0rCR^1qyOE>|eV;k%G8Yzlp*U+D;UNBOp!;0doaMcy$bO0w+#V zI;s;4ND$R+v>=JfCbsZ_urc;7$9S<83&1BZcxYgg&%7H2Nm|1l2ue(D3LPq1OGDO$ z=g~4b;!UtnO^44Ea4xpW1q$$;7lw?8HGBVksS7L5|E4_5HnV1TC#hKFWAP*!e>fH(ntJO!K^TODVrdq(i)qE+=P0TOgk+*L_f4%Sgx zqn*I%aQN6vWc#R`smCW-PcLWMxVrIN5lZ&o_hxfz(q3<_gwj)x&Q4SHEor1czp#`N zPB_!tuTBo%pzB0sG8`#!+w3)+qx7ccx3}mug8fp^~j?vzTb-o_LnXlj}Yi0*PJt#EQ7p3z9pjn zT6lT*r_vMGeeoK-?Yn-jr+R;}*`I}=2Ah{@Vz!H>7yGQqB>uf$^$t%8PoxnVh8bA4@BiSf~7g5I1qvodS`ZdHnn74evkRsv z;GEfE*~_}X*b&j1lgwcs2P%f}eE$HSouK~HJA?Ml0H9;tcaLuf zLMG@E<=_SE0nGzD3EVGyN|2R$(R?e9nNq@gIlJH`G=C=zg+*#2=9rpUs_cZZT#=Ww zRSq*GLZ0U4U@LhZ&BY+b3WpPTIe`Sx*k@Tew#9?IWST0MbAn?fNcgfp$nZaK;NmU4 ztMtMru~+1{e^$l0-vEzW1X@+M+VDfT@Z9aT>lc{`PjYk5R3GM<+thM)}ISSRtyy0q)Ks&exBt@LY>74Vg?N-18Ypk7J?&t{$cp z-OFIovCx-_zcEfIpp4P&aWvOI^-BBBO(sL1x?S7fH-~Dib}4&h8ei>hE$%9yDP(B& z{ED-(f1iA5;-`yRn2^?^<`=eAM8o6;%(B;Ac)SQ5#qAQW<>lpFWNX7DI~{`UoJizf zykQyY8ypeT{BR>m(=S^dcvikg5=qq`yLA*!?U{FVB^G7$ZfnYoJdS(PBtcG4B5*33 zUtsmU?q^vY5~)3^p>`wbwJXJO8vkai5I>D5^99L{3xw3L+NIXiMYi_L=;`(6AerVz z18ZPBV9h#*!F2205^7_?iYU{2V4uIr{^m(u=kM)5Bx9d$U)CBIoAgn8qr`(q5GXaH z==M7Rg=P_zDsS*9tc3gBgPO_|i#gd8Y7n*>c{b`m*98iF&vm%ff!Bv32Bip*FyLW@ zagk_k0^nY^DJoD9rk|1y7zejMl0Cl)>_HfleD?uuN>h%|&qBuk$4si~QF%0z3suBbd zCMkmpg{PONrgG|fh=7V7idZOk5HZpiZv{k=5f;2qlVS5(58jU=8df9v6v$NsSKy;X zIDDu2q7eRnjiqrg51aS}&e{AheDQ{s88H7yaH$%Qs~)Oo4YxeUXVDPAxes;PYpA`~ z`Y21mK;qhOGyfoT@!*N&z4-muHRE5frZU^!EBRdPfa^gCy>_?;U9%iFkj9M6r1k`x z=y*A)dkM7B-q9xGG-zx|d`!BZx6JEo;JAci`1l_1%_&XYFo~^T{ic4{&T8Fbox4LSG