To use Kotest's property-based testing you need to add the module io.kotest:kotest-property-jvm:<version> to your build. If you are using a multiplatform project, then you can add io.kotest:kotest-property:<version> to your commonTest dependencies.
Upgrading from 3.x? The kotest-property module is only available in version 4.0+. It replaces the previous property test classes which are now deprecated.
Developers typically write example-based tests. These are your garden variety unit tests you know and love. You provide the inputs and the expected values, and a test framework like Kotest checks that the two align, failing the build if they don't match up.
The problem is it's very easy to miss errors that occur when edge cases or inputs outside of what you considered are fed in. With property testing, hundreds or thousands of values are fed into the same test, and the values are randomly generated by your property test framework.
A good property test framework will include things like negative infinity, empty lists, strings with non-ascii characters, and so on. Things we often forget about when writing example based tests.
Property tests were originally conceived in frameworks like Quickcheck with the notion of testing a property on some object, something that should hold true for all inputs. An example is the length of string A plus the length of string B should always be equal to the length of A + B.
This is where the term property testing originates.
Kotest supports this through the io.kotest.property.forAll function which accepts an n-arity function (a, ..., n) -> Boolean that tests the property.
For example, here is the property test that we mentioned just a few paragraphs ago. It checks that for any two Strings, the length of a + b
is the same as the length of a plus the length of b. In this example Kotest would
execute the test 1000 times for random String combinations.
class PropertyExample: StringSpec({
"String size" {
forAll<String, String> { a, b ->
(a + b).length == a.length + b.length
}
}
})Notice that the function must evaluate to a boolean value. We provide the type parameters to forAll so the framework knows which type of values to generate (in this case strings).
If we don't want to provide a property that returns a boolean, Kotest also provides for io.kotest.property.checkAll which accepts an n-arity function (a, ..., n) -> Unit in which you can simply execute assertions against the inputs. For example:
class PropertyExample: StringSpec({
"integers under addition should have an identity value" {
checkAll<Int, Int, Int> { a, b, c ->
a + 0 shouldbe a
0 + a shouldBe a
}
}
})The checkAll approach will consider a test valid if no exceptions were thrown.
By default, Kotest will run the property test 1000 times. We can easily customize this by specifying the iteration count when invoking the test method.
Let's say we want to run a test 10,000 times.
class PropertyExample: StringSpec({
"some test" {
checkAll<Double, Double>(10000) { a, b ->
// test here
}
}
})Kotest provides for the ability to specify some configuration options when running a property test. We do this by passing
in an instance of PropTestConfig to the test methods.
For example:
class PropertyExample: StringSpec({
"String size" {
forAll<String, String>(PropTestConfig(options here...)) { a,b ->
(a + b).length == a.length + b.length
}
}
})The most common configuration option is specifying the seed for the random instance. This is used when you want to reliably create the same values each time the test is run. You might want to do this if you find a test failure, and you want to ensure that that particular set of values continues to be executed in the future as a kind of regression test.
Note: Whenever a property test fails, Kotest will output the seed that was used, so you can copy it into another test to "fix" that seed value.
For example:
class PropertyExample: StringSpec({
"String size" {
forAll<String, String>(PropTestConfig(seed = 127305235)) { a,b ->
(a + b).length == a.length + b.length
}
}
})By default, Kotest tolerates no failure. Perhaps you want to run some non-deterministic test a bunch of times, and you're happy to accept some small number of failures. You can specify that in config.
class PropertyExample: StringSpec({
"some flakey test" {
forAll<String, String>(PropTestConfig(maxFailure = 3)) { a,b ->
// max of 3 inputs can fail
}
}
})Generated values are provided by instances of the sealed class Gen. You can think of a Gen as kind of like an input stream but for property testing.
Each Gen will provide a (usually) infinite stream of these values.
Kotest has two types of generators - Arb for arbitrary (random) values and Exhaustive for a finite set of values in a closed space.
Both types of gens can be mixed and matched in property tests. For example, you could test a function with 100 random positive integers (arbitrary) alongside every even number from 0 to 200 (exhaustive).
Some generators are only available on the JVM. See the full list here.
Arbs generate random values across a given space. The values may be repeated, and some values may never be generated at all. For example generating 1000 random integers between 0 and Int.MAX will clearly not return all possible values, and some values may happen to be generated more than once.
An arb will generate an infinite stream of values.
Typical arbs include numbers across a wide number line, strings in the unicode set, random lists, random data classes, emails, codepoint and chars.
Exhaustives generate all values from a given space. This is useful when you want to ensure every value in that space is used. For instance for enum values, it is usually more helpful to ensure each enum is used, rather than picking randomly from the enums values and potentially missing some and duplicating others.
Typical exhaustives include small collections, enums, boolean values, powerset of a list or set, pre-defined small integer ranges, and predefined string ranges.
You saw earlier when using forAll or checkAll that if we specify the type parameters, Kotest will provide an appropriate gen.
This is fine for basic tests but often we want more control over the sample space.
To do this, we can instantiate the generators ourselves by using extension functions on Arb and/or Exhaustive and passing
those into the assert/check methods.
For example, we may want to test a function for numbers in a certain range only.
class PropertyExample: StringSpec({
"is allowed to drink in Chicago" {
forAll(Arb.int(21..150)) { a ->
isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
}
}
"is allowed to drink in London" {
forAll(Arb.int(18..150)) { a ->
isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
}
}
})Actually, ages are a small space, it would probably be better not to leave the values to chance.
class PropertyExample: StringSpec({
"is allowed to drink in Chicago" {
forAll(Exhaustive.int(21..150)) { a ->
isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
}
}
"is allowed to drink in London" {
forAll(Exhaustive.int(18..150)) { a ->
isDrinkingAge(a) // assuming some function that calculates if we're old enough to drink
}
}
})You can mix and match arbs and exhaustives in the same test of course, since they are both generators.
class PropertyExample: StringSpec({
"some dummy test" {
checkAll(Arb.emails(), Exhaustive.enum<Foo>) { email, foo ->
// test here
}
}
})See here for a list of the built in generators.
To write your own generator for a type T, you just create an instance of Arb<T> or Exhaustive<T>.
When writing a custom arb we can use the arb builder which accepts a lambda that must return a sequence of the type we are generating for.
The parameter to this lambda is a RandomSource parameter which contains the seed and the Random instance. We should typically
use the provided RandomSource if we need access to a kotlin.Random instance, as this instance will have been seeded by the framework to allow for repeatable tests.
For example, here is a custom arb that random generates an int between 3 and 6 using the arb builder. When using the arb builder
we. We can do setup code in the outer function if required.
val sillyArb = arb { rs ->
generateSequence {
rs.random.nextInt(3..6)
}
}We can also use this random if we are composing other arbs when building ours.
For example, here is an Arb that supports a custom class called Person, delegating to a String arb and an Int arb.
data class Person(val name: String, val age: Int)
val personArb = arb { rs ->
val names = Arb.string().values(rs)
val ages = Arb.int().values(rs)
names.zip(ages).map { (name, age) -> Person(name.value, age.value) }
}Although in reality this Arb could have been easier written using bind, it demonstrates the principal.
When writing a custom exhaustive we can use the .exhaustive() extension function on a List. Nothing more to it than that really!.
val singleDigitPrimes = listOf(2,3,5,7).exhaustive()class PropertyExample: StringSpec({
"testing single digit primes" {
checkAll(singleDigitPrimes) { prime ->
isPrime(prime) shouldBe true
isPrime(prime * prime) shouldBe false
}
}
})Two generates can be merged together, so that elements 0, 2, 4, ... are taken from the first generator, and elements 1, 3, 5, ... are taken from the second generator.
val merged = arbA.merge(arbB)If you want to use an Arb to just return a value (even outside of a property test), then you can call next on it.
val arbA: Arb<A> = ...
val a = arbA.next() // use Random.Default
val a2 = arbA.next(rs) // pass in RandomIf you have an arb and you want to create a new arb that provides a subset of values, you can call filter on the source arb. For example, one way of generating even numbers is to take the integer arb, and filter out odd values. Viz:
val evens = Arb.int().filter { it.value % 2 == 0 }
val odds = Arb.int().filter { it.value % 2 == 1 }