Update to Scalatest 3.1.x

[nightscape]

The update is a 2-step process where you first update to 3.0.8 and use Scalafix to fix deprecations.
Then you repeat the same process with 3.1.x.
I've made 2 commits out of those but you can also squash them into one if you don't care about the intermediate step.

[bluesheeptoken]

@holdenk Hello, I am mentioning you here to hopefully draw your attention.

Is there any interest in merging this PR ? This would be helpful to bump scalatest in different projects (as this is a blocker).
Please let me know if there is anything I can do to help.

All the best

[MrPowers]

@nightscape - Your PR looks good, but it might be best to remove the Scalatest dependency from spark-testing-base altogether. As you know, Scalatest isn't just a test dependency, it's a project dependency.

Removing the Scalatest dependency would allow spark-testing-base to be used with other testing frameworks like munit and utest. It'll also prevent the cross compilation matrix from getting too complicated... not sure we want to cross compile with Scala version / Spark version / Scalatest version.

Removing the Scalatest dependency should be relatively easy and could be done in a backwards compatible manner. Let me know your thoughts!

[nightscape]

Hey @MrPowers, good idea!
I would actually like to switch from ScalaTest + ScalaCheck to Zio Test and have used your spark-fast-test library to replace some parts of spark-testing-base.
I unfortunately won't have time for this anytime soon, so in the mean time it would still be useful to get this merged.
Last time I've heard from @holdenk though she had had a motorbike accident and I'm not sure if she has recovered yet...

[MrPowers]

@holdenk - hope you're recovering well. Really sorry to hear about the motorbike accident.

@nightscape - Would this PR make spark-testing-base unusable for Scalatest 2.x users? I am assuming that import org.scalatest.funsuite.AnyFunSuite would error out for folks that are using Scalatest 2.x...

It's unfortunate Scalatest made these breaking changes.

It's really hard to maintain a Spark library with all these Scalatest / Spark / Scala version moving parts. Big shout out to @holdenk for maintaining this lib for all these years!

[bluesheeptoken]

@holdenk thanks for your hard work and I hope you are getting better.

To what I understand, you are not really in favor of the merge of this PR as is due to compatibilities breaks ?

I am not sure if this is a bad idea to merge this PR as is. Since it would encourage people to bump their scalatest versions due to compatibility.

How would you remove the scalatest dependency @MrPowers ?

As a side note : 3.1.X is compatible with 2.X, 3.0.X and 3.2.X. This might be worth to merge this, release and open a ticket to remove scalatest dependency ? WDYT ?

[MrPowers]

Here's a code snippet that is extending from a scalatest trait.

trait DataFrameSuiteBase extends TestSuite
    with SharedSparkContext with DataFrameSuiteBaseLike { self: Suite =>

The main reason to extend Suite is to add some custom assertions.

We'd have to reimplement methods like assertEmpty that are used here.

assertEmpty wouldn't be too hard to reimplement, you could just check to make sure the RDD is empty and throw an error if it's not. This would touch a lot of the code, so it'd be a major refactoring. Think we'd need to get Holden's blessing before doing any work. If Holden wants to do this refactoring, I am definitely happy to help out!

[nightscape]

I just rebased this commit.
@MrPowers do you think people have upgraded ScalaTest in the mean time?
Unfortunately I never found the time to do more serious work in the direction of making ScalaTest optional.
I still think it would be good to get at least some updates in.

[vimal3271]

@holdenk Could you please merge this PR ? We are not able to use spark-testing base for 3.2.1 upgrade. I saw your release version 1.1.2 but it is still failing with 3.2.1

[vimal3271]

#354 - We have created the issue here.

[holdenk]

Looks good, I'll probably have to do a larger version bump for the dep change.

[vimal3271]

@holdenk thanks for merging this. Could you please help in releasing the jar ?

[holdenk]

Should be available as 1.2.0

[vimal3271]

Thanks @holdenk . https://github.com/holdenk/spark-testing-base/releases - Release is not updated here. It is available in mvn repo.