Scala spark-shell mode

1 Motivation

Spark-shell is a good place to develop and test spark / spark sql code interactively. As a data scientist / engineer, I wish I have a editor support for that. I hope the editor would looks like RStudio / emacs ESS / ipython notebook. Not able to find an available emacs binding, here I made a slightly change to scala-mode (from https://github.com/scala/scala-tool-support) to support interactive spark script development with spark-shell.

2 Features

2.1 scala-mode for spark-shell

• Features from scala-mode like keyword highlight etc
• Interactive spark script development, with ess / R-mode like key-binding

2.2 literature programming support with org-mode

• Can run code block and print result to org document as an org-babel plugin

3 Install / configuration

Requirement:

• emacs 24.5 and org-mode version >= 8.2.10
• it is conflict with other scala-mode, probably need to remove them, sadly

3.1 Basic support

You can git clone this repo, and put the following code to your .emacs to load them

(progn
  (cd "<<path-to-scala-spark-shell-mode-elisp-dir>>")
  (normal-top-level-add-subdirs-to-load-path))

(require 'scala-mode)

3.1.1 Modify spark-shell launch command

You can modify the spark-shell to some other thing to load your own jar and add your own start configuration like –num-executors, etc by redefine scala-interpreter

(defcustom scala-interpreter "spark-shell"
  "The interpreter that `run-scala' should run. This should
 be a program in your PATH or the full pathname of the scala interpreter."
  :type 'string
  :group 'scala-mode-inf)

In a file buffer with .scala or .spark suffix, M-x scala-run-scala will open an spark-shell buffer.

3.1.2 Send code and eval them in the spark-shell buffer

• Line / block / region / buffer evaluation.
• The key-binding is ess / R-mode flavor

(progn (define-key scala-mode-map "\C-c\C-i" 'scala-run-scala)
       (define-key scala-mode-map "\C-c\C-r" 'scala-eval-region)
       (define-key scala-mode-map "\C-c\C-j" 'scala-eval-line)
       (define-key scala-mode-map "\C-c\C-c" 'scala-eval-block)
       (define-key scala-mode-map "\C-c\C-b" 'scala-eval-buffer)
       (define-key scala-mode-map "\C-c\C-d" 'scala-eval-definition)
       (define-key scala-mode-map "\C-c\C-z" 'scala-switch-to-interpreter))

3.2 Literature programming setup

Add the spark-shell language support to your org-babel-load-languages list, like this:

(org-babel-do-load-languages
 'org-babel-load-languages
 '((R . t)
   (spark-shell . t)
   (python . t)
   (org . t))
)

Here goes an spark-shell literature programming example.

4 TODO Future Tasks [0/2]

4.1 TODO [#A] Figure out a way to clean up the output of spark-shell (executed results kept there)

4.2 TODO [#B] Add code to handle output of spark-shell code chunk into a org-table