Issue/237

[NickEdwards7502]

Major issues and features addressed in this update

• VariantSpark's python wrapper has been refactored to create Random Forest models from a standalone class

  • Previously, in the non-hail VariantSpark release, the model was initialised and trained from the context of importance analyses. This did not seem appropriate for supporting future releases
  • A new scala function was created to return a trained RandomForest model without using hail
  • Files updated/created
    • python/varspark/rfmodel.py
    • python/varspark/core.py
    • python/varspark/__init__.py
    • src/main/scala/au/csiro/variantspark/api/GetRFModel.scala

• A non-hail export model function was created

  • This function now processes trees in batches to remediate OOM errors with very large models
  • Files created
    • src/main/scala/au/csiro/variantspark/api/ExportModel.scala

• The FeatureSource class, which provides wrapper functionalities for initialising genotype data for model training, has been moved to a standalone class

  • For better separation of concerns, this class is now imported to the core python wrapper
  • head(nrows, ncols) allows the first n rows and columns to be viewed as a pandas DataFrame
  • Files updated/created
    • python/varspark/featuresource.py
    • python/varspark/core.py
    • src/main/scala/au/csiro/variantspark/input/FeatureSource.scala

• Covariate support was extended

  • Covariate sources can be created from a transposed/non-transposed .csv or .txt files with optional per-feature type specification
  • From there, covariates can be unioned with a genotype FeatureSource and passed to the model as training data
  • Since Covariates are initialised from the same FeatureSource wrapper class and are also of type RDD[Feature], they also support head()
  • Note that feature and covariate sources can be unioned multiple times
  • Local FDR was updated to remove non-genotype information from manhattan plotting
  • Files updated/created
    • `python/varspark/core.py
    • src/main/scala/au/csiro/variantspark/api/VSContext.scala
    • src/main/scala/au/csiro/variantspark/input/CsvStdFeatureSource.scala
    • src/main/scala/au/csiro/variantspark/input/UnionedFeatureSource.scala
    • python/varspark/lfdrvsnohail.py

• Importance analyses were moved to a standalone python wrapper class

  • Importance analyses are now created from the context of a random forest model
  • Functionality remains largely the same, with a few changes
    • Both important_variables() and variable_importance() are now returned as pandas DataFrames
    • Split counts are now included in the DataFrame returned by variable_importance() (required for Local FDR calculations)
    • Optional parameter precision supports rounding for variable_importance()
    • Optional parameter normalized indicates whether to normalise importances for both functions
  • Files updated/created
    • python/varspark/importanceanalysis.py
    • python/varspark/core.py
    • src/main/scala/au/csiro/variantspark/api/ImportanceAnalysis.scala
    • src/main/scala/au/csiro/variantspark/api/AnalyticsFunctions.scala

• Move lfdr file to non-hail python directory

  • Created function for manhattan plotting lfdr derived p-values
  • Files removed/created
    • python/varspark/hail/lfdrvs.py
    • python/varspark/lfdrvs.py

• Updated all test cases according to the above changes

  • Files updated/removed/created
    • src/test/scala/au/csiro/variantspark/api
      • /CommonPairwiseOperationTest.scala
      • /ImportanceApiTest.scala
    • src/test/scala/au/csiro/variantspark/misc
      • /ReproducibilityTest.scala
      • /CovariateReproducibilityTest.scala
    • src/test/scala/au/csiro/variantspark/test
      • /TestSparkContext.scala
    • python/varspark/test
      • /test_core.py
      • /test_hail.py
      • /test_pvalues_calculation.py
    • src/test/scala/au/csiro/variantspark/work/hail
      • /HailApiApp.scala

• Removed all files used exclusively in hail version

  • python/varspark/hail
    • __init__.py
    • context.py
    • hail.py
    • methods.py
    • plot.py
  • src/main/scala/au/csiro/variantspark/hail/methods
    • RFModel.scala

• Removed hail installation from pom.xml