CHANGELOG.md

vaex 4.0.0 (unreleased)

• Breaking changes:
  • Arrow is now a core dependency, vaex-arrow is deprecated. All methods that return string, will return Arrow arrays #517
  • Opening an .arrow file will expose the arrays as Apache Arrow arrays, not numpy arrays. #984
  • Columns (e.g. df.column['x']) may now return a ColumnProxy, instead of the original data, slice it [:] to get the underlying data (or call .to_numpy()/to_arrow() or try converting it with np.array(..) or pa.array(..)). #993
  • All plot methods went into the df.viz accesssor #923

vaex-arrow (DEPRECATED)

This is now part of vaex-core.

vaex-astro 0.8.0 (unreleased)

• Requirement changed to vaex-core >=4,<5

vaex-core 4.0.0 (unreleased)

• Fixes
  • Repeated dropna/dropnan/dropmissing could report cached length. #874
  • Trimming concatenated columns. #860
  • percentile_approx works for 0 and 100 percentile. #818
  • Expression containing kwarg=True were treated as invalid. #861
  • Unicode column names fully supported #974
• Features
  • Datetime floor method #843
  • dropinf (similar to dropna) #821
  • Support for streaming from Google Cloud Storage. #898
  • IPython autocomplete support (e.g. df['hom' (tab)) #961
  • Out of core Parquet support using Arrow Dataset scanning #993
• Refactor
  • Use arrow.compute for several string functions/kernels. #885
  • Separate DataFrame and Dataset. #865
• Performance
  • concat (vaex.concat or df.concat) is about 100x faster. #994

vaex-distributed (DEPRECATED)

This is now part of vaex-enterprise (was a proof of content, never functional).

vaex-graphql 0.2.0 (unreleased)

• Requirement changed to vaex-core >=4,<5

vaex-hdf5 0.7.0 (unreleased)

• Requirement changed vaex-core >=4,<5

vaex-jupyter 0.6.0 (unreleased)

• Requirement changed to vaex-core >=4,<5

vaex-ml 0.11.0 (unreleased)

• Features
  • Batch training for CatBoost. #819
  • Support for predict_proba and predict_log_proba for sklearn classifiers. #927

vaex-server 0.4.0 (unreleased)

• Requirement changed to vaex-core >=4,<5

vaex-viz 0.5.0 (unreleased)

• Requirement changed to vaex-core >=4,<5

vaex 3.1.0 (unreleased)

vaex-jupyter 0.5.2 (2020-6-12)

• Features
  • Normalize histogram and change selection mode. #826

vaex-ml 0.11.0-dev0 (unreleased)

* Features
  * Autogenerate the fast (or functional) API [#512](https://github.com/vaexio/vaex/pull/512)

vaex-core 2.0.3 (2020-6-10)

• Performance
  • isin uses hashmaps, leading to a 2x-4x performance increase for primitives, 200x for strings in some cases #822

vaex-jupyter 0.5.1 (2020-6-4)

• Features
  • Selection toggle list. #797

vaex-server 0.3.1 (2020-6-4)

• Fixes
  • Remote dataframe was still using dtype, not data_type. #797

vaex-ml 0.10.0 (2020-6-4)

• Features
  • Implementation of GroupbyTransformer #479

vaex-arrow 0.6.1 (2020-6-4)

• Fixes
  • Various fixes for aliased columns (column names with invalid identifiers) #768

vaex-hdf5 0.6.1 (2020-6-4)

• Fixes
  • Masked arrays supported in hdf5 files on s3 #781
  • Various fixes for aliased columns (column names with invalid identifiers) #768

vaex-core 2.0.2 (2020-6-4)

• Fixes
  • Masked arrays supported in hdf5 files on s3 #781
  • Expression.map always uses masked arrays to be state transferrable (a new dataset might have missing values) #479
  • Support importing Pandas dataframes with version 0.23 #794
  • Various fixes for aliased columns (column names with invalid identifiers) #768 #793

vaex-core 2.0.1 (2020-5-28)

• Fixes
  • Join could in rare cases point to row 0, when there were values in the left, not present in the right #765
  • Tabulate 0.8.7 escaped html, undo this to print dataframes nicely.

vaex 3.0.0 (2020-5-24)

• Breaking changes:
  • Python 2 is not supported anymore
  • Variables don't have access to pi and e anymore
  • df.rename_column is now df.rename (and also renames variables)
  • DataFrame uses a normal dict instead of OrderedDict, requiring Python >= 3.6
  • Default limits (e.g. for plots) is minmax, so we don't miss outliers
  • df.get_column_names() returns the aliased names (invalid identifiers), pass alias=False to get the internal column name
  • Default value of virtual is True in method df.export, df.to_dict, df.to_items, df.to_arrays.
  • df.dtype is a property, to get data types for expressions, use df.data_type(), df.expr.dtype is still behaving the same
  • df.categorize takes min_value and max_value, and no longer needs the check argument, also the labels do not have to be strings.
  • vaex.open/from_csv etc does not copy the pandas index by default #756
  • df.categorize takes an inplace argument, similar to most methods, and returns the dataframe affected.

vaex-core 2.0.0 (2020-5-24)

• Performance

  • Printing out of dataframes done in 1 evaluate call, making remote dataframe printing faster. #571
  • Joining is faster and uses less memory (2x speedup measured) #586
  • Faster typechecks when adding columns of dtype=object (as often happens when coming from pandas) #612
  • Groupby 2x to 4x faster #730

• Refactor

  • Task system is refactored, with task execution on CPU being default, and makes (de)serialization easier. #571
  • Serialization/encoding of data structures is more flexible, allowing binary blobs and json over the wire. #571
  • Execution and tasks support async await #654

• Fixes

  • Renaming columns fixes #571
  • Joining with virtual columns but different data, and name collision fixes #570
  • Variables are treated similarly as columns, and respected in join #573
  • Arguments to lazy function which are numpy arrays gets put in the variables #573
  • Executor does not block after failed/interrupted tasks. #571
  • Default limits (e.g. for plots) is minmax, so we don't miss outliers #581
  • Do no fail printing out dataframe with 0 rows #582
  • Give proper NameError when using non-existing column names #299
  • Several fixes for concatenated dataframes. #590
  • dropna/nan/missing only dropped rows when all column values were missing, if no columns were specified. #600
  • Flaky test for RobustScaler skipped for p36 #614
  • Copying/printing sparse matrices #615
  • Sparse columns names with invalid identifiers are not rewritten. #617
  • Column names with invalid identifiers which are rewritten are shown when printing the dataframe. #617
  • Column name rewriting for invalid identifiers also works on virtual columns. #617
  • Fix the links to the example datasets. #609
  • Expression.isin supports dtype=object #669
  • Fix colum_count, now only counts hidden columns if expicitly specified #593
  • df.values respects masked arrays #640
  • Rewriting a virtual column and doing a state transfer does not lead to ValueError: list.remove(x): x not in list #592
  • df.<stat>(limits=...) will now respect the selection #651
  • Using automatic names for aggregators led to many underscores in name #687
  • Support Python3.8 #559

• Features

  • New lazy numpy wrappers: np.digitize and np.searchsorted #573
  • df.to_arrow_table/to_pandas_df/to_items/df.to_dict/df.to_arrays now take a chunk_size argument for chunked iterators #589 (vaexio#699)
  • Filtered datasets can be concatenated. #590
  • DataFrames/Executors are thread safe (meaning you can schedule/compute from any thread), which makes it work out of the box for Dash and Flask #670
  • df.count/mean/std etc can output in xarray.DataArray array type, makes plotting easier #671
  • Column names can have unicode, and we use str.isidentifier to test, also dont accidently hide columns. #617
  • Percentile approx can take a sequence of percentages #527
  • Polygon testing, useful in combinations with geo/geojson data #685
  • Added dt.quarter property and dt.strftime method to expression (by Juho Lauri) #682

vaex-server 0.3.0 (2020-5-24)

• Refactored server, can return multiple binary blobs, execute multiple tasks, cancel tasks, encoding/serialization is more flexible (like returning masked arrays). #571

vaex-viz 0.4.0 (2020-5-24)

• Requirement of vaex-core >=2,<3

vaex-graphql 0.1.0 (2020-5-24)

• Requirement of vaex-core >=2,<3

vaex-astro 0.7.0 (2020-5-24)

• Requirement of vaex-core >=2,<3

vaex-hdf5 0.6.0 (2020-5-24)

• Requirement of vaex-core >=2,<3

vaex-ml 0.9.0 (2020-5-24)

• Requirement of vaex-core >=2,<3

vaex-arrow 0.5.0 (2020-5-24)

• Requirement of vaex-core >=2,<3
• Fixes
  • Booleans were negated, and didn't respect offsets.

vaex-jupyter 0.5.0 (2020-5-24)

• Requirement of vaex-core >=2,<3
• Breaking changes
  • vaex-jupyter is refactored #654

vaex 2.6.0 (2020-1-21)

vaex-core 1.5.0

• Features
  • df.evalute_iterator for efficient parallel chunked evaluation #515
  • Widget progress bar has time estimation #545
• Fixes
  • Slicing arrow string arrays with masked arrays is respected/working #530]

vaex-ml 0.8.0

• Performance
  • IncrementalPredictor uses parallel chunked support (2x speedup possible) #515
• Fix
  • IncrementalPredictor: epochs now iterate over the whole DataFrame instead on a batch level #523
  • Rename vaex.ml.sklearn.SKLearnPredictor to vaex.ml.sklearn.Predictor #524
  • IncrementalPredictor can be used with sklearn.linear_model.SGDClassifier 539
• Features
  • CycleTransformer #532
  • BayesianTargetEncoder #533
  • WeightOfEvidenceEncoder #534
  • Improve the consistency of the vaex.ml API for model transformers #552

vaex 2.5.0 (2019-12-16)

vaex-core 1.4.0

• Performance
  • Dataframes are always true (implements __bool__) to avoid calling __len__ #496
• Fixes
  • Do not duplicate column when joining DataFrames on a column with the same name #480
  • Better error messages/stack traces, and work better with debugger. #488
  • Accept numpy scalars in expressions. #462
  • Expression.astype can create datetime64 columns out of (arrow) strings arrays. #440
  • Invalid mask access triggered when memory-mapped read only for strings. #459
• Features
  • Expressions are not evaluated for filtered data #483 #496 and selections 505
  • Filtering (using df.filter) allows more flexible (and growing/expanding!) filter. #489
  • Filtering and selections allow for booleans (True or False) to auto 'broadcast', to allow 'conditional filtering'. #489

vaex-ml 0.7.0

• Features
  • IncrementalPredictor for scikit-learn models that support the .partial_fit method #497
• Fixes
  • Adding unique function names to dataframes to enable adding a predictor twice #492

vaex-arrow 0.4.2

  * Compatibility with vaex-core 1.4.0

vaex 2.4.0 (2019-11-26)

vaex-core 1.3.0

• Performance
  • Parallel df.evaluate #474
  • Avoid calling df.get_column_names (1000x for 1 billion rows per column use) #473
  • Slicing e.g df[1:-1] goes much faster for filtered dataframes #471
  • Dataframe copying and expression rewriting was slow #470
  • Double indices columns were not using index cache since empty dict is falsy #439
• Features
  • multi-key sorting of a DataFrame #463
  • vaex expression to pandas.Series support #456
  • Dask array support #449 #476 example
  • isin() method for expressions #441 docs
  • Existing expressions are rewritten, to make them behave like arrays #450

vaex-hdf5 0.5.6

• requires vaex-core >=1.3,<2 for parallel evaluate

vaex-jupyter 0.4.1

• Fixes:
  • bqplot 0.12 revealed a bug/inconsistency with heatmap #465

vaex-arrow 0.4.1

• Fixes
  • Support for Apache Arrow >= 0.15

vaex-ml 0.6.2

• Fixes
  • Docstrings and minor improvements

vaex-graphql 0.0.1 (2019-10-15)

• initial release 0.1

vaex 2.3.0 (2019-10-15)

vaex-core 1.2.0

• feature: auto upcasting for sum #435
• fix: selection/filtering fix when using masked values #431
• fix: masked string array fixes #434
• fix: memory usage fix for joins #439

vaex-arrow 0.4.1

• fix: support for Apache Arrow >= 0.15