ARROW-1990: [JS] C++ Refactor, Add DataFrame

This PR moves the `Table` class out of the Vector hierarchy and adds optimized dataframe operations to it. Currently implements an optimized `scan()` method, `filter(predicate)`, `count()`, and `countBy(column_name)` (only works on dictionary-encoded columns).

Some usage examples, based on the file generated by `js/test/data/tables/generate.py`:
``` js
> let table = Table.from(...);
> table.count()
1000000
> table.filter(col('lat').gteq(0)).count()
499718
> table.countBy('origin').toJSON()
{ Charlottesville: 166839,
  'New York': 166251,
  'San Francisco': 166642,
  Seattle: 166659,
  'Terre Haute': 166756,
  'Washington, DC': 166853 }
> table.filter(col('lng').gteq(0)).countBy('origin').toJSON()
{ Charlottesville: 83109,
  'New York': 83221,
  'San Francisco': 83515,
  Seattle: 83362,
  'Terre Haute': 83314,
  'Washington, DC': 83479 }
```
There are performance tests for the dataframe operations, to run them you must first generate the test data by running `npm run create:perfdata`.

The PR also includes @trxcllnt's refactor of the JS implementation to make it more closely resemble the C++ implementation. This refactor resolves multiple JIRAs: ARROW-1903, ARROW-1898, ARROW-1502, ARROW-1952 (partially), and ARROW-1985

Author: Paul Taylor <[email protected]>
Author: Brian Hulette <[email protected]>
Author: Brian Hulette <[email protected]>

Closes apache#1482 from TheNeuralBit/table-scan-perf and squashes the following commits:

52f1e0e [Brian Hulette] <, > are not commutative, misc cleanup
04b1838 [Brian Hulette] even more table tests
16b9ccb [Brian Hulette] Merge pull request #4 from trxcllnt/js-cpp-refactor
fe300df [Paul Taylor] fix closure es5/umd toString() iterator
3d5240a [Paul Taylor] fix more externs
10c48ad [Paul Taylor] Merge branch 'table-scan-perf' of github.com:ccri/arrow into js-cpp-refactor
dbe7f81 [Brian Hulette] Add more Table unit tests
1910962 [Brian Hulette] Add optional bind callback to scan
5bdf17f [Brian Hulette] Fix perf
8cf2473 [Brian Hulette] Merge remote-tracking branch 'origin/master' into table-scan-perf
4a41b18 [Paul Taylor] add src/predicate to the list of exports we should save from uglify
5a91fab [Paul Taylor] add more view, predicate externs
f6adfb3 [Brian Hulette] Create predicate namespace
f7bb0ed [Paul Taylor] Merge branch 'table-scan-perf' of github.com:ccri/arrow into js-cpp-refactor
e148ee4 [Paul Taylor] Merge branch 'extern-woes' into js-cpp-refactor
25cdc4a [Paul Taylor] add src/predicate to the list of exports we should save from uglify
dc7c728 [Paul Taylor] add more view, predicate externs
25e6af7 [Brian Hulette] Create predicate namespace
579ab1f [Brian Hulette] Merge pull request #2 from trxcllnt/js-cpp-refactor
f3cde1a [Paul Taylor] fix lint
9769773 [Paul Taylor] fix vector perf tests
016ba78 [Brian Hulette] Merge pull request #1 from trxcllnt/js-cpp-refactor
272d293 [Paul Taylor] Merge pull request #4 from ccri/empty-table
7bc7363 [Brian Hulette] Fix exception for empty Table
8ddce0a [Paul Taylor] check bounds in getChildAt(i) to avoid NPEs
f1dead0 [Paul Taylor] compute chunked nested childData list correctly
18807c6 [Paul Taylor] rename ChunkData's fields so it's more clear they're not semantically similar to other similarly named fields
7e43b78 [Paul Taylor] add test:integration npm script
a5f200f [Paul Taylor] Merge pull request #3 from ccri/table-from-struct
c8cd286 [Brian Hulette] Add Table.fromStruct
a00415e [Brian Hulette] Fix perf
54d4f5b [Paul Taylor] lazily allocate table and recordbatch columns, support NestedView's getChildAt(i) method in ChunkedView
40b3638 [Paul Taylor] run integration tests with local data for coverage stats
fe31ee0 [Paul Taylor] slice the flat data values before returning an iterator of them
e537789 [Paul Taylor] make it easier to run all integration tests from local data
c0fd2f9 [Paul Taylor] use the dictionary of the last chunked vector list for chunked dictionary vectors
e33c068 [Paul Taylor] Merge pull request #2 from ccri/fixed-size-list
5bb63af [Brian Hulette] Don't read OFFSET vector for FixedSizeList
614b688 [Paul Taylor] add asEpochMs to date and timestamp vectors
87334a5 [Paul Taylor] Merge branch 'table-scan-perf' of github.com:ccri/arrow into js-cpp-refactor
b7f5bfb [Paul Taylor] rename numRows to length, add table.getColumn()
e81082f [Paul Taylor] export vector views, allow cloning data as another type
700a47c [Paul Taylor] export visitors
e859e13 [Paul Taylor] fix package.json bin entry
0620cfd [Brian Hulette] use Math.fround
0126dc4 [Brian Hulette] Don't recompute total length
e761eee [Brian Hulette] Rename asJSON to toJSON
6c91ed4 [Paul Taylor] Merge branch 'master' of github.com:apache/arrow into js-cpp-refactor-merge_with-table-scan-perf
d2b18d5 [Paul Taylor] Merge remote-tracking branch 'ccri/table-scan-perf' into js-cpp-refactor-merge_with-table-scan-perf
f3f3b86 [Paul Taylor] rename table.ts to recordbatch.ts in preparation for merging latest
e3f629d [Paul Taylor] fix rest of the mangling issues
fa7c17a [Paul Taylor] passing all tests except es5 umd mangler ones
e20decd [Brian Hulette] Add license headers
edcbdbe [Brian Hulette] cleanup
20717d5 [Brian Hulette] Fixed countBy(string)
7244887 [Brian Hulette] Add table unit tests...
6719147 [Brian Hulette] Add DataFrame.countBy operation
2f4a349 [Brian Hulette] Minor tweaks
2e118ab [Brian Hulette] linter
a788db3 [Brian Hulette] Cleanup
a9fff89 [Brian Hulette] Move Table out of the Vector hierarchy
1d60aa1 [Brian Hulette] Moved DataFrame ops to Table. DataFrame is now an interface
e8979ba [Brian Hulette] Refactor DataFrame to extend Vector<StructRow>
6a41d68 [Brian Hulette] clean up table benchmarks
2744c63 [Brian Hulette] Remove Chunked/Simple DataFrame distinction
aa999f8 [Brian Hulette] Add DictionaryVector optimization for equals predicate
4d9e8c0 [Brian Hulette] Add concept of predicates for filtering dataframes
796f45d [Brian Hulette] add DataFrame filter and count ops
30f0330 [Brian Hulette] Add basic DataFrame impl ...
a1edac2 [Brian Hulette] Add perf tests for table scans
d18d915 [Paul Taylor] fix struct and map rows
61dc699 [Paul Taylor] WIP -- refactor types to closer match arrow-cpp
62db338 [Paul Taylor] update dependencies and add es6+ umd targets to jest transform ignore patterns to fix ci
6ff18e9 [Paul Taylor] ship es2015 commonJS in main package to avoid confusion
74e828a [Paul Taylor] fix typings issues (ARROW-1903)

js/bin/integration.js

@@ -17,6 +17,8 @@
 // specific language governing permissions and limitations
 // under the License.
 
+var fs = require('fs');
+var glob = require('glob');
 var path = require('path');
 var gulp = require.resolve(path.join(`..`, `node_modules/gulp/bin/gulp.js`));
 var child_process = require(`child_process`);
@@ -29,12 +31,14 @@ var optionList = [
     {
         type: String,
         name: 'arrow', alias: 'a',
-        description: 'The Arrow file to read/write'
+        multiple: true, defaultValue: [],
+        description: 'The Arrow file[s] to read/write'
     },
     {
         type: String,
         name: 'json', alias: 'j',
-        description: 'The JSON file to read/write'
+        multiple: true, defaultValue: [],
+        description: 'The JSON file[s] to read/write'
     }
 ];
 
@@ -66,20 +70,60 @@ function print_usage() {
     process.exit(1);
 }
 
-if (!argv.arrow || !argv.json || !argv.mode) {
+let jsonPaths = argv.json;
+let arrowPaths = argv.arrow;
+
+if (!argv.mode) {
+    return print_usage();
+}
+
+let mode = argv.mode.toUpperCase();
+if (mode === 'VALIDATE' && !jsonPaths.length) {
+    jsonPaths = glob.sync(path.resolve(__dirname, `../test/data/json/`, `*.json`));
+    if (!arrowPaths.length) {
+        [jsonPaths, arrowPaths] = jsonPaths.reduce(([jsonPaths, arrowPaths], jsonPath) => {
+            const { name } = path.parse(jsonPath);
+            for (const source of ['cpp', 'java']) {
+                for (const format of ['file', 'stream']) {
+                    const arrowPath = path.resolve(__dirname, `../test/data/${source}/${format}/${name}.arrow`);
+                    if (fs.existsSync(arrowPath)) {
+                        jsonPaths.push(jsonPath);
+                        arrowPaths.push(arrowPath);
+                        console.log('-j', jsonPath, '-a', arrowPath, '\\');
+                    }
+                }
+            }
+            return [jsonPaths, arrowPaths];
+        }, [[], []]);
+    }
+} else if (!jsonPaths.length) {
     return print_usage();
 }
 
-switch (argv.mode.toUpperCase()) {
+switch (mode) {
     case 'VALIDATE':
+        const args = [`test`, `-i`].concat(argv._unknown || []);
+        jsonPaths.forEach((p, i) => {
+            args.push('-j', p, '-a', arrowPaths[i]);
+        });
         child_process.spawnSync(
-            gulp,
-            [`test`, `-i`].concat(process.argv.slice(2)),
+            gulp, args,
             {
                 cwd: path.resolve(__dirname, '..'),
                 stdio: ['ignore', 'inherit', 'inherit']
             }
         );
+        // for (let i = -1, n = jsonPaths.length; ++i < n;) {
+        //     const jsonPath = jsonPaths[i];
+        //     const arrowPath = arrowPaths[i];
+        //     child_process.spawnSync(
+        //         gulp, args.concat(['-j', jsonPath, '-a', arrowPath]),
+        //         {
+        //             cwd: path.resolve(__dirname, '..'),
+        //             stdio: ['ignore', 'inherit', 'inherit']
+        //         }
+        //     );
+        // }
         break;
     default:
         print_usage();

js/gulp/argv.js

@@ -15,20 +15,22 @@
 // specific language governing permissions and limitations
 // under the License.
 
+const fs = require('fs');
+const glob = require('glob');
+const path = require('path');
+
 const argv = require(`command-line-args`)([
     { name: `all`, type: Boolean },
     { name: 'update', alias: 'u', type: Boolean },
     { name: 'verbose', alias: 'v', type: Boolean },
     { name: `target`, type: String, defaultValue: `` },
     { name: `module`, type: String, defaultValue: `` },
     { name: `coverage`, type: Boolean, defaultValue: false },
-    { name: `json_file`, alias: `j`, type: String, defaultValue: null },
-    { name: `arrow_file`, alias: `a`, type: String, defaultValue: null },
     { name: `integration`, alias: `i`, type: Boolean, defaultValue: false },
     { name: `targets`, alias: `t`, type: String, multiple: true, defaultValue: [] },
     { name: `modules`, alias: `m`, type: String, multiple: true, defaultValue: [] },
-    { name: `sources`, alias: `s`, type: String, multiple: true, defaultValue: [`cpp`, `java`] },
-    { name: `formats`, alias: `f`, type: String, multiple: true, defaultValue: [`file`, `stream`] },
+    { name: `json_files`, alias: `j`, type: String, multiple: true, defaultValue: [] },
+    { name: `arrow_files`, alias: `a`, type: String, multiple: true, defaultValue: [] },
 ], { partial: true });
 
 const { targets, modules } = argv;
@@ -38,4 +40,25 @@ argv.module && !modules.length && modules.push(argv.module);
 (argv.all || !targets.length) && targets.push(`all`);
 (argv.all || !modules.length) && modules.push(`all`);
 
+if (argv.coverage && (!argv.json_files || !argv.json_files.length)) {
+
+    let [jsonPaths, arrowPaths] = glob
+        .sync(path.resolve(__dirname, `../test/data/json/`, `*.json`))
+        .reduce((paths, jsonPath) => {
+            const { name } = path.parse(jsonPath);
+            const [jsonPaths, arrowPaths] = paths;
+            ['cpp', 'java'].forEach((source) => ['file', 'stream'].forEach((format) => {
+                const arrowPath = path.resolve(__dirname, `../test/data/${source}/${format}/${name}.arrow`);
+                if (fs.existsSync(arrowPath)) {
+                    jsonPaths.push(jsonPath);
+                    arrowPaths.push(arrowPath);
+                }
+            }));
+            return paths;
+        }, [[], []]);
+
+    argv.json_files = jsonPaths;
+    argv.arrow_files = arrowPaths;
+}
+
 module.exports = { argv, targets, modules };

js/gulp/closure-task.js

@@ -36,7 +36,7 @@ const closureTask = ((cache) => memoizeTask(cache, function closure(target, form
     const src = targetDir(target, `cls`);
     const out = targetDir(target, format);
     const entry = path.join(src, mainExport);
-    const externs = path.join(src, `${mainExport}.externs`);
+    const externs = path.join(`src/Arrow.externs.js`);
     return observableFromStreams(
         gulp.src([
 /*   external libs first --> */ `node_modules/tslib/package.json`,
@@ -46,7 +46,6 @@ const closureTask = ((cache) => memoizeTask(cache, function closure(target, form
                                 `node_modules/text-encoding-utf-8/package.json`,
                                 `node_modules/text-encoding-utf-8/src/encoding.js`,
 /*    then sources globs --> */ `${src}/**/*.js`,
-/* and exclusions last -->  */ `!${src}/Arrow.externs.js`,
         ], { base: `./` }),
         sourcemaps.init(),
         closureCompiler(createClosureArgs(entry, externs)),
@@ -60,14 +59,15 @@ const closureTask = ((cache) => memoizeTask(cache, function closure(target, form
 }))({});
 
 const createClosureArgs = (entry, externs) => ({
+    externs,
     third_party: true,
     warning_level: `QUIET`,
     dependency_mode: `STRICT`,
     rewrite_polyfills: false,
-    externs: `${externs}.js`,
     entry_point: `${entry}.js`,
     module_resolution: `NODE`,
-    // formatting: `PRETTY_PRINT`, debug: true,
+    // formatting: `PRETTY_PRINT`,
+    // debug: true,
     compilation_level: `ADVANCED`,
     allow_method_call_decomposing: true,
     package_json_entry_names: `module,jsnext:main,main`,

js/gulp/package-task.js

@@ -45,10 +45,11 @@ const createMainPackageJson = (target, format) => (orig) => ({
     ...createTypeScriptPackageJson(target, format)(orig),
     name: npmPkgName,
     main: mainExport,
+    types: `${mainExport}.d.ts`,
     module: `${mainExport}.mjs`,
     dist: `${mainExport}.es5.min.js`,
     [`dist:es2015`]: `${mainExport}.es2015.min.js`,
-    [`@std/esm`]: { esm: `mjs` }
+    [`@std/esm`]: { esm: `mjs`, warnings: false, sourceMap: true }
 });
 
 const createTypeScriptPackageJson = (target, format) => (orig) => ({
@@ -63,18 +64,20 @@ const createTypeScriptPackageJson = (target, format) => (orig) => ({
 
 const createScopedPackageJSON = (target, format) => (({ name, ...orig }) =>
     conditionallyAddStandardESMEntry(target, format)(
-      packageJSONFields.reduce(
-        (xs, key) => ({ ...xs, [key]: xs[key] || orig[key] }),
-        { name: `${npmOrgName}/${packageName(target, format)}`,
-          version: undefined, main: `${mainExport}.js`, types: `${mainExport}.d.ts`,
-          dist: undefined, [`dist:es2015`]: undefined, module: undefined, [`@std/esm`]: undefined }
-      )
+        packageJSONFields.reduce(
+            (xs, key) => ({ ...xs, [key]: xs[key] || orig[key] }),
+            {
+                name: `${npmOrgName}/${packageName(target, format)}`,
+                version: undefined, main: `${mainExport}.js`, types: `${mainExport}.d.ts`,
+                dist: undefined, [`dist:es2015`]: undefined, module: undefined, [`@std/esm`]: undefined
+            }
+        )
     )
 );
 
 const conditionallyAddStandardESMEntry = (target, format) => (packageJSON) => (
-    format !== `esm`
-      ? packageJSON
-      : { ...packageJSON, [`@std/esm`]: { esm: `js` } }
+    format !== `esm` && format !== `cls`
+        ?      packageJSON
+        : { ...packageJSON, [`@std/esm`]: { esm: `js`, warnings: false, sourceMap: true } }
 );
 

js/gulp/test-task.js

@@ -44,15 +44,15 @@ const testOptions = {
 const testTask = ((cache, execArgv, testOptions) => memoizeTask(cache, function test(target, format, debug = false) {
     const opts = { ...testOptions };
     const args = !debug ? [...execArgv] : [...debugArgv, ...execArgv];
-    args.push(`test/${argv.integration ? `integration/*` : `unit/*`}`);
+    if (!argv.coverage) {
+        args.push(`test/${argv.integration ? `integration/*` : `unit/*`}`);
+    }
     opts.env = { ...opts.env,
         TEST_TARGET: target,
         TEST_MODULE: format,
-        JSON_PATH: argv.json_file,
-        ARROW_PATH: argv.arrow_file,
         TEST_TS_SOURCE: !!argv.coverage,
-        TEST_SOURCES: JSON.stringify(Array.isArray(argv.sources) ? argv.sources : [argv.sources]),
-        TEST_FORMATS: JSON.stringify(Array.isArray(argv.formats) ? argv.formats : [argv.formats]),
+        JSON_PATHS: JSON.stringify(Array.isArray(argv.json_files) ? argv.json_files : [argv.json_files]),
+        ARROW_PATHS: JSON.stringify(Array.isArray(argv.arrow_files) ? argv.arrow_files : [argv.arrow_files]),
     };
     return !debug ?
         child_process.spawn(jest, args, opts) :

js/gulp/typescript-task.js

@@ -34,7 +34,7 @@ const typescriptTask = ((cache) => memoizeTask(cache, function typescript(target
     const tsProject = ts.createProject(path.join(`tsconfig`, tsconfigFile), { typescript: require(`typescript`) });
     const { stream: { js, dts } } = observableFromStreams(
       tsProject.src(), sourcemaps.init(),
-      tsProject(ts.reporter.fullReporter(true))
+      tsProject(ts.reporter.defaultReporter())
     );
     const writeDTypes = observableFromStreams(dts, gulp.dest(out));
     const writeJS = observableFromStreams(js, sourcemaps.write(), gulp.dest(out));
@@ -52,12 +52,12 @@ function maybeCopyRawJSArrowFormatFiles(target, format) {
         return Observable.empty();
     }
     return Observable.defer(async () => {
-        const outFormatDir = path.join(targetDir(target, format), `format`, `fb`);
+        const outFormatDir = path.join(targetDir(target, format), `fb`);
         await del(path.join(outFormatDir, '*.js'));
         await observableFromStreams(
-            gulp.src(path.join(`src`, `format`, `fb`, `*_generated.js`)),
+            gulp.src(path.join(`src`, `fb`, `*_generated.js`)),
             gulpRename((p) => { p.basename = p.basename.replace(`_generated`, ``); }),
             gulp.dest(outFormatDir)
         ).toPromise();
     });
-}
+}

js/gulp/uglify-task.js

@@ -29,7 +29,7 @@ const webpack = require(`webpack`);
 const { memoizeTask } = require('./memoize-task');
 const { Observable, ReplaySubject } = require('rxjs');
 const UglifyJSPlugin = require(`uglifyjs-webpack-plugin`);
-const esmRequire = require(`@std/esm`)(module, { cjs: true, esm: `js` });
+const esmRequire = require(`@std/esm`)(module, { cjs: true, esm: `js`, warnings: false });
 
 const uglifyTask = ((cache, commonConfig) => memoizeTask(cache, function uglifyJS(target, format) {
 
@@ -84,11 +84,20 @@ module.exports = uglifyTask;
 module.exports.uglifyTask = uglifyTask;
 
 const reservePublicNames = ((ESKeywords) => function reservePublicNames(target, format) {
-    const publicModulePath = `../${targetDir(target, format)}/${mainExport}.js`;
-    return [
-        ...ESKeywords,
-        ...reserveExportedNames(esmRequire(publicModulePath))
+    const src = targetDir(target, format);
+    const publicModulePaths = [
+        `../${src}/data.js`,
+        `../${src}/type.js`,
+        `../${src}/table.js`,
+        `../${src}/vector.js`,
+        `../${src}/util/int.js`,
+        `../${src}/predicate.js`,
+        `../${src}/recordbatch.js`,
+        `../${src}/${mainExport}.js`,
     ];
+    return publicModulePaths.reduce((keywords, publicModulePath) => [
+        ...keywords, ...reserveExportedNames(esmRequire(publicModulePath, { warnings: false }))
+    ], [...ESKeywords]);
 })(ESKeywords);
 
 // Reflect on the Arrow modules to come up with a list of keys to save from Uglify's
@@ -104,8 +113,8 @@ const reserveExportedNames = (entryModule) => (
         .map((name) => [name, entryModule[name]])
         .reduce((reserved, [name, value]) => {
             const fn = function() {};
-            const ownKeys = value && Object.getOwnPropertyNames(value) || [];
-            const protoKeys = typeof value === `function` && Object.getOwnPropertyNames(value.prototype) || [];
+            const ownKeys = value && typeof value === 'object' && Object.getOwnPropertyNames(value) || [];
+            const protoKeys = typeof value === `function` && Object.getOwnPropertyNames(value.prototype || {}) || [];
             const publicNames = [...ownKeys, ...protoKeys].filter((x) => x !== `default` && x !== `undefined` && !(x in fn));
             return [...reserved, name, ...publicNames];
         }, []

js/gulp/util.js

@@ -87,7 +87,7 @@ const ESKeywords = [
     // EventTarget
     `addListener`, `removeListener`, `addEventListener`, `removeEventListener`,
     // Arrow properties
-    `low`, `high`, `data`, `index`, `field`, `validity`, `columns`, `fieldNode`, `subarray`,
+    `low`, `high`, `data`, `index`, `field`, `columns`, 'numCols', 'numRows', `values`, `valueOffsets`, `nullBitmap`, `subarray`
 ];
 
 function taskName(target, format) {
@@ -108,14 +108,13 @@ function targetDir(target, format) {
 
 function logAndDie(e) {
     if (e) {
-        console.error(e);
         process.exit(1);
     }
 }
 
 function observableFromStreams(...streams) {
-    const pumped = streams.length <= 1 ? streams[0]
-        : pump(...streams, logAndDie);
+    if (streams.length <= 0) { return Observable.empty(); }
+    const pumped = streams.length <= 1 ? streams[0] : pump(...streams, logAndDie);
     const fromEvent = Observable.fromEvent.bind(null, pumped);
     const streamObs = fromEvent(`data`)
                .merge(fromEvent(`error`).flatMap((e) => Observable.throw(e)))