Merge branch 'release/0.10'

.version

@@ -1 +1 @@
-0.10.1
+0.10.2

CHANGELOG.md

@@ -1,5 +1,15 @@
 ## Changelog
 
+#### Version 0.10.2 (August 24, 2019)
+* Fixed a bug that caused an error to be thrown when creating a `Criteria` containing a navigation key using the `Criteria#parse` factory.
+* Added an option to limit the length of substrings that are indexed for fulltext search. It is rare to add functionality in a patch release, but this improvement was needed to alleviate Concourse deployments that experience `OutOfMemory` exceptions because abnormally large String values are stuck in the Buffer waiting to be indexed. This option is **turned off by default** to maintain consistency with existing Concourse expectations for fulltext indexing. The option can be enabled by specifying a positive integer value for the newly added `max_search_substring_length` preference. When a value is supplied, the Storage Engine won't index any substrings longer than the provided value for any word in a String value.
+* Made internal improvements to the search indexing algorithm to reduce the number of itermediary objects created which should decrease the number of garbage collection cycles that are triggered.
+* Fixed a bug that caused the Engine to fail to accumulate metadata stats for fulltext search indices. This did not have any data correctness or consistency side-effects, but had the potential to make some search-related operations inefficient.
+* Fixed a bug that made it possible for the Database to appear to lose data when starting up after a crash or unexpected shutdown. This happened because the Database ignored data in Block files that erroneously appeared to be duplicates. We've fixed this issue by improving the logic and workflow that the Database uses to test whether Block files contain duplicate data.
+* Fixed a regression introduced in version `0.10.0` that made it possible for the Database to ignore errors that occurred when indexing writes. When indexing errors occur, the Database creates log entries and stops the indexing process, which is the behaviour that existed prior to version `0.10.0`.
+* Fixed a bug that made it possible for fatal indexing errors to occur when at least two writes were written to the same `Buffer` page for the same key with values that have different types (i.e. `Float` vs `Integer`) but are *essentially* equal (i.e. `18.0` vs `18`). In accordance with Concourse's weak typing system, values that are *essentially* the same will be properly indexed and queryable across associated types. This change **requires a reindex of all block files which is automatically done when upgrading from a previous version**.
+* Fixed a bug that causes the `ManagedConcourseServer` in the `concourse-ete-test-core` framework to take longer than necessary to allow connections in `ClientServerTest` cases.
+
 #### Version 0.10.1 (August 6, 2019)
 * Fixed a regression that caused an error when attempting an action with a CCL statement containing an unquoted string value with one or more periods.
 

README.md

@@ -1,13 +1,13 @@
 # Concourse
 
- ![](https://img.shields.io/badge/version-0.10.1-green.svg)
+ ![](https://img.shields.io/badge/version-0.10.2-green.svg)
  ![](https://img.shields.io/badge/status-alpha-orange.svg) ![](https://img.shields.io/badge/license-Apache%202-blue.svg)
  [![Join the chat at https://gitter.im/cinchapi/concourse](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/cinchapi/concourse?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
  [![](https://circleci.com/gh/cinchapi/concourse.svg?style=shield&circle-token=954a20e6114d649b1b6a046d95b953e7d05d2e2f)](https://circleci.com/gh/cinchapi/concourse)
 
 > [Concourse](http://concoursedb.com) is a distributed database warehouse for transactions search and analytics across time. Developers prefer Concourse because it simplifies building misssion-critical systems with on-demand data intelligence. Furthermore, Concourse makes end-to-end data management trivial by requiring no extra infrastructure, no prior configuration and no continuous tuning–all of which greatly reduce costs, and allow developers to focus on core business problems.
 
-This is version 0.10.1 of Concourse.
+This is version 0.10.2 of Concourse.
 
 ## Quickstart
 ### Docker

build.gradle

@@ -113,7 +113,7 @@ subprojects {
         compile 'joda-time:joda-time:2.2'
         compile 'org.apache.thrift:libthrift:0.12.0'
         compile 'commons-configuration:commons-configuration:1.9'
-        compile group: 'com.cinchapi', name: 'accent4j', version: '1.6.0', changing:true
+        compile group: 'com.cinchapi', name: 'accent4j', version: '1.8.0', changing:true
         compile 'com.cinchapi:lib-config:1.5.1'
 
         testCompile 'junit:junit:4.11'

concourse-driver-java/README.md

@@ -12,14 +12,14 @@ The concourse jar is available at [Maven Central](http://search.maven.org/#searc
 	}
 
 	dependencies {
-    	compile 'com.cinchapi:concourse-driver-java:0.10.1+'
+    	compile 'com.cinchapi:concourse-driver-java:0.10.2+'
 	}
 
 If you prefer to use another dependency manager like Maven or Ivy, then use the following project information when declaring the dependency:
 
 	GroupId: com.cinchapi
 	ArtifactId: concourse-driver-java
-	Version: 0.10.1+
+	Version: 0.10.2+
 
 Alternatively, you can [download](http://cinchapi.org/concourse/download-api) the latest jar and manually add it to your project's classpath.
 

concourse-driver-java/src/main/java/com/cinchapi/concourse/lang/Language.java

@@ -20,6 +20,7 @@
 
 import com.cinchapi.ccl.grammar.ConjunctionSymbol;
 import com.cinchapi.ccl.grammar.KeySymbol;
+import com.cinchapi.ccl.grammar.NavigationKeySymbol;
 import com.cinchapi.ccl.grammar.OperatorSymbol;
 import com.cinchapi.ccl.grammar.ParenthesisSymbol;
 import com.cinchapi.ccl.grammar.Symbol;
@@ -58,7 +59,9 @@ public static Symbol translateFromThriftSymbol(TSymbol tsymbol) {
             return ConjunctionSymbol.valueOf(tsymbol.getSymbol().toUpperCase());
         }
         else if(tsymbol.getType() == TSymbolType.KEY) {
-            return new KeySymbol(tsymbol.getSymbol());
+            String $symbol = tsymbol.getSymbol();
+            return $symbol.contains(".") ? new NavigationKeySymbol($symbol)
+                    : new KeySymbol($symbol);
         }
         else if(tsymbol.getType() == TSymbolType.VALUE) {
             Object symbol = Convert.stringToJava(tsymbol.getSymbol());
@@ -130,22 +133,22 @@ public static Criteria translateFromThriftCriteria(TCriteria tcriteria) {
      * @return The analogous TSymbol
      */
     public static TSymbol translateToThriftSymbol(Symbol symbol) {
-        if(symbol.getClass() == ConjunctionSymbol.class) {
+        if(symbol instanceof ConjunctionSymbol) {
             return new TSymbol(TSymbolType.CONJUNCTION, symbol.toString());
         }
-        else if(symbol.getClass() == KeySymbol.class) {
+        else if(symbol instanceof KeySymbol) {
             return new TSymbol(TSymbolType.KEY, symbol.toString());
         }
-        else if(symbol.getClass() == ValueSymbol.class) {
+        else if(symbol instanceof ValueSymbol) {
             return new TSymbol(TSymbolType.VALUE, symbol.toString());
         }
-        else if(symbol.getClass() == ParenthesisSymbol.class) {
+        else if(symbol instanceof ParenthesisSymbol) {
             return new TSymbol(TSymbolType.PARENTHESIS, symbol.toString());
         }
-        else if(symbol.getClass() == OperatorSymbol.class) {
+        else if(symbol instanceof OperatorSymbol) {
             return new TSymbol(TSymbolType.OPERATOR, symbol.toString());
         }
-        else if(symbol.getClass() == TimestampSymbol.class) {
+        else if(symbol instanceof TimestampSymbol) {
             return new TSymbol(TSymbolType.TIMESTAMP, symbol.toString());
         }
         else {

concourse-driver-php/composer.json

@@ -1,7 +1,7 @@
 {
     "name": "cinchapi/concourse-driver-php",
     "description": "Official PHP driver for Concourse",
-    "version": "0.10.1",
+    "version": "0.10.2",
     "type": "library",
     "keywords": ["concourse", "nosql", "bigdata"],
     "homepage": "http://concoursedb.com",

concourse-ete-test-core/src/main/java/com/cinchapi/concourse/server/ManagedConcourseServer.java

@@ -906,7 +906,7 @@ public Client(String username, String password, String environment) {
         private Client(String username, String password, String environment,
                 int retries) {
             Object delegate = null;
-            while (retries > 0) {
+            while (retries > 0 && delegate == null) {
                 --retries;
                 try {
                     this.loader = new URLClassLoader(

concourse-ete-tests/src/test/java/com/cinchapi/concourse/ete/performance/CrossVersionPerformanceBenchmarkTest.java

@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2013-2019 Cinchapi Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.cinchapi.concourse.ete.performance;
+
+import java.util.concurrent.TimeUnit;
+
+import org.junit.Test;
+
+import com.cinchapi.common.profile.Benchmark;
+import com.cinchapi.concourse.Tag;
+import com.cinchapi.concourse.test.CrossVersionTest;
+import com.cinchapi.concourse.test.runners.CrossVersionTestRunner.Versions;
+import com.cinchapi.concourse.time.Time;
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Unit test for performance across versions.
+ *
+ * @author Jeff Nelson
+ */
+@Versions({ "0.9.6", "0.10.1", "latest" })
+public class CrossVersionPerformanceBenchmarkTest extends CrossVersionTest {
+
+    @Test
+    public void testWrite() {
+        Benchmark benchmark = new Benchmark(TimeUnit.MILLISECONDS) {
+
+            @Override
+            public void action() {
+                for (int i = 0; i < 10000; ++i) {
+                    client.add("foo", Time.now(), i);
+                }
+            }
+
+        };
+        double avg = benchmark.run(10) / 10;
+        record("write", avg);
+    }
+
+    @Test
+    public void testRead() {
+        for (int i = 0; i < 10000; ++i) {
+            client.insert(ImmutableMap.of("foo", Time.now(), "bar", true, "baz",
+                    "hello", "bang", Tag.create("mafia")));
+        }
+        Benchmark benchmark = new Benchmark(TimeUnit.MILLISECONDS) {
+
+            @Override
+            public void action() {
+                client.select("foo < " + Time.now());
+            }
+
+        };
+        double avg = benchmark.run(10) / 10;
+        record("read", avg);
+    }
+}

concourse-integration-tests/src/test/java/com/cinchapi/concourse/bugrepro/CON662.java

@@ -0,0 +1,39 @@
+/*
+ * Copyright (c) 2013-2019 Cinchapi Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.cinchapi.concourse.bugrepro;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.cinchapi.concourse.lang.Criteria;
+import com.cinchapi.concourse.test.ConcourseIntegrationTest;
+
+/**
+ * Unit tests to repro the conditions in CON-661.
+ *
+ * @author Jeff Nelson
+ */
+public class CON662 extends ConcourseIntegrationTest {
+
+    @Test
+    public void test() {
+        Criteria criteria = Criteria.parse("foo.bar eq 1");
+        client.find(criteria);
+        Assert.assertTrue(true); // lack of Exception means the test passes
+
+    }
+
+}

concourse-server/conf/concourse.prefs

@@ -140,6 +140,24 @@
 # DEFAULT: INFO
 #log_level = INFO
 
+# The length of the longest substring that will be indexed for fulltext search.
+#
+# This value does not mean that longer words will not be indexed. It simply means 
+# that, for any indexable value (e.g. a String), any substring that is longer than
+# the value of this preference will not be added to the search index. The effect is
+# that search strings containing any words with a length greater than the value of 
+# this preference will return 0 results. 
+
+# For best performance, this value should be set to the longest word length of any 
+# possible search string. To be safe, we recommend setting this value to be the 
+# length of the longest possible word in the search language. For example, the longest 
+# possible word in English is about 40 characters long.
+#
+# By default, search substrings are not limited.
+#
+# DEFAULT: no limit
+#max_search_substring_length = 40
+
 # The listener port (1-65535) for shutdown commands. Choose a port between
 # 49152 and 65535 to minimize the possibility of conflicts with other services
 # on this host. In general, you shouldn't need to specify a value unless you

concourse-server/src/main/java/com/cinchapi/concourse/security/LegacyAccessManager.java

@@ -23,6 +23,7 @@
 
 import com.cinchapi.common.base.CheckedExceptions;
 import com.cinchapi.concourse.annotate.Restricted;
+import com.cinchapi.concourse.server.io.ByteSink;
 import com.cinchapi.concourse.server.io.Byteable;
 import com.cinchapi.concourse.server.io.ByteableCollections;
 import com.cinchapi.concourse.server.io.FileSystem;
@@ -243,7 +244,7 @@ public String toString() {
         }
 
         @Override
-        public void copyTo(ByteBuffer buffer) {
+        public void copyTo(ByteSink buffer) {
             buffer.put(ByteBuffers.decodeFromHex(password));
             buffer.put(ByteBuffers.decodeFromHex(salt));
             buffer.put(ByteBuffers.decodeFromHex(username));

concourse-server/src/main/java/com/cinchapi/concourse/server/GlobalState.java

@@ -251,6 +251,21 @@ public final class GlobalState extends Constants {
     public static boolean ENABLE_CONSOLE_LOGGING = RUNNING_FROM_ECLIPSE ? true
             : false;
 
+    /**
+     * The length of the longest substring that will be indexed for fulltext
+     * search.
+     * <p>
+     * Regardless of a word's length, this value controls the maximum length of
+     * any substring of the word that is added to the search index. For optimal
+     * performance, this value should be set to the longest possible substring
+     * of a word that would be entered for a search operation. To be safe, it is
+     * recommended to set this value to the length of the longest possible word
+     * in the target language. For example, the longest possible word in English
+     * is about 40 characters long.
+     * </p>
+     */
+    public static int MAX_SEARCH_SUBSTRING_LENGTH = -1;
+
     static {
         List<String> files = ImmutableList.of(
                 "conf" + File.separator + "concourse.prefs",
@@ -312,6 +327,9 @@ public final class GlobalState extends Constants {
                 Networking.getCompanionPort(CLIENT_PORT, 4));
 
         SYSTEM_ID = getSystemId();
+
+        MAX_SEARCH_SUBSTRING_LENGTH = config.getOrDefault(
+                "max_search_substring_length", MAX_SEARCH_SUBSTRING_LENGTH);
         // =================== PREF READING BLOCK ====================
     }