Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PostgreSQL sink creates table when entity id contains special characters #2220

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

PostgreSQL sink creates table when entity id contains special characters #2220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSIPostgreSQLSink.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -616,32 +616,32 @@ public String buildTableName(String servicePath, String entity, String entityTyp
+ "dm-by-service-path data model");
} // if

name = NGSIUtils.encode(servicePath, true, false);
name = NGSIUtils.encodePostgresTable(servicePath);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the changes in this PR, is encode() still used in other parts of the code? Or it can be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encode() is used in below part of the code:
When "enable encoding=false" that time encode() is used to build DBName and SchemaName in class NGSIPostgreSQLSink. Also it is used in other sink like NGSIPostgisSink, NGSIOracleSink,NGSIMysqlSink etc. So we can't remove it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification

NTC

break;
case DMBYENTITYDATABASE:
case DMBYENTITYDATABASESCHEMA:
case DMBYENTITY:
String truncatedServicePath = NGSIUtils.encode(servicePath, true, false);
String truncatedServicePath = NGSIUtils.encodePostgresTable(servicePath);
name = (truncatedServicePath.isEmpty() ? "" : truncatedServicePath + '_')
+ NGSIUtils.encode(entity, false, true);
+ NGSIUtils.encodePostgresTable(entity);
break;
case DMBYENTITYTYPEDATABASE:
case DMBYENTITYTYPEDATABASESCHEMA:
case DMBYENTITYTYPE:
truncatedServicePath = NGSIUtils.encode(servicePath, true, false);
truncatedServicePath = NGSIUtils.encodePostgresTable(servicePath);
name = (truncatedServicePath.isEmpty() ? "" : truncatedServicePath + '_')
+ NGSIUtils.encode(entityType, false, true);
+ NGSIUtils.encodePostgresTable(entityType);
break;
case DMBYATTRIBUTE:
truncatedServicePath = NGSIUtils.encode(servicePath, true, false);
truncatedServicePath = NGSIUtils.encodePostgresTable(servicePath);
name = (truncatedServicePath.isEmpty() ? "" : truncatedServicePath + '_')
+ NGSIUtils.encode(entity, false, true)
+ '_' + NGSIUtils.encode(attribute, false, true);
+ NGSIUtils.encodePostgresTable(entity)
+ '_' + NGSIUtils.encodePostgresTable(attribute);
break;
case DMBYFIXEDENTITYTYPEDATABASE:
case DMBYFIXEDENTITYTYPEDATABASESCHEMA:
case DMBYFIXEDENTITYTYPE:
name = NGSIUtils.encode(entityType, false, true);
name = NGSIUtils.encodePostgresTable(entityType);
break;
default:
throw new CygnusBadConfiguration("Unknown data model '" + dataModel.toString()
Expand Down
6 changes: 3 additions & 3 deletions cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/utils/NGSICharsets.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,9 @@ public static String encodePostgreSQL(String in) {
char c = in.charAt(i);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where encodePostgreSQL() function is used?

It seems this PR fixes it but it is not clear to me where it is used...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When "Enable ecoding=true", encodePostgreSQL() function is used while building schema name, db name and table name.

int code = c;

if (code >= 97 && code <= 119) { // a-w --> a-w
if (code >= 97 && code <= 119 || code >= 65 && code <= 87) { // a-w--> a-w,A-W-->A-W
out += c;
} else if (c == 'x') {
} else if (c == 'x'|| c == 'X') {
String next4;

if (i + 4 < in.length()) {
Expand All @@ -57,7 +57,7 @@ public static String encodePostgreSQL(String in) {
} else { // x --> x
out += c;
} // if else
} else if (code == 121 || code == 122) { // yz --> yz
} else if (code == 121 || code == 122 || code ==89 || code == 90) { // yz --> yz,YZ-->YZ
out += c;
} else if (code >= 48 && code <= 57) { // 0-9 --> 0-9
out += c;
Expand Down
15 changes: 15 additions & 0 deletions cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/utils/NGSIUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ public final class NGSIUtils {
private static final Pattern ENCODEHIVEPATTERN = Pattern.compile("[^a-zA-Z0-9]");
private static final Pattern ENCODESTHDBPATTERN = Pattern.compile("[=\\/\\\\.\\$\" ]");
private static final Pattern ENCODESTHCOLLECTIONPATTERN = Pattern.compile("[=\\$]");
private static final Pattern ENCODEPOSTGRESTABLEPATTERN=Pattern.compile("[^a-zA-Z0-9\\.]");

/**
* Constructor. It is private since utility classes should not have a public or default constructor.
Expand Down Expand Up @@ -71,6 +72,20 @@ public static String encode(String in, boolean deleteSlash, boolean encodeSlash)
return ENCODEPATTERNSLASH.matcher(in).replaceAll("_");
} // if else
} // encode

/**
* Encodes a string replacing all '/', '\', '.', ' ', '"', '-'and '$' by '_'.
* @param in
* @return The encoded version of the input string
*/

public static String encodePostgresTable(String in) {
if(in.startsWith("/")) {
return ENCODEPOSTGRESTABLEPATTERN.matcher(in.substring(1)).replaceAll("_");
}else {
return ENCODEPOSTGRESTABLEPATTERN.matcher(in).replaceAll("_");
}
}

/**
* Encodes a string replacing all '/', '\', '.', ' ', '"' and '$' by '_'.
Expand Down
8 changes: 4 additions & 4 deletions cygnus-ngsi/src/test/java/com/telefonica/iot/cygnus/sinks/NGSICartoDBSinkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -991,7 +991,7 @@ public void testBuildSchemaName() throws Exception {

try {
String builtSchemaName = sink.buildSchemaName(service);
String expectedSchemaName = "somex0053ervice";
String expectedSchemaName = "someService";

try {
assertEquals(expectedSchemaName, builtSchemaName);
Expand Down Expand Up @@ -1042,7 +1042,7 @@ public void testBuildTableNameNonRootServicePathDataModelByServicePath() throws

try {
String builtTableName = sink.buildTableName(servicePath, entity, attribute);
String expecetedTableName = "x002fsomex0050ath";
String expecetedTableName = "x002fsomePath";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -1096,7 +1096,7 @@ public void testBuildTableNameNonRootServicePathDataModelByEntity() throws Excep
String builtTableName = sink.buildTableName(servicePath, entity, attribute);

try {
assertEquals("x002fsomex0050athxffffsomex0049dxffffsomex0054ype", builtTableName);
assertEquals("x002fsomePathxffffsomeIdxffffsomeType", builtTableName);
System.out.println(getTestTraceHead("[NGSICartoDBSink.buildTableName]")
+ "- OK - '" + builtTableName + "' is equals to the lower case of "
+ "x002f<servicePath>xffff<entityId>xffff<entityType>");
Expand Down Expand Up @@ -1198,7 +1198,7 @@ public void testBuildTableNameRootServicePathDataModelByEntity() throws Exceptio

try {
String builtTableName = sink.buildTableName(servicePath, entity, attribute);
String expectedTableName = "x002fxffffsomex0049dxffffsomex0054ype";
String expectedTableName = "x002fxffffsomeIdxffffsomeType";

try {
assertEquals(expectedTableName, builtTableName);
Expand Down
8 changes: 4 additions & 4 deletions cygnus-ngsi/src/test/java/com/telefonica/iot/cygnus/sinks/NGSIOracleSQLSinkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ public void testBuildSchemaNameNewEncoding() throws Exception {

try {
String builtSchemaName = sink.buildSchemaName(service, servicePath);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -557,7 +557,7 @@ public void testBuildDBNameNewEncodingDatabaseDataModel() throws Exception {

try {
String builtSchemaName = sink.buildDbName(service);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -604,7 +604,7 @@ public void testBuildDBNameNewEncodingEntityTypeDatabaseDataModel() throws Excep

try {
String builtSchemaName = sink.buildDbName(service);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -706,7 +706,7 @@ public void testBuildTableNameNonRootServicePathDataModelByServicePathNewEncodin

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fsomex0050ath";
String expecetedTableName = "x002fsomePath";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down
18 changes: 9 additions & 9 deletions cygnus-ngsi/src/test/java/com/telefonica/iot/cygnus/sinks/NGSIPostgisSinkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,7 @@ public void testBuildSchemaNameNewEncoding() throws Exception {

try {
String builtSchemaName = sink.buildSchemaName(service, servicePath);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -568,7 +568,7 @@ public void testBuildDBNameNewEncodingDatabaseDataModel() throws Exception {

try {
String builtSchemaName = sink.buildDBName(service);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -615,7 +615,7 @@ public void testBuildDBNameNewEncodingEntityTypeDatabaseDataModel() throws Excep

try {
String builtSchemaName = sink.buildDBName(service);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -715,7 +715,7 @@ public void testBuildTableNameNonRootServicePathDataModelByServicePathNewEncodin

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fsomex0050ath";
String expecetedTableName = "x002fsomePath";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -821,7 +821,7 @@ public void testBuildTableNameNonRootServicePathDataModelByEntityNewEncoding() t

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fsomex0050athxffffsomex0049dxffffsomex0054ype";
String expecetedTableName = "x002fsomePathxffffsomeIdxffffsomeType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -929,7 +929,7 @@ public void testBuildTableNameNonRootServicePathDataModelByEntityTypeNewEncoding

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fsomex0050athxffffsomex0054ype";
String expecetedTableName = "x002fsomePathxffffsomeType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -1031,7 +1031,7 @@ public void testBuildTableNameNonRootServicePathDataModelByFixedEntityTypeNewEnc

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "somex0054ype";
String expecetedTableName = "someType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -1225,7 +1225,7 @@ public void testBuildTableNameRootServicePathDataModelByEntityNewEncoding() thro

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fxffffsomex0049dxffffsomex0054ype";
String expecetedTableName = "x002fxffffsomeIdxffffsomeType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -1323,7 +1323,7 @@ public void testBuildTableNameRootServicePathDataModelByEntityTypeNewEncoding()

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fxffffsomex0054ype";
String expecetedTableName = "x002fxffffsomeType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down
14 changes: 7 additions & 7 deletions cygnus-ngsi/src/test/java/com/telefonica/iot/cygnus/sinks/NGSIPostgreSQLSinkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ public void testBuildSchemaNameNewEncoding() throws Exception {

try {
String builtSchemaName = sink.buildSchemaName(service, servicePath);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -564,7 +564,7 @@ public void testBuildDBNameNewEncodingDatabaseDataModel() throws Exception {

try {
String builtSchemaName = sink.buildDBName(service);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -611,7 +611,7 @@ public void testBuildDBNameNewEncodingEntityTypeDatabaseDataModel() throws Excep

try {
String builtSchemaName = sink.buildDBName(service);
String expectedDBName = "somex0053ervice";
String expectedDBName = "someService";

try {
assertEquals(expectedDBName, builtSchemaName);
Expand Down Expand Up @@ -713,7 +713,7 @@ public void testBuildTableNameNonRootServicePathDataModelByServicePathNewEncodin

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fsomex0050ath";
String expecetedTableName = "x002fsomePath";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -819,7 +819,7 @@ public void testBuildTableNameNonRootServicePathDataModelByEntityNewEncoding() t

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fsomex0050athxffffsomex0049dxffffsomex0054ype";
String expecetedTableName = "x002fsomePathxffffsomeIdxffffsomeType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -1031,7 +1031,7 @@ public void testBuildTableNameNonRootServicePathDataModelByFixedEntityTypeNewEnc

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "somex0054ype";
String expecetedTableName = "someType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down Expand Up @@ -1227,7 +1227,7 @@ public void testBuildTableNameRootServicePathDataModelByEntityNewEncoding() thro

try {
String builtTableName = sink.buildTableName(servicePath, entity, entityType, attribute);
String expecetedTableName = "x002fxffffsomex0049dxffffsomex0054ype";
String expecetedTableName = "x002fxffffsomeIdxffffsomeType";

try {
assertEquals(expecetedTableName, builtTableName);
Expand Down
3 changes: 1 addition & 2 deletions cygnus-ngsi/src/test/java/com/telefonica/iot/cygnus/utils/NGSICharsetsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,7 @@ public void testEncodePostgreSQLUpperCase() {
System.out.println(getTestTraceHead("[NGSICharsets.encodePostgreSQL]")
+ "-------- Upper case not accented characters are encoded");
String in = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
String expected = "x0041x0042x0043x0044x0045x0046x0047x0048x0049x004ax004bx004cx004dx004ex004f"
+ "x0050x0051x0052x0053x0054x0055x0056x0057x0058x0059x005a";
String expected = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
String out = NGSICharsets.encodePostgreSQL(in);

try {
Expand Down