Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api): add slug parser in the api requests #6705

Merged
merged 23 commits into from
Oct 23, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
bd88fa6
feat(api): allow passing workflow identifier as id in new v2 workflow…
djabarovgeorge Oct 14, 2024
c0fa7d5
feat(api): add slug api parser
djabarovgeorge Oct 15, 2024
a18552c
feat(api): refactor parsing
djabarovgeorge Oct 16, 2024
a3236b5
feat(api): update submodule hash
djabarovgeorge Oct 17, 2024
cc61194
Merge remote-tracking branch 'origin/next' into nv-4494-add-slug-pars…
djabarovgeorge Oct 17, 2024
46151b4
Merge branch 'next' into nv-4494-add-slug-parser-in-the-api-requests
djabarovgeorge Oct 20, 2024
7a321ca
fix(api): lock file
djabarovgeorge Oct 20, 2024
347b7fc
fix(api): lock file
djabarovgeorge Oct 20, 2024
541510d
Merge branch 'next' into nv-4494-add-slug-parser-in-the-api-requests
djabarovgeorge Oct 20, 2024
6fd09a6
Merge branch 'next' into nv-4494-add-slug-parser-in-the-api-requests
djabarovgeorge Oct 21, 2024
89606cc
refactor(api): update after pr comments
djabarovgeorge Oct 22, 2024
1ac74bd
fix(api): revert test script
djabarovgeorge Oct 22, 2024
2059789
Merge branch 'next' into nv-4494-add-slug-parser-in-the-api-requests
djabarovgeorge Oct 22, 2024
aa9ed71
feat(api): encode workflow and step ids on api response (#6725)
djabarovgeorge Oct 22, 2024
f74d73c
Merge remote-tracking branch 'origin/next' into nv-4494-add-slug-pars…
djabarovgeorge Oct 22, 2024
2a64335
fix(api): after next merge
djabarovgeorge Oct 22, 2024
06cbea7
fix(api): after next merge
djabarovgeorge Oct 22, 2024
61c1a5d
fix(api): after next merge
djabarovgeorge Oct 22, 2024
2a15a3b
fix(api): after next merge
djabarovgeorge Oct 22, 2024
cf9526b
fix(api): fix id
tatarco Oct 23, 2024
bc632b6
Merge remote-tracking branch 'origin/next' into nv-4494-add-slug-pars…
djabarovgeorge Oct 23, 2024
d5b30f6
Merge remote-tracking branch 'origin/nv-4494-add-slug-parser-in-the-a…
djabarovgeorge Oct 23, 2024
5d27235
feat: revert hash to next
djabarovgeorge Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .source
3 changes: 2 additions & 1 deletion apps/api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
"@types/newrelic": "^9.14.0",
"@upstash/ratelimit": "^0.4.4",
"axios": "^1.6.8",
"base-x": "^5.0.0",
Copy link
Contributor

@rifont rifont Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: Can we write our own base62 encoder from scratch? The pipe logic is effectively using unvalidated data. I'm concerned that any vulnerabilities (intentionally or maliciously) added to the base-x package could expose us‏ to an attack vector

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think if will have fixed version 5.0.0? Will it solve the concern for us? Then, we could upgrade the version if needed and only to a stable and secure version.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a know security issue with the package?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The truth is that a custom implementation with two functions, encodeBase62, decodeBase62 would be much much simpler, no need for pipes, no need to worry about potential issues.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠ issue: Can we write our own base62 encoder from scratch? The pipe logic is effectively using unvalidated data. I'm concerned that any vulnerabilities (intentionally or maliciously) added to the base-x package could expose us‏ to an attack vector

i had a hard time implementing the code from scratch, i added up comping the code from base-x so we won't get any vulnerabilities by mistake.

Is there a know security issue with the package?

not at the moment.

The truth is that a custom implementation with two functions, encodeBase62, decodeBase62 would be much much simpler, no need for pipes, no need to worry about potential issues.

i implemented encodeBase62, decodeBase62 based on base-x, pipes are actually a nice touch of nest js making the boundaries of controller and use case responsibility cleaner

"bcrypt": "^5.0.0",
"body-parser": "^1.20.0",
"bull": "^4.2.1",
Expand Down Expand Up @@ -91,7 +92,7 @@
"shortid": "^2.2.16",
"slugify": "^1.4.6",
"swagger-ui-express": "^4.4.0",
"twilio": "^4.14.1",
"twilio": "^4.14.1",
"json-schema-to-ts": "^3.0.0",
"uuid": "^8.3.2"
},
Expand Down
14 changes: 1 addition & 13 deletions apps/api/src/app/auth/services/passport/jwt.strategy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,19 +39,7 @@ export class JwtStrategy extends PassportStrategy(Strategy) {
const environmentIdFromHeader =
(req.headers[HttpRequestHeaderKeysEnum.NOVU_ENVIRONMENT_ID.toLowerCase()] as string) || '';

let currentEnvironmentId = '';

const environments = await this.environmentRepository.findOrganizationEnvironments(session.organizationId);
const environmentIds = environments.map((env) => env._id);
const developmentEnvironmentId = environments.find((env) => env.name === 'Development')?._id || '';

currentEnvironmentId = developmentEnvironmentId;

if (environmentIds.includes(environmentIdFromHeader)) {
currentEnvironmentId = environmentIdFromHeader;
}

// eslint-disable-next-line no-param-reassign
session.environmentId = currentEnvironmentId;
session.environmentId = environmentIdFromHeader;
}
}
22 changes: 22 additions & 0 deletions apps/api/src/app/shared/helpers/base62.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import basex from 'base-x';

const BASE62 = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
const { encode, decode } = basex(BASE62);

export function encodeBase62(value: string): string {
const buffer = Buffer.from(value, 'hex');

return encode(buffer);
}

export function decodeBase62(encoded: string): string {
const uint8Array = decode(encoded);

return Buffer.from(uint8Array).toString('hex');
}

export function isBase62(input: string): boolean {
const base62Regex = /^[0-9A-Za-z]+$/;

return base62Regex.test(input);
}
djabarovgeorge marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions apps/api/src/app/shared/helpers/index.ts
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
export * from './utils';
export * from './base62';
65 changes: 65 additions & 0 deletions apps/api/src/app/workflows-v2/pipes/parse-slug-Id.pipe spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import { expect } from 'chai';
import { ArgumentMetadata } from '@nestjs/common';

import { ParseSlugIdPipe } from './parse-slug-Id.pipe';
import { encodeBase62 } from '../../shared/helpers';

describe('ParseSlugIdPipe', () => {
let pipe: ParseSlugIdPipe;

beforeEach(() => {
pipe = new ParseSlugIdPipe();
});

it('should return the original value for non-slug IDs', () => {
const workflowIdentifier = 'non-slug-id';
expect(pipe.transform(workflowIdentifier, {} as ArgumentMetadata)).to.equal(workflowIdentifier);

const internalId = '6615943e7ace93b0540ae377';
expect(pipe.transform(internalId, {} as ArgumentMetadata)).to.equal(internalId);
});

it('should handle invalid encoded IDs', () => {
const invalidSlugId = 'my-workflow_invalidEncoding';
expect(pipe.transform(invalidSlugId, {} as ArgumentMetadata)).to.equal(invalidSlugId);
});

it('should not trim or decode internalId', () => {
const internalId = '6615943e7ace93b0540ae377';
expect(pipe.transform(internalId, {} as ArgumentMetadata)).to.equal(internalId);
});

it.skip('should trim prefix and decode base62 encoded internalId', () => {
const internalId = '6615943e7ace93b0540ae377';
const encodedId = encodeBase62(`wf_${internalId}`);
expect(pipe.transform(`wf_${encodedId}`, {} as ArgumentMetadata)).to.equal(internalId);
});

it('should not trim or decode simple workflow identifier', () => {
const identifier = 'my-workflow';
expect(pipe.transform(identifier, {} as ArgumentMetadata)).to.equal(identifier);
});

it.skip('should trim, decode, and remove prefix for a valid slug ID', () => {
const internalId = '6615943e7ace93b0540ae377';
const encodedId = encodeBase62(`wf_${internalId}`);
expect(pipe.transform(`my-workflow_${encodedId}`, {} as ArgumentMetadata)).to.equal(internalId);
});

it('should return original value for invalid encoded ID', () => {
const invalidSlug = 'my-workflow_invalid';
expect(pipe.transform(invalidSlug, {} as ArgumentMetadata)).to.equal(invalidSlug);
});

it('should handle slug IDs without known prefixes', () => {
const internalId = '6615943e7ace93b0540ae377';
const encodedId = encodeBase62(internalId);
expect(pipe.transform(`my-workflow_${encodedId}`, {} as ArgumentMetadata)).to.equal(internalId);
});

it.skip('should handle slug IDs with multiple underscores', () => {
const internalId = '6615943e7ace93b0540ae377';
const encodedId = encodeBase62(`wf_${internalId}`);
expect(pipe.transform(`my_complex_workflow_${encodedId}`, {} as ArgumentMetadata)).to.equal(internalId);
});
});
55 changes: 55 additions & 0 deletions apps/api/src/app/workflows-v2/pipes/parse-slug-Id.pipe.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import { ArgumentMetadata, Injectable, PipeTransform } from '@nestjs/common';
import { BaseRepository } from '@novu/dal';
import { decodeBase62 } from '../../shared/helpers';

type InternalId = string;
const INTERNAL_ID_LENGTH = 24;
const ENCODED_ID_LENGTH = 16;

function isWorkflowId(value: string) {
return value.length < ENCODED_ID_LENGTH;
}

function isInternalId(value: string) {
return BaseRepository.isInternalId(value) && value.length === INTERNAL_ID_LENGTH;
}

function lookoutForId(value: string): string | null {
if (isInternalId(value)) {
return value;
}

if (isWorkflowId(value)) {
return value;
}

return null;
}

export function parseSlugId(value: string): InternalId {
const validId = lookoutForId(value);
if (validId) {
return validId;
}

const encodedValue = value.slice(-16);
let decodedValue: string;
try {
decodedValue = decodeBase62(encodedValue);
} catch (error) {
return value;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if not base 62

}
const validDecodedId = lookoutForId(decodedValue);
if (validDecodedId) {
return validDecodedId;
}

return value;
}

@Injectable()
export class ParseSlugIdPipe implements PipeTransform<string, InternalId> {
transform(value: string, metadata: ArgumentMetadata): InternalId {
return parseSlugId(value);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ import { IsDefined, IsString } from 'class-validator';
export class DeleteWorkflowCommand extends EnvironmentWithUserObjectCommand {
@IsString()
@IsDefined()
workflowId: string;
workflowIdOrIdentifier: string;
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,35 +8,37 @@ import {
} from '@novu/dal';

import { DeleteWorkflowCommand } from './delete-workflow.command';
import { WorkflowNotFoundException } from '../../exceptions/workflow-not-found-exception';
import { GetWorkflowByIdsUseCase } from '../get-workflow-by-ids/get-workflow-by-ids.usecase';
import { GetWorkflowByIdsCommand } from '../get-workflow-by-ids/get-workflow-by-ids.command';

@Injectable()
export class DeleteWorkflowUseCase {
constructor(
private notificationTemplateRepository: NotificationTemplateRepository,
private messageTemplateRepository: MessageTemplateRepository,
private getWorkflowByIdsUseCase: GetWorkflowByIdsUseCase,
private controlValuesRepository: ControlValuesRepository
) {}

async execute(command: DeleteWorkflowCommand): Promise<void> {
const workflow = await this.notificationTemplateRepository.findByIdQuery({
id: command.workflowId,
environmentId: command.user.environmentId,
});
if (!workflow) {
throw new WorkflowNotFoundException(command.workflowId);
}
await this.deleteRelatedEntities(command, workflow);
const workflowEntity: NotificationTemplateEntity | null = await this.getWorkflowByIdsUseCase.execute(
GetWorkflowByIdsCommand.create({
...command,
workflowIdOrIdentifier: command.workflowIdOrIdentifier,
})
);

await this.deleteRelatedEntities(command, workflowEntity);
}

private async deleteRelatedEntities(command: DeleteWorkflowCommand, workflow) {
private async deleteRelatedEntities(command: DeleteWorkflowCommand, workflow: NotificationTemplateEntity) {
await this.controlValuesRepository.deleteMany({
_environmentId: command.user.environmentId,
_organizationId: command.user.organizationId,
_workflowId: command.workflowId,
_workflowId: workflow._id,
});
await this.removeMessageTemplatesIfNeeded(workflow, command);
await this.notificationTemplateRepository.delete(buildDeleteQuery(command));
await this.notificationTemplateRepository.delete(buildDeleteQuery(command, workflow._id));
}

private async removeMessageTemplatesIfNeeded(workflow: NotificationTemplateEntity, command: DeleteWorkflowCommand) {
Expand All @@ -50,9 +52,9 @@ export class DeleteWorkflowUseCase {
}
}
}
function buildDeleteQuery(command: DeleteWorkflowCommand) {
function buildDeleteQuery(command: DeleteWorkflowCommand, _workflowId: string) {
return {
_id: command.workflowId,
_id: _workflowId,
_organizationId: command.user.organizationId,
_environmentId: command.user.environmentId,
};
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { EnvironmentWithUserObjectCommand } from '@novu/application-generic';
import { IsDefined, IsString } from 'class-validator';

export class GetWorkflowByIdsCommand extends EnvironmentWithUserObjectCommand {
@IsString()
@IsDefined()
workflowIdOrIdentifier: string;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import { Injectable } from '@nestjs/common';

import { NotificationTemplateEntity, NotificationTemplateRepository } from '@novu/dal';

import { GetWorkflowByIdsCommand } from './get-workflow-by-ids.command';
import { WorkflowNotFoundException } from '../../exceptions/workflow-not-found-exception';

@Injectable()
export class GetWorkflowByIdsUseCase {
constructor(private notificationTemplateRepository: NotificationTemplateRepository) {}
async execute(command: GetWorkflowByIdsCommand): Promise<NotificationTemplateEntity> {
const isInternalId = NotificationTemplateRepository.isInternalId(command.workflowIdOrIdentifier);

let workflowEntity: NotificationTemplateEntity | null;

if (isInternalId) {
workflowEntity = await this.notificationTemplateRepository.findById(
command.workflowIdOrIdentifier,
command.user.environmentId
);
} else {
workflowEntity = await this.notificationTemplateRepository.findByTriggerIdentifier(
command.user.environmentId,
command.workflowIdOrIdentifier
);
}

if (!workflowEntity) {
throw new WorkflowNotFoundException(command.workflowIdOrIdentifier);
}

return workflowEntity;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ import { IsDefined, IsString } from 'class-validator';
export class GetWorkflowCommand extends EnvironmentWithUserObjectCommand {
@IsString()
@IsDefined()
_workflowId: string;
workflowIdOrIdentifier: string;
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,50 +4,51 @@ import {
ControlValuesEntity,
ControlValuesRepository,
NotificationStepEntity,
NotificationTemplateRepository,
NotificationTemplateEntity,
} from '@novu/dal';
import { ControlValuesLevelEnum, WorkflowResponseDto } from '@novu/shared';
import { GetPreferences, GetPreferencesCommand } from '@novu/application-generic';

import { GetWorkflowCommand } from './get-workflow.command';
import { WorkflowNotFoundException } from '../../exceptions/workflow-not-found-exception';
import { toResponseWorkflowDto } from '../../mappers/notification-template-mapper';
import { GetWorkflowByIdsUseCase } from '../get-workflow-by-ids/get-workflow-by-ids.usecase';
import { GetWorkflowByIdsCommand } from '../get-workflow-by-ids/get-workflow-by-ids.command';

@Injectable()
export class GetWorkflowUseCase {
constructor(
private notificationTemplateRepository: NotificationTemplateRepository,
private getWorkflowByIdsUseCase: GetWorkflowByIdsUseCase,
private controlValuesRepository: ControlValuesRepository,
private getPreferencesUseCase: GetPreferences
) {}
async execute(command: GetWorkflowCommand): Promise<WorkflowResponseDto> {
const notificationTemplateEntity = await this.notificationTemplateRepository.findByIdQuery({
id: command._workflowId,
environmentId: command.user.environmentId,
});
const workflowEntity: NotificationTemplateEntity | null = await this.getWorkflowByIdsUseCase.execute(
GetWorkflowByIdsCommand.create({
...command,
workflowIdOrIdentifier: command.workflowIdOrIdentifier,
})
);

if (!notificationTemplateEntity) {
throw new WorkflowNotFoundException(command._workflowId);
}
const stepIdToControlValuesMap = await this.getControlsValuesMap(notificationTemplateEntity.steps, command);
const stepIdToControlValuesMap = await this.getControlsValuesMap(workflowEntity.steps, command, workflowEntity._id);
const preferences = await this.getPreferencesUseCase.safeExecute(
GetPreferencesCommand.create({
environmentId: command.user.environmentId,
organizationId: command.user.organizationId,
})
);

return toResponseWorkflowDto(notificationTemplateEntity, preferences, stepIdToControlValuesMap);
return toResponseWorkflowDto(workflowEntity, preferences, stepIdToControlValuesMap);
}

private async getControlsValuesMap(
steps: NotificationStepEntity[],
command: GetWorkflowCommand
command: GetWorkflowCommand,
_workflowId: string
): Promise<{ [key: string]: ControlValuesEntity }> {
const acc: { [key: string]: ControlValuesEntity } = {};

for (const step of steps) {
const controlValuesEntity = await this.buildControlValuesForStep(step, command);
const controlValuesEntity = await this.buildControlValuesForStep(step, command, _workflowId);
if (controlValuesEntity) {
acc[step._templateId] = controlValuesEntity;
}
Expand All @@ -57,12 +58,13 @@ export class GetWorkflowUseCase {
}
private async buildControlValuesForStep(
step: NotificationStepEntity,
command: GetWorkflowCommand
command: GetWorkflowCommand,
_workflowId: string
): Promise<ControlValuesEntity | null> {
return await this.controlValuesRepository.findFirst({
_environmentId: command.user.environmentId,
_organizationId: command.user.organizationId,
_workflowId: command._workflowId,
_workflowId,
_stepId: step._templateId,
level: ControlValuesLevelEnum.STEP_CONTROLS,
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { EnvironmentWithUserObjectCommand } from '@novu/application-generic';
import { CreateWorkflowDto, UpdateWorkflowDto } from '@novu/shared';

export class UpsertWorkflowCommand extends EnvironmentWithUserObjectCommand {
workflowDatabaseIdForUpdate?: string;
workflowIdOrIdentifier?: string;

workflowDto: CreateWorkflowDto | UpdateWorkflowDto;
}
Loading
Loading