Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Components] zip_archive_api #11789 #11827

Merged
merged 8 commits into from
Sep 5, 2024
Merged

[Components] zip_archive_api #11789 #11827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ea2e20f
Added actions
lcaresia May 6, 2024
1795653
Fixed requested changes
lcaresia May 6, 2024
fa0ec12
Adding requested features
lcaresia May 15, 2024
a831338
Added actions
lcaresia May 17, 2024
ce4983e
Added actions
lcaresia May 22, 2024
2eb196c
Requested changes done
lcaresia Aug 30, 2024
7cffdea
Requested changes done
lcaresia Aug 30, 2024
fcbe336
Merge branch 'master' into issue-11789
lcaresia Sep 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions components/zip_archive_api/actions/compress-files/compress-files.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
import app from "../../zip_archive_api.app.mjs";
import fs from "fs";
import FormData from "form-data";

export default {
key: "zip_archive_api-compress-files",
name: "Compress Files",
description: "Compress files provided through URLs into a zip folder. [See the documentation](https://archiveapi.com/rest-api/file-compression/)",
version: "0.0.1",
type: "action",
props: {
app,
uploadType: {
propDefinition: [
app,
"uploadType",
],
},
archiveName: {
propDefinition: [
app,
"archiveName",
],
},
compressionLevel: {
propDefinition: [
app,
"compressionLevel",
],
},
password: {
propDefinition: [
app,
"password",
],
},
files: {
propDefinition: [
app,
"files",
],
},
},
async run({ $ }) {
let data = {
files: this.files,
archiveName: this.archiveName,
compressionLevel: this.compressionLevel,
password: this.password,
};

if (this.uploadType === "File") {
data = new FormData();

if (this.password) data.append("Password", this.password);
if (this.compressionLevel) data.append("CompressionLevel", this.compressionLevel);
data.append("ArchiveName", this.archiveName);

for (const file of this.files) {
data.append("Files", fs.createReadStream(file));
}
}

const headers = this.uploadType === "File"
? data.getHeaders()
: {};

const response = await this.app.compressFiles({
$,
data,
headers,
responseType: "arraybuffer",
});

const tmpPath = `/tmp/${this.archiveName}`;
fs.writeFileSync(tmpPath, response);

$.export("$summary", `Successfully compressed the files into '${tmpPath}'`);

return tmpPath;
},
Comment on lines +44 to +81
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the run Method

The run method is structured to handle different types of uploads based on uploadType. The use of FormData for file uploads is appropriate. The method handles the compression settings and file handling robustly. Key observations:

  1. Dynamic Data Handling: The method adjusts the data structure based on the uploadType, which is a good practice for handling different input types.
  2. File Stream Handling: Using fs.createReadStream for file uploads ensures that large files can be handled efficiently.
  3. Response Handling: The response is written to a temporary path, which is then returned. This is a practical approach for handling file downloads.

However, there are a few areas that could be improved:

  • Error Handling: There is no explicit error handling in case the API call fails or file operations throw an error.
  • Security Concern: Storing files temporarily on the server could lead to security risks if not managed properly.

Consider adding try-catch blocks around the API call and file operations to handle errors gracefully. Additionally, ensure that temporary files are securely handled or cleaned up after use to avoid potential security risks.

Here's a suggested refactor for error handling:

+ try {
  const response = await this.app.compressFiles({
    $,
    data,
    headers,
    responseType: "arraybuffer",
  });
  const tmpPath = `/tmp/${this.archiveName}`;
  fs.writeFileSync(tmpPath, response);
  $.export("$summary", `Successfully compressed the files into '${tmpPath}'`);
  return tmpPath;
+ } catch (error) {
+   $.export("$error", `Failed to compress files: ${error.message}`);
+   throw error;
+ }
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
async run({ $ }) {
let data = {
files: this.files,
archiveName: this.archiveName,
compressionLevel: this.compressionLevel,
password: this.password,
};
if (this.uploadType === "File") {
data = new FormData();
if (this.password) data.append("Password", this.password);
if (this.compressionLevel) data.append("CompressionLevel", this.compressionLevel);
data.append("ArchiveName", this.archiveName);
for (const file of this.files) {
data.append("Files", fs.createReadStream(file));
}
}
const headers = this.uploadType === "File"
? data.getHeaders()
: {};
const response = await this.app.compressFiles({
$,
data,
headers,
responseType: "arraybuffer",
});
const tmpPath = `/tmp/${this.archiveName}`;
fs.writeFileSync(tmpPath, response);
$.export("$summary", `Successfully compressed the files into '${tmpPath}'`);
return tmpPath;
},
async run({ $ }) {
let data = {
files: this.files,
archiveName: this.archiveName,
compressionLevel: this.compressionLevel,
password: this.password,
};
if (this.uploadType === "File") {
data = new FormData();
if (this.password) data.append("Password", this.password);
if (this.compressionLevel) data.append("CompressionLevel", this.compressionLevel);
data.append("ArchiveName", this.archiveName);
for (const file of this.files) {
data.append("Files", fs.createReadStream(file));
}
}
const headers = this.uploadType === "File"
? data.getHeaders()
: {};
try {
const response = await this.app.compressFiles({
$,
data,
headers,
responseType: "arraybuffer",
});
const tmpPath = `/tmp/${this.archiveName}`;
fs.writeFileSync(tmpPath, response);
$.export("$summary", `Successfully compressed the files into '${tmpPath}'`);
return tmpPath;
} catch (error) {
$.export("$error", `Failed to compress files: ${error.message}`);
throw error;
}
},

};
6 changes: 6 additions & 0 deletions components/zip_archive_api/common/constants.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
export default {
UPLOAD_TYPES: [
"URL",
"File",
],
};
8 changes: 6 additions & 2 deletions components/zip_archive_api/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@pipedream/zip_archive_api",
"version": "0.0.1",
"version": "0.1.0",
"description": "Pipedream Zip Archive API Components",
"main": "zip_archive_api.app.mjs",
"keywords": [
Expand All @@ -11,5 +11,9 @@
"author": "Pipedream <[email protected]> (https://pipedream.com/)",
"publishConfig": {
"access": "public"
},
"dependencies": {
"@pipedream/platform": "^1.6.5",
"form-data": "^4.0.0"
}
}
}
70 changes: 65 additions & 5 deletions components/zip_archive_api/zip_archive_api.app.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,71 @@
import { axios } from "@pipedream/platform";
import constants from "./common/constants.mjs";

export default {
type: "app",
app: "zip_archive_api",
propDefinitions: {},
propDefinitions: {
uploadType: {
type: "string",
label: "Upload Type",
description: "The upload type of the file",
options: constants.UPLOAD_TYPES,
},
archiveName: {
type: "string",
label: "Archive Name",
description: "Compressed archive name",
},
compressionLevel: {
type: "integer",
label: "Compression Level",
description: "Archive compression level. Value range: 1-9",
optional: true,
},
password: {
type: "string",
label: "Password",
description: "The compressed ZIP archive password",
optional: true,
},
files: {
type: "string[]",
label: "Files URLs",
description: "The URLs or path of the files to be compressed",
},
file: {
type: "string",
label: "File URL",
description: "The URL or path of the archive to extract the files from",
},
},
methods: {
// this.$auth contains connected account data
authKeys() {
console.log(Object.keys(this.$auth));
_baseUrl() {
return "https://api.archiveapi.com";
},
async _makeRequest(opts = {}) {
const {
$ = this,
path,
params,
...otherOpts
} = opts;

return axios($, {
...otherOpts,
url: this._baseUrl() + path,
params: {
...params,
secret: `${this.$auth.secret}`,
},
});
},
Comment on lines +46 to +62
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: _makeRequest method.

The _makeRequest method is designed to handle API requests using axios. It correctly deconstructs the opts parameter to extract necessary values and constructs the request with appropriate parameters, including authentication details.

  • Security Concern: The method includes the authentication secret directly in the request parameters. This approach might expose sensitive information in logs or to an interceptor. It's recommended to ensure that the secret is handled securely, possibly by using headers or other secure means of transmission.

  • Error Handling: There is no explicit error handling within this method. It would be beneficial to include error handling to manage and log errors from the axios request, which would improve the robustness of the API interactions.

Consider the following improvements:

  • Securely handle the authentication secret.
  • Add error handling to manage API request failures.

async compressFiles(args = {}) {
return this._makeRequest({
method: "post",
path: "/zip",
...args,
});
},
},
};
};
7 changes: 6 additions & 1 deletion pnpm-lock.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading