- Content
- Introduction
- Project Folder Structure
- Initial setup
- Project Configurations
- Scripting test cases
- Scriptless keywords
- Commandline executions
- Slack Notification
- AWS CodePipeline Integration
- Docker Integration
- Credits
Thank you for using BEx framework for your automation. In the next list of items, you will be walked through things, which we have felt amazing while developing also.
├─── BEx
├          └─── Framework
├                            └─── Config
├                                          └─── FrameworkConfig.json
and TestSuite.json
├─── Project
├          └─── TestScript
├                            └─── TestCase.json
/TestCase.cs
├          └─── TestData
├                            └─── TestData.xlsx
Please follow the below steps to get onboarded to the framework:
- Create a Console Framework for .NET application
- Download the latest framework release from framework repository
- In Project -> Dependencies -> Browser -> Browser
- Browse
BEx.dll
files and add to your project - Configure Framework/Config/FrameworkConfig.json. For more details, click here
Suggested IDE : Microsoft Visual Stidio
- Add your test data files in
TestData
folder - Add your test scripts in
TestScripts
folder. For syntax related information of creating tests, click here - Build your code from Visual Studio.
- Run your tests by running
ProjectName.exe
file in your build folder.
Suggested IDE : Visual Studio Code
- Configure TestSuite.json file. For more details click here
- Add your test case files in
Project/TestScipts/
location. - Write your test cases. For more details, click here
- Run your tests by running
BEx.exe
file in your build folder.
File name is strict in nature
Below list of keywords are mandatory and cannot be removed
Keyword | Data type | Description | Possible values |
DataFilePath | String | Path for test data with file name/ test file directory. Can be abosolute or relative. This is not required if your project is fully scriptless as test data is passed from TestCase.json file. Although if you are giving test file directory, also mention TestDataFormat(below mentioned keyword) | .\\Project\\TestData\\TestData.xlsx Project\\TestData\\TestData.xlsx C:\\Users\\username\\Desktop\\ProjectName\\Project\\TestData\\TestData.xlsx ./Project/TestData/TestData.xlsx Project/TestData/TestData.xlsx C:/Users/username/Desktop/ProjectName/Project/TestData/TestData.xlsx |
TestDataFormat | String | File extension(for excel, only .xlsx file is supported) | Excel |
TestSuiteConfig | String | Location of test suite config file. Can be abosolute or relative. Only JSON files in provided sample format is accepted. | C:\\Users\\username\\Desktop\\ProjectName\\BEx\\Framework\\Config\\TestSuite.json C:/Users/username/Desktop/ProjectName/BEx/Framework/Config/TestSuite.json .\\BEx\\Framework\\Config\\TestSuite.json ./BEx/Framework/Config/TestSuite.json BEx\\Framework\\Config\\TestSuite.json BEx/Framework/Config/TestSuite.json |
TestCasePath | String | Location of test scripts. Can be abosolute or relative. Only JSON files for scriptless in provided format is accepted. | .\\Project\\TestScript\\ C:/Users/username/Desktop/ProjectName/Project/TestScript C:\\Users\\username\\Desktop\\ProjectName\\Project\\TestScript ./Project/TestScript \\Project\TestScript \\Project\\TestScript |
SlackNotify | Boolean | Set true if you want to enable notification of execution results in slack channel. Optional parameter | true, false |
ProjectName | String | Requried if SlackNotify is true. Project name | |
WebHookUrl | String | Requried if SlackNotify is true. | Your project name |
Environment | String | Requried if using Environment variables file. | DEV, TEST, UAT |
RunLocation | String | Requried if slack notify is true. Default is LOCAL | LOCAL/PIPELINE |
Tag | String[] | If you want to tag specifi list of persons in channel | ["@shubhendu.gupta", "@kanika.goyal"] |
Below are custom examples for coded
and scriptless
scripting styles
{
"DataFilePath": "Project\\TestData\\TestData.xlsx",
"TestDataFormat": "",
"TestSuiteConfig": ".\\BEx\\Framework\\Config\\TestSuite.json",
"TestCasePath": ".\\Project\\TestScript\\"
}
Below will search for TestData.xlsx
or RunManager.xlsx
{
"DataFilePath": "Project\\TestData\\",
"TestDataFormat": "Excel",
"TestSuiteConfig": ".\\BEx\\Framework\\Config\\TestSuite.json",
"TestCasePath": ".\\Project\\TestScript\\"
}
{
"DataFilePath": "Project\\TestData\\",
"TestDataFormat": "",
"TestSuiteConfig": ".\\BEx\\Framework\\Config\\TestSuite.json",
"TestCasePath": ".\\Project\\TestScript\\"
}
File name is strict in nature Below list of keywords are mandatory and cannot be removed
Keyword | Data type | Description | Possible values |
ToExecute | String | Set "Yes" if you want to execute this test case. "No" if you want to skip this test case | Yes, No, yes, no, y, n |
TestSource | String | "Coded" if test case is coded test case. "Scriptless" if your test case is scriptless | Coded, Scriptless, coded, scriptless |
Suite | String | Assign test cases to suites using "Suite" keyword | Smoke, Regression, Functional, Sanity, etc. |
DataKey | String | Keyword for data driven test cases. You can assign test cases to a data key for driving the test cases from an external souce of data(currently excel). Implementation here |
{
"TestCaseName": {
"ToExecute": "Yes",
"TestSource": "Scriptless",
"Suite": [ "Regression" ],
"DataKey": "GetCase"
},
"TC001": {
"ToExecute": "Yes",
"TestSource": "Coded"
}
}
If you want to pass sensitive data from a seperate files, you can introduce a new file Environment.json
.
This makes the FrameworkConfig.json
key Environment
as a MANDATORY item.
Configure it based on your project environments.
Below is the file format to be used:
{
"DEV": {
"BaseUrl": "https://dev.beatapps.net/"
},
"TEST": {
"BaseUrl": "https://test.beatapps.net/"
},
"UAT": {
"BaseUrl": "https://uat.beatapps.net/"
}
}
Below are the ways you can call an environment variable:
To fetch data from current working environment:
"EndPoint": "[from-env:BaseUrl]beatflow/oneapp/"
To fetch data from specific environment:
"EndPoint": "[from-env:UAT:BaseUrl]beatflow/oneapp/"
Only EndPoint and Payload is allowed to use environment variables
Pushing values to environment vairables is not yet implemented. You can use Store
instead
- Create test cases in the specified path in
FrameworkConfig.json
file. - Add new class file with name
TestCaseName.cs
. - Inherit
BEx.Framework.Base.ITestCaseBase
and implement all the methods. - Below is the format you should expect:
public void AfterSuite(Data data)
{
throw new NotImplementedException();
}
public void AfterTest(Data data)
{
throw new NotImplementedException();
}
public void BeforeSuite(Data data)
{
throw new NotImplementedException();
}
public void BeforeTest(Data data)
{
throw new NotImplementedException();
}
public void Test(Data data)
{
throw new NotImplementedException();
}
- Remove all the
throw new NotImplementedException();
autogenerated lines in methods. - Start scripting codes in formats.
- Bussiness components should reside in
root/Project/Components
folder.
- Create test cases in the specified path in
FrameworkConfig.json
file. - Register test case in
TestSuite.json
- Add new JSON file with name
TestCaseName.json
.
{
"TestCaseName": {
"ToExecute": "Yes",
"SerialNo": 1,
"StatusCode": 200,
"RequestType": "GET",
"EndPoint": "https://reqres.in/api/users?page=2",
"Headers": {
"ContentType": "application/json"
},
"ReportResponse": true
}
}
- Run
.exe
file to run the test case.
Below are the list of keywords which can be used in any required combinations for test case setup:
Keyword | Required/Optional | Description | Data type | Possible values |
ToExecute | Required | Set "Yes" if you want to execute the test step. "No" if you want ot skip this step. | String | Yes/ No/ yes/ no/ y/ n |
SerialNo | Required | Set execution sequence for this step. Cannot be repeatative. Should be consecutive in order(Cannot move from test step 1 to step 3. step 2 is required) | Number | 1 |
StatusCode | Required | Expected status code for the api call | Number | 200 |
RequestType | Required | Call type for the api | String | GET/ POST/ PUT/ DELETE |
EndPoint | Required | Api end point | String | Refer API docs for more details |
Headers | Optional | If your call wants headers to be added, add this. If no headers provided, default is "ContentType": "application/json" |
Dictionary<String, String> | "Headers": { "ContentType": "application/json" } |
Parameters | Optional | If parameters need to be passed from URL, add this to your test case | Doctionary<String, String> | { "id": "1212" } |
ReportResponse | Optional | Set true if you want to add whole response to report. Else set false. If not provided, default is false | Boolean | "ReportResponse": true |
Store | Optional | If you want to store a value from request/response payload to framework instance memory, use this key-value pair to store data. | Dictionary<String, String> | "Store": { "token": "response:root.access_token" } |
Expected | Optional, Dependant to Validate | Actual expected value from response | Array of expected values in string(strict) | ["1212", "Jack"] |
Validate | Optional, Dependant to Expected | Actual value validation from response | Array of json paths in string(string) | ["response:root.id", "response:root.firstname"] |
ShowPayload | Optional | If you want to view the payload in logs/console, set this key as true | Boolean | true, false |
CustomCode | Optional | If you want to execute any C# code, you can make use of this. For more details, click here | String | Code inside main() |
Loop | Optional | If test step repeatation should be decided over the previous execution, this is useful to this scenario. Click here for more details | {} | "Loop": { "Condition": "raw-response==true&&this.StatusCode==200", "Count": "10" } |
DownloadFileAs | Required | WARNING:This is an indication that the api call is a going to download the file | String | FileName.ext |
EnsureFile | Optional | Useful when user wants to validate the downloadd file in the download path | Boolean | true/ false |
If your test case requires to execute some complex logic, you can make use of this.
using System;
using Newtonsoft.Json.Linq;
String httpResponse; // Current response body
String Environment; // Current environment
String httpResponseHeaders; // Current http response headers
Int32 StatusCode; // Response status code
String httpRequestPayload; // Current request payload
Dynamic data can be accessed using below keyword:
sharedObjects[KEY]
For example, if we want to access access_token key, below is the code:
sharedObjects["access_token"]
If we want to store values inside the custom code to framework memory, you can store the values in a Dictionary<String, String>()
and return the dictionary values
You need to write the code content inside the Main()
For example, for parsing and printing the response JSON:
JObject obj = JObject.Parse(httpResponse);
Console.WriteLine(httpResponse);
"Loop": {
"Condition": "raw-response==true&&this.StatusCode==200",
"Count": "10"
}
Condition
: This is a raw condition which is needed to Break out from the loop
In the condition shown above, to break the loop, raw-response==true&&this.StatusCode==200
has to be satisfied
Count
: This is the maximum number of execution, the step should repeat if the condition fails. Default is 10.
If an API need to be tested over multiple sets of data/ from an external souce of data, you can make use of this. In order to data drive a test case, follow below steps:
- Create an xlsx workbook in
root/Project/Datatable
folder - Add the data key to
TestSuite.json
file as mentioned below:
"ToExecute": "Yes",
"TestSource": "Scriptless",
"Suite": [ "Regression" ],
"DataKey": "GetCase"
where GetCase
is the data key.
- Add excel sheet with same name as mentioned as DataKey. In the above case, excel sheet should be of name as
GetCase
. - Add your keyword data set in this sheet save the sheet
- Now if you dont add those keys to
TestCase.json
file also, it will not give any error. - Run your test cases.
Data validations from an API response can be categorized into 2 parts:
- Simple: Where validations are straight and no complex code logic is required
- Complex: Where validations requires code logics like loops, iterations are required
When Simple
validations are considered, we can use Validate
and Expected
keys to validate data.
When Complex
validations are considered, we can use CustomCodes
with Expected
keys.
If you want to fetch any stored value from the framework memory and want to append it to any other keyword, use
[Fetch:<JSONPath_TO_VARIABLE>]
to fetch and append.
For example, if you want to append stored value to EndPoint
keyword:
"EndPoint": "https://reqres.in/api/users?user=[Fetch:id]"
Necessary condition to use this is, you should store values before using this
If you have multipart form data to upload file, use this extension to upload the file.
Below is an example to this:
...
"Payload": {
"files": "upload-file:Uploads\\SampleFile.pdf"
}
...
If you wants to upload multiple files, you can call this keyword, but the values to upload-file:
should be array type.
Below is an example:
...
"Payload": {
"files": "upload-file:[Uploads\\SampleFile.pdf,Uploads\\SampleFile.pdf]"
}
...
If you want to pass any data from external file content, use this extension to the Keyword.
Below is an example to this:
...
"Payload": {
"user_details": "[from-file:Payloads\\Payload.txt]"
}
...
If you want to execute some custom codes within your execution, you can use this keyword
Below are some examples to use this:
...
"Payload": {
"user_details": "[from-code:Scripts\\Code.txt]"
}
...
...
"Payload": {
"from-code": "Scripts\\Code.cs"
}
...
...
"Parameters": {
"from-code": "Scripts\\Payload.cs"
}
...
For coding related documents, refer here
If you want to store data which is returned from <REQUEST/RESPONSE>
after reading <JSON_TO_VARIABLE>
,
you can use this extension to the keyword.
The stored file will be named as: TestStepName_json.path.txt
Below is an example to this:
...
"Store": {
"to-file": "response:root.id"
}
...
Above will fetch root.id
from response
and store to file with name TestStepName_root.id.txt
If the response payload is not a JSON type, you can use this extension with Store
. It can also be used with
to-file
.
Below is an example to this:
...
"Store": {
"to-file": "raw-response:root"
}
...
If you want to generate random strings of length
number of characters, you can use this.
Usage:
Below generates random string of 10 characters
"Payload": {
"name": "RandomName(10)"
}
If you want to generate datetime in mentioned format
you can use this.
Usage:
Below generates date of dddd, dd MMMM yyyy
format
"Payload": {
"name": "GetDate(\"dddd, dd MMMM yyyy\")"
}
Refer here for list of supported date time formats
If you want to data drive a test step, you can follow the below steps:
- Create an xlsx workbook in
root/Project/Datatable
folder - Add the data key to
TestSuite.json
file as mentioned below:
"ToExecute": "Yes",
"TestSource": "Scriptless",
"Suite": [ "Regression" ],
"DataKey": "GetCase"
where GetCase
is the data key.
- Add excel sheet with same name as mentioned as DataKey. In the above case, excel sheet should be of name as
GetCase
. - Add your keyword data set in this sheet save the sheet
- Now if you dont add those keys to
TestCase.json
file also, it will not give any error. - Run your test cases.
If you want to pass the test steps from a seperate files, you can use this feature.
NOTE API segregation cannot be used for only 1 API in a test case. That means if you want to segregate 1 API, in a test case, you should segregate all other APIs also in the same test case.
Belo are the steps to implement this feature:
- Add a key
ApiLibraryPath
in FrameworkConfig.json:
{
...
"ApiLibraryPath": "ApiLibrary",
...
}
- Copy the test steps to a file with the same name as the test step name in the
ApiLibraryPath
directory. That means, if your test step name isListAllValues
, file name inApiLibraryPath
directory should beListAllValues.json
- Add all the test steps in array form in your test case file.
If you want to pass a list of pre-requiste test steps which will be ran before test case execution has started,
you can use Before
in TestSuite.json
. Below is the rule:
{
"Before": "BeforeSteps",
"GetAllCases": {
"ToExecute": "Yes",
"TestSource": "Scriptless",
"Suite": [ "Regression" ],
"DataKey": "GetCase"
},
"SuccessfulTransaction": {
"ToExecute": "Yes",
"TestSource": "Scriptless",
"Suite": [ "Regression" ],
"DataKey": "PostCase"
}
}
where BeforeSteps
is a test case file, residing in TestScipts folder with same format as test cases.
Let us say your test case file looks like this:
{
"TestStep1": {
"RequestType": "GET",
"SerialNo": 1,
"ToExecute": "Yes",
"EndPoint": "[from-env:BaseUrl]api/[from-file:Endpoint\\GET_Test_With_Parameters_Store_Test.txt]",
"Parameters": {
"from-file": "Parameters\\GET_GET_Test_With_Parameters_Store_Test.json"
},
"StatusCode": 200,
"Loop": {
"Condition": "this.StatusCode==200",
"Count": "5"
},
"ReportResponse": true,
"Validate": [ "response:root.page" ],
"Expected": [ "2" ],
"Store": [
{
"page": "response:root.page"
}
]
},
"TestStep2": {
"RequestType": "GET",
"SerialNo": 1,
"ToExecute": "Yes",
"EndPoint": "[from-env:BaseUrl]api/[from-file:Endpoint\\GET_Test_With_Parameters_Store_Test.txt]",
"Parameters": {
"from-file": "Parameters\\GET_GET_Test_With_Parameters_Store_Test.json"
},
"StatusCode": 200,
"Loop": {
"Condition": "this.StatusCode==200",
"Count": "5"
},
"ReportResponse": true,
"Validate": [ "response:root.page" ],
"Expected": [ "2" ],
"Store": [
{
"page": "response:root.page"
}
]
}
}
Create files in ApiLibraryPath
as names: TestStep1
and TestStep2
and copy the test step contents to the respective files.
And atleast, change the TestCase file content from above to:
[
"TestStep1",
"TestStep2"
]
If you want to run specific test cases, you can provide the test case name as command line arguments to BEx.exe
.
The format for providing argument is provided below:
BEx --tests=TC001
If more than 1 test cases are to be provided, you can append them as comma seperated values. Below is the format:
BEx --tests=TC001,TC002,...
During the script development, we dont want to generate reports as it becomes a heavy array of auto generated files. In that cases, where you don't want to generate reports, you can use this argument like below:
BEx --toreport=false
NOTE: You can use the arguments with any combinations
If you have segregated your tests with Suite
in TestSuite.json
, you can run your tests which are tagged to some suite name.
Use below command to run your tests:
bex --suite=Smoke
To list the current version of BEx, you can use this option
BEx --version
If you want to download a specific version of the framework, they can use this option which will download the mentioned version in the libs
folder.
Usage:
If you want to download at specific path, you can add one more option --path=DownloadPath
like below:
BEx [email protected] --path=DownloadPath
You can generate a framework summary report containing number of test case and number of apis automated using this command.
BEx --summary
Latest release is capable to send execution reports in slack channel. The default notifications are sent to a
channel named #automation
. This accepts three more keys in FrameworkConfig.json
: SlackNotify
, ProjectName
, WebHookUrl
, Channel
, Tag
, RunLocation
.
This feature will look for other keys only if you provide SlackNotify
as true
.
After successful implementation, you will receive message in below format:
For creating new webhooks and modifying refer to wiki docs.
AWS Codepipeline integration is required for CI/CD pipeline integrations. Below are the list of files while are to be placed in project root:
appspec.yml
BeforeInstall.ps1
buildspec.yml
run.bat
At line number:5
change according to project name and save:
destination: C:\inetpub\wwwroot\PROJECT_NAME
At line numner:15
change according to project name and save:
Remove-Item "C:\inetpub\wwwroot\PROJECT_NAME\*" -Recurse
No change required
At line numner:1
change according to project name and save:
cd C:\inetpub\wwwroot\PROJECT_NAME\libs
For coded
scripts, change line number:2
to your project .exe
file.
Please ensure you have docker instaled in your local system
Below are the list of files while are to be placed in project root:
- Dockerfile
- run.bat
At line number:3, 4, 5
modify based on following and save:
ADD Framework/ inetpub/wwwroot/PROJECT_NAME/Framework
ADD libs/ inetpub/wwwroot/PROJECT_NAME/libs
ADD Project/ inetpub/wwwroot/PROJECT_NAME/Project
Format | Result |
DateTime.Now.ToString("MM/dd/yyyy") | 05/29/2015 |
DateTime.Now.ToString("dddd, dd MMMM yyyy") | Friday, 29 May 2015 |
DateTime.Now.ToString("dddd, dd MMMM yyyy") | Friday, 29 May 2015 05:50 |
DateTime.Now.ToString("dddd, dd MMMM yyyy") | Friday, 29 May 2015 05:50 AM |
DateTime.Now.ToString("dddd, dd MMMM yyyy HH:mm:ss") | Friday, 29 May 2015 05:50:06 |
DateTime.Now.ToString("MM/dd/yyyy HH:mm") | 05/29/2015 05:50 |
DateTime.Now.ToString("MM/dd/yyyy hh:mm tt") | 05/29/2015 05:50 AM |
DateTime.Now.ToString("MM/dd/yyyy H:mm") | 05/29/2015 5:50 |
DateTime.Now.ToString("MM/dd/yyyy h:mm tt") | 05/29/2015 5:50 AM |
DateTime.Now.ToString("MM/dd/yyyy HH:mm:ss") | 05/29/2015 05:50:06 |
DateTime.Now.ToString("MMMM dd") | May 29 |
DateTime.Now.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss.fffffffK") | 2015-05-16T05:50:06.7199222-04:00 |
DateTime.Now.ToString("ddd, dd MMM yyy HH':'mm':'ss 'GMT'") | Fri, 16 May 2015 05:50:06 GMT |
DateTime.Now.ToString("yyyy'-'MM'-'dd'T'HH:'mm':'ss") | 2015-05-16T05:50:06 |
DateTime.Now.ToString("HH:mm") | 05:50 |
DateTime.Now.ToString("hh:mm tt") | 05:50 AM |
DateTime.Now.ToString("H:mm") | 5:50 |
DateTime.Now.ToString("h:mm tt") | 5:50 AM |
DateTime.Now.ToString("HH:mm:ss") | 05:50:06 |
DateTime.Now.ToString("yyyy MMMM") | 2015 May |
d
-> Represents the day of the month as a number from 1 through 31.
dd
-> Represents the day of the month as a number from 01 through 31.
ddd
-> Represents the abbreviated name of the day (Mon, Tues, Wed, etc).
dddd
-> Represents the full name of the day (Monday, Tuesday, etc).
h
-> 12-hour clock hour (e.g. 4).
hh
-> 12-hour clock, with a leading 0 (e.g. 06)
H
-> 24-hour clock hour (e.g. 15)
HH
-> 24-hour clock hour, with a leading 0 (e.g. 22)
m
-> Minutes
mm
-> Minutes with a leading zero
M
-> Month number(eg.3)
MM
-> Month number with leading zero(eg.04)
MMM
-> Abbreviated Month Name (e.g. Dec)
MMMM
-> Full month name (e.g. December)
s
-> Seconds
ss
-> Seconds with leading zero
t
-> Abbreviated AM / PM (e.g. A or P)
tt
-> AM / PM (e.g. AM or PM
y
-> Year, no leading zero (e.g. 2015 would be 15)
yy
-> Year, leading zero (e.g. 2015 would be 015)
yyy
-> Year, (e.g. 2015)
yyyy
-> Year, (e.g. 2015)
K
-> Represents the time zone information of a date and time value (e.g. +05:00)
z
-> With DateTime values represents the signed offset of the local operating system's time zone from
Coordinated Universal Time (UTC), measured in hours. (e.g. +6)
zz
-> As z, but with leading zero (e.g. +06)
zzz
-> With DateTime values represents the signed offset of the local operating system's time zone from UTC, measured in hours and minutes. (e.g. +06:00)
f
-> Represents the most significant digit of the seconds' fraction; that is, it represents the tenths of a second in a date and time value.
ff
-> Represents the two most significant digits of the seconds' fraction in date and time
fff
-> Represents the three most significant digits of the seconds' fraction; that is, it represents the milliseconds in a date and time value.
ffff
-> Represents the four most significant digits of the seconds' fraction; that is, it represents the ten-thousandths of a second in a date and time value. While it is possible to display the ten-thousandths of a second component of a time value, that value may not be meaningful.
fffff
-> Represents the five most significant digits of the seconds' fraction; that is, it represents the hundred-thousandths of a second in a date and time value.
ffffff
-> Represents the six most significant digits of the seconds' fraction; that is, it represents the millionths of a second in a date and time value.
fffffff
-> Represents the seven most significant digits of the second's fraction; that is, it represents the ten-millionths of a second in a date and time value.
- Shubhendu Shekhar Gupta