Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OPENAI_KEY as env variable and extra tests #27

Merged
merged 10 commits into from
Apr 29, 2024
Merged

OPENAI_KEY as env variable and extra tests #27

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
37b40b6
Update ci.yml
vpapanasta Apr 24, 2024
a3f3440
Update ci.yml
vpapanasta Apr 24, 2024
9cf4e4c
Update topenAIChat.m
vpapanasta Apr 24, 2024
a62530b
Update topenAIChat.m
vpapanasta Apr 24, 2024
751b348
Update topenAIChat.m
vpapanasta Apr 24, 2024
fa61f1f
Update topenAIChat.m
vpapanasta Apr 24, 2024
807b778
Update topenAIMessages.m
vpapanasta Apr 25, 2024
3975f2c
Update textractOpenAIEmbeddings.m
vpapanasta Apr 25, 2024
857fd45
Update topenAIImages.m
vpapanasta Apr 25, 2024
78279d2
Update topenAIChat.m
vpapanasta Apr 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ jobs:
products: Text_Analytics_Toolbox
cache: true
- name: Run tests and generate artifacts
env:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OPENAI_API_KEY to match the env variable that we expect?

We might have to remove this part from the test:

function saveEnvVar(testCase)
           % Ensures key is not in environment variable for tests
           openAIEnvVar = "OPENAI_API_KEY";
           if isenv(openAIEnvVar)
               key = getenv(openAIEnvVar);
               unsetenv(openAIEnvVar);
               testCase.addTeardown(@(x) setenv(openAIEnvVar, x), key);
           end
       end

I have added that because when I was running local tests on some behaviors related to checking the key it would turn out that the key was already in my path. We should just move that to where it's actually needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For local debugging we should still use the manually defined OPENAI_API_KEY. I do not think it harms to keep this check to make sure that OPENAI_API_KEY is not in environment variable for tests. OPENAI_KEY defined in .github/workflows/ci.yml is a different environment variable and it is secret.

uses: matlab-actions/run-tests@v2
with:
test-results-junit: test-results/results.xml
Expand Down
63 changes: 52 additions & 11 deletions tests/topenAIChat.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ function saveEnvVar(testCase)
end

properties(TestParameter)
InvalidConstructorInput = iGetInvalidConstructorInput;
InvalidGenerateInput = iGetInvalidGenerateInput;
InvalidValuesSetters = iGetInvalidValuesSetters;
InvalidConstructorInput = iGetInvalidConstructorInput();
InvalidGenerateInput = iGetInvalidGenerateInput();
InvalidValuesSetters = iGetInvalidValuesSetters();
end

methods(Test)
Expand All @@ -34,11 +34,8 @@ function generateAcceptsMessagesAsInput(testCase)
chat = openAIChat(ApiKey="this-is-not-a-real-key");
messages = openAIMessages;
messages = addUserMessage(messages, "This should be okay.");
testCase.verifyWarningFree(@()generate(chat,messages));
end

function constructMdlWithInvalidParameters(testCase)
testCase.verifyError(@()openAIChat(ApiKey="this-is-not-a-real-key", ModelName="gpt-4", ResponseFormat="json"), "llms:invalidOptionAndValueForModel");
testCase.verifyWarningFree(@()generate(chat,messages));
end

function keyNotFound(testCase)
Expand All @@ -59,6 +56,7 @@ function constructChatWithAllNVP(testCase)
chat = openAIChat(systemPrompt, Tools=functions, ModelName=modelName, ...
Temperature=temperature, TopProbabilityMass=topP, StopSequences=stop, ApiKey=apiKey,...
FrequencyPenalty=frequenceP, PresencePenalty=presenceP, TimeOut=timeout);

testCase.verifyEqual(chat.ModelName, modelName);
testCase.verifyEqual(chat.Temperature, temperature);
testCase.verifyEqual(chat.TopProbabilityMass, topP);
Expand All @@ -69,27 +67,37 @@ function constructChatWithAllNVP(testCase)

function verySmallTimeOutErrors(testCase)
chat = openAIChat(TimeOut=0.0001, ApiKey="false-key");

testCase.verifyError(@()generate(chat, "hi"), "MATLAB:webservices:Timeout")
end

function errorsWhenPassingToolChoiceWithEmptyTools(testCase)
chat = openAIChat(ApiKey="this-is-not-a-real-key");

testCase.verifyError(@()generate(chat,"input", ToolChoice="bla"), "llms:mustSetFunctionsForCall");
end

function settingToolChoiceWithNone(testCase)
functions = openAIFunction("funName");
chat = openAIChat(ApiKey="this-is-not-a-real-key",Tools=functions);

testCase.verifyWarningFree(@()generate(chat,"This is okay","ToolChoice","none"));
end

function settingSeedToInteger(testCase)
chat = openAIChat(ApiKey="this-is-not-a-real-key");

testCase.verifyWarningFree(@()generate(chat,"This is okay", "Seed", 2));
end

function invalidInputsConstructor(testCase, InvalidConstructorInput)
testCase.verifyError(@()openAIChat(InvalidConstructorInput.Input{:}), InvalidConstructorInput.Error);
end

function invalidInputsGenerate(testCase, InvalidGenerateInput)
f = openAIFunction("validfunction");
chat = openAIChat(Tools=f, ApiKey="this-is-not-a-real-key");

testCase.verifyError(@()generate(chat,InvalidGenerateInput.Input{:}), InvalidGenerateInput.Error);
end

Expand All @@ -107,18 +115,43 @@ function invalidGenerateInputforModel(testCase)
image_path = "peppers.png";
emptyMessages = openAIMessages;
inValidMessages = addUserMessageWithImages(emptyMessages,"What is in the image?",image_path);

testCase.verifyError(@()generate(chat,inValidMessages), "llms:invalidContentTypeForModel")
end

function noStopSequencesNoMaxNumTokens(testCase)
chat = openAIChat(ApiKey="this-is-not-a-real-key");

testCase.verifyWarningFree(@()generate(chat,"This is okay"));
end

function createOpenAIChatWithStreamFunc(testCase)
sf = @(x)fprintf("%s", x);
chat = openAIChat(ApiKey="this-is-not-a-real-key", StreamFun=sf);

testCase.verifyWarningFree(@()generate(chat, "Hello world."));
end
vpapanasta marked this conversation as resolved.
Show resolved Hide resolved

function warningJSONResponseFormatGPT35(testCase)
chat = @() openAIChat("You are a useful assistant", ...
ApiKey="this-is-not-a-real-key", ...
ResponseFormat="json", ...
ModelName="gpt-3.5-turbo");

testCase.verifyWarning(@()chat(), "llms:warningJsonInstruction");
vpapanasta marked this conversation as resolved.
Show resolved Hide resolved
end

function createOpenAIChatWithOpenAIKey(testCase)
chat = openAIChat("You are a useful assistant", ...
ApiKey=getenv("OPENAI_KEY"));

testCase.verifyWarningFree(@()generate(chat, "Hello world."));
end

end
end

function invalidValuesSetters = iGetInvalidValuesSetters
function invalidValuesSetters = iGetInvalidValuesSetters()

invalidValuesSetters = struct( ...
"InvalidTemperatureType", struct( ...
Expand Down Expand Up @@ -222,7 +255,7 @@ function noStopSequencesNoMaxNumTokens(testCase)
"Error", "MATLAB:notGreaterEqual"));
end

function invalidConstructorInput = iGetInvalidConstructorInput
function invalidConstructorInput = iGetInvalidConstructorInput()
validFunction = openAIFunction("funName");
invalidConstructorInput = struct( ...
"InvalidResponseFormatValue", struct( ...
Expand All @@ -233,6 +266,10 @@ function noStopSequencesNoMaxNumTokens(testCase)
"Input",{{"ResponseFormat", ["text" "text"] }},...
"Error", "MATLAB:validation:IncompatibleSize"), ...
...
"InvalidResponseFormatModelCombination", struct( ...
"Input", {{"ApiKey", "this-is-not-a-real-key", "ModelName", "gpt-4", "ResponseFormat", "json"}}, ...
"Error", "llms:invalidOptionAndValueForModel"), ...
...
"InvalidStreamFunType", struct( ...
"Input",{{"StreamFun", "2" }},...
"Error", "MATLAB:validators:mustBeA"), ...
Expand Down Expand Up @@ -366,7 +403,7 @@ function noStopSequencesNoMaxNumTokens(testCase)
"Error","MATLAB:validators:mustBeTextScalar"));
end

function invalidGenerateInput = iGetInvalidGenerateInput
function invalidGenerateInput = iGetInvalidGenerateInput()
emptyMessages = openAIMessages;
validMessages = addUserMessage(emptyMessages,"Who invented the telephone?");

Expand Down Expand Up @@ -409,5 +446,9 @@ function noStopSequencesNoMaxNumTokens(testCase)
...
"InvalidToolChoiceSize",struct( ...
"Input",{{ validMessages "ToolChoice" ["validfunction", "validfunction"] }},...
"Error","MATLAB:validators:mustBeTextScalar"));
"Error","MATLAB:validators:mustBeTextScalar"),...
...
"InvalidSeed",struct( ...
"Input",{{ validMessages "Seed" "2" }},...
"Error","MATLAB:validators:mustBeNumericOrLogical"));
end