Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream Files missing from Feature Extraction window #446

Open
jepa88 opened this issue Jan 29, 2021 · 9 comments
Open

Stream Files missing from Feature Extraction window #446

jepa88 opened this issue Jan 29, 2021 · 9 comments

Comments

@jepa88
Copy link

jepa88 commented Jan 29, 2021

Hello and thank you for this tool!
I am currently learning how to use the tool so I am using some example videos.
When I am using the annotation ability of the tool, I can see that I have two videos. Please see attached image
Smiling_database
However, when I open the feature extraction tab I no longer see these videos
Extract_Features
The same when opening the training tab
train_models
The two videos are located in a folder /data/name_of_database/name_of_session/ and they are named as role.video_name.mp4
Could you please explain how I should fix this?

@tobiasbaur
Copy link
Member

Hi! Thanks for raising this.

Indeed it is not yet described well in the documentation and we are hosting only a limited amount of features on github for now. We plan to add additional ones to the interface, but at this time you need to add them manually.

Make sure you point your cml directory in the database settings also to the cml folder. If you only downloaded the exe you will have to add the CML folder from here https://github.com/hcmlab/nova/tree/master/bin/cml

As you are intending to recognise smiles, i recommend using a featureset for facial features, such as openface. We are currently not hosting this publicly on github but I added a nova-ready-to-use openface2 (https://github.com/TadasBaltrusaitis/OpenFace) component here:

https://mediastore.rz.uni-augsburg.de/get/N1IsN68cRx/
(link might expire at some point)

Please see Licence instructions in the Openface Git.

Make sure you extract the content of this folder so you have the structure nova/cml/chains/video/openface2.

In the first screenshot you posted, under "streams" -> edit make sure the files have the type "video" (needs to match the cml/chains/video folder structure). and that the samplerate of the video is correct (e.g 25 or 30 frames per second)

Now you should see the video and the openface2 chain. When you extract the features for the first time, it will download all necessary files. It should then create the stream files automatically and add them to the database. (might take some time depending on the length of your video and the performance of your pc).

For the training process, make sure you have the the cml/models/templates/.. folder checked out. also note that you will need at least one annotation in the database to train the model. (You can try to start with linsvm which is very fast)

Please let me know if this helps, or if you need further assistance.

Best,
Tobias Baur

@jepa88
Copy link
Author

jepa88 commented Feb 2, 2021

Thank you very much for this detailed answer. It helped a lot
I was able to proceed and extract features but when I tried to train the model the nova application crushes
I have created some annotations for the videos shown in the following figure.
image

Could you please point at some directions to what to look at to solve this issue?
I am using the latest release NOVA 1.0.7.1 on windows 10.
Thank you in advance

@tobiasbaur
Copy link
Member

Hi, does this also happen on the 30hz stream? I could image it may be a problem with the 29.99 and 29.98 streams. (try to add it as 30hz and extract the features again on these).

If that is not the problem: Do you get any error before the crash? did you download the cml tools (via ? -> update cml tools)

@jepa88
Copy link
Author

jepa88 commented Feb 5, 2021

This issue happens also with the 30hz video.
I changed the video settings from 29.99 and 29.98 to 30hz, I extracted features again but the tool crashes.
Unfortunately, I do not get any error: When I select the stream with the extracted features and the session and I click on 'train', the mouse point changes into 'loading' and the program is killed without any warning message to debug it.
I clicked on 'update cml tools' and I got the message that the tools are already up to date.
Could you let me know, what else I should look for?
Thank you very much for your help.

@tobiasbaur
Copy link
Member

hi, depending on how long you have been using NOVA you might want to update the cml tools (?-> Update CML tools) and, let the dialog delete the dlls, so they are redownloaded. In case there is no update option, please delete the cmltrain.exe and perform this step again. Please let me know if that helps.

@jepa88
Copy link
Author

jepa88 commented Feb 15, 2021

Thank you very much for your feedback.
I tried your suggestions but unfortunately nova keeps exiting without any message.
Attached I upload a video of the procedure I follow to train the classifier after I have extracted features. Each session contains a video with annotations. I tried linsvm, svm, sklearn decision tree and sklearn svm.
https://user-images.githubusercontent.com/78227691/107977305-27aabb00-6fbb-11eb-8e6f-cd09412d9635.mp4

@jepa88
Copy link
Author

jepa88 commented Feb 17, 2021

Dear Dr Baur, I checked the event viewer on windows and I attach the message I get. From my understanding, a folder is missing.

Application: nova.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.DirectoryNotFoundException
at System.IO.__Error.WinIOError(Int32, System.String)
at System.IO.Directory.InternalCreateDirectory(System.String, System.String, System.Object, Boolean)
at System.IO.Directory.InternalCreateDirectoryHelper(System.String, Boolean)
at ssi.DatabaseCMLTrainAndPredictWindow.Apply_Click(System.Object, System.Windows.RoutedEventArgs)
at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean)
at System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject, System.Windows.RoutedEventArgs)
at System.Windows.Controls.Primitives.ButtonBase.OnClick()
at System.Windows.Controls.Button.OnClick()
at System.Windows.Controls.Primitives.ButtonBase.OnMouseLeftButtonUp(System.Windows.Input.MouseButtonEventArgs)
at System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate, System.Object)
at System.Windows.RoutedEventHandlerInfo.InvokeHandler(System.Object, System.Windows.RoutedEventArgs)
at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean)
at System.Windows.UIElement.ReRaiseEventAs(System.Windows.DependencyObject, System.Windows.RoutedEventArgs, System.Windows.RoutedEvent)
at System.Windows.UIElement.OnMouseUpThunk(System.Object, System.Windows.Input.MouseButtonEventArgs)
at System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate, System.Object)
at System.Windows.RoutedEventHandlerInfo.InvokeHandler(System.Object, System.Windows.RoutedEventArgs)
at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean)
at System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject, System.Windows.RoutedEventArgs)
at System.Windows.UIElement.RaiseTrustedEvent(System.Windows.RoutedEventArgs)
at System.Windows.Input.InputManager.ProcessStagingArea()
at System.Windows.Input.InputManager.ProcessInput(System.Windows.Input.InputEventArgs)
at System.Windows.Input.InputProviderSite.ReportInput(System.Windows.Input.InputReport)
at System.Windows.Interop.HwndMouseInputProvider.ReportInput(IntPtr, System.Windows.Input.InputMode, Int32, System.Windows.Input.RawMouseActions, Int32, Int32, Int32)
at System.Windows.Interop.HwndMouseInputProvider.FilterMessage(IntPtr, MS.Internal.Interop.WindowMessage, IntPtr, IntPtr, Boolean ByRef)
at System.Windows.Interop.HwndSource.InputFilterMessage(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)
at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef)
at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame)
at System.Windows.Window.ShowHelper(System.Object)
at System.Windows.Window.ShowDialog()
at ssi.MainHandler.showDialogClearWorkspace(System.Windows.Window)
at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean)
at System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject, System.Windows.RoutedEventArgs)
at System.Windows.Controls.MenuItem.InvokeClickAfterRender(System.Object)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
at System.Windows.Threading.DispatcherOperation.InvokeImpl()
at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Windows.Threading.DispatcherOperation.Invoke()
at System.Windows.Threading.Dispatcher.ProcessQueue()
at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)
at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef)
at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame)
at System.Windows.Application.RunDispatcher(System.Object)
at System.Windows.Application.RunInternal(System.Windows.Window)
at ssi.App.Main()

@tobiasbaur
Copy link
Member

Hello, yes

Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.DirectoryNotFoundException
at System.IO.__Error.WinIOError(Int32, System.String)
at System.IO.Directory.InternalCreateDirectory(System.String, System.String, System.Object, Boolean)
at System.IO.Directory.InternalCreateDirectoryHelper(System.String, Boolean)

this looks like NOVA tries to create a new folder (probably in cml/models/trainer/....)
but maybe it does not have the rights to do so. Is the cml path in some restricted folder? (E.g. C:Programs) Did you try to run it as admin? (Depending on the configuration of your machine, there might be some problem with writing rights)

I can have a look at the code as well, just never ran into this before, so might be system specific.

@jepa88
Copy link
Author

jepa88 commented Feb 18, 2021

Thank you very much for your reply. I had tried to run it as admin and it does not work either. The path is not in a restricted folder, I also moved it to another location to make sure that it is not a path issue.
I really appreciate all your help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants