Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SynchWeb software configuration issues #809

Open
Baokangwen opened this issue Jul 21, 2024 · 27 comments
Open

SynchWeb software configuration issues #809

Baokangwen opened this issue Jul 21, 2024 · 27 comments

Comments

@Baokangwen
Copy link

@stuartcampbell @ndevenish @markbasham @jlmuir @isikhar
Dear everyone,
I have some questions during the configuration of Synchweb below.

1.The data collected by our beamline station is stored locally, how should I get the data source?

2.Whether the data source is obtained depends on the configuration(config.php) here

# Auth type
# Can be cas, ldap, oidc
$authentication_type = 'cas';

# CAS url (if using cas, assume https)
$cas_url = 'cas.server.ac.uk';

# Follow CAS SSO
$cas_sso = true;
$sso_url = "sso.server.ac.uk";

# OIDC (or OAuth2) client ID and secret. Only useful if authentication_type is set to OIDC
$oidc_client_id = "oidcClientId";
$oidc_client_secret = "oidcClientSecret";
# Cookie key used for SSO/cookie based authentication
$cookie_key = "synchweb-auth";

# CAS CA Cert (for SSO)
$cacert = '/etc/certs/ca-bundle.crt';

3.Does the syncweb software include data processing functions, can I use the data processing software to process the results first, and then use the syncweb to display the data, do I need to configure some other software that provides data processing?

@ndg63276
Copy link
Collaborator

That part of the config file is to do with authentication, not the data. Data just needs to be mounted at the path given in the database.
If you run processing software, you need to push the results into the database to see it in Synchweb.

@Baokangwen
Copy link
Author

@ndg63276
Thank you.
I used data software to process the data and generated a lot of processing files, which require me to write scripts to extract the parameters? There are other github repositories that can do this.

@ndg63276
Copy link
Collaborator

Yes, for example, https://github.com/DiamondLightSource/ispyb-api

@Baokangwen
Copy link
Author

@ndg63276
1721665571600
As shown in Fig.
Is this step of authentication required to be configured?

1721665728780
It doesn't exist when I request the database person.
The lack of authentication is the reason why my test results cannot be displayed on the web.

@ndg63276
Copy link
Collaborator

You can set

$authentication_type = 'dummy';

to test, then any password will work as long as the username is valid. If you want to use simple, you will need to add a password column to the Person table.

@Baokangwen
Copy link
Author

@ndg63276
Yes,I did it to you.
Databases also add a password columm to Person table.
This problem still cannot be solved.
7d32bc114df0265480c57cbb4d851cd

@ndg63276
Copy link
Collaborator

Ok, so you need to set

$authentication_type = 'simple';

and I presume you have populated some rows of the Person table with login and password.

If you want more help you need to show either the server log or the dev console in your browser, I'm not psychic.

@Baokangwen
Copy link
Author

@ndg63276
The image below shows my run log and the output of the browser console.
Is this RSA authentication necessary, I have configured Ldap, can I use Ldap instead of RSA? I don't know what to do here, this question has been bothering me for a long time.
9d11ec8a59c6aad4332a62673f9359f
c27580ac679a28f6711ed6f99649f66

@ndg63276
Copy link
Collaborator

ndg63276 commented Jul 23, 2024

There is code for LDAP included, you would have to set $authentication_type = 'ldap' and then set all these variables:

$ldap_server
$ldap_search
$ldap_use_tls
$ldap_server_type
$active_directory_domain

However I am not an LDAP expert nor am I familiar with your LDAP settings so I cannot help much with that.

I strongly suggest you set $authentication_type = 'dummy' and start from there, when you know other things are working, you can turn on Simple or LDAP authentication.

You might need to paste all your config.php file here (apart from database password obviously) so I can see if I can replicate what you have.

@Baokangwen
Copy link
Author

@ndg63276
Dear Mark,
We use PHP5.4 environment, can we use simple authentication?
Another question is how does SyncWeb integrate with MXCabe and EDNA?

Thank you for your continued guidance, really much appreciated.

@ndg63276
Copy link
Collaborator

I don't see why you cannot use simple authentication, it just looks at the login and password columns from the Person table. Not the most secure but good for getting started.

Synchweb doesn't do anything with MXCube, we use GDA here, and GDA talks directly to the ISPyB database. We see EDNA results in Synchweb, but again, they have to be pushed into the database by something running the EDNA process.

@KarlLevik
Copy link
Member

KarlLevik commented Jul 23, 2024

For integration with MXCube, you might want to check with Elettra, the Italian synchrotron, because they have been using SynchWeb and MXCube together. It's not really a question of integrating SynchWeb with MXCube, but rather how to make MXCube write to the ISPyB database without the old Java/Wildfly webservices (unless you want the pain of installing, configuring and maintaining those as well). I believe Elettra was using our ispyb-api together with custom code.

@Baokangwen
Copy link
Author

There is code for LDAP included, you would have to set $authentication_type = 'ldap' and then set all these variables:

$ldap_server
$ldap_search
$ldap_use_tls
$ldap_server_type
$active_directory_domain

However I am not an LDAP expert nor am I familiar with your LDAP settings so I cannot help much with that.

I strongly suggest you set $authentication_type = 'dummy' and start from there, when you know other things are working, you can turn on Simple or LDAP authentication.

You might need to paste all your config.php file here (apart from database password obviously) so I can see if I can replicate what you have.

I only made changes to the database login in to my config.php file and $authentication_type, and I barely changed the other configurations
6fbfb1fd8de0d7f24befad2862e626a

@Baokangwen
Copy link
Author

Baokangwen commented Jul 24, 2024

@ndg63276
I've changed the authentication_type = 'dummy';
And in the dummy.php file, modify the authenticate function to

 function authenticate($login, $password)
    {
       $file  = '/tmp/log.txt';
       $content = "ddddddddddd\n";

       if($f  = file_put_contents($file, $content,FILE_APPEND)){
        echo "dddddddddd<br />";
        }
        return true;
    }

But when I click on the page to log in, there is no log.txt file in the /tmp folder.

Access on the dev console debug interface in my browser (https://10.30.62.223:8082/client/api/users/current) response is

<head>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=0" />
    <meta name="viewport" content="initial-scale=1.0"/>
    
    <title>ISPyB</title>

    <!-- Add Google Analytics if defined -->
    

<script defer src="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e-bundle.js"></script><script defer src="/dist/da53c52/main-bundle.js"></script><link href="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e.css" rel="stylesheet"><link href="/dist/da53c52/main.css" rel="stylesheet"></head>

<body>
    <div id="synchweb-app"></div>
</body>

Access on https://10.30.62.223:8082/client/api/authenticate
response is

<head>

    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=0" />
    <meta name="viewport" content="initial-scale=1.0"/>
    
    <title>ISPyB</title>

    <!-- Add Google Analytics if defined -->
    

<script defer src="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e-bundle.js"></script><script defer src="/dist/da53c52/main-bundle.js"></script><link href="/dist/da53c52/vendors-node_modules_backbone-validation_dist_backbone-validation-amd_js-node_modules_backbon-9d0c0e.css" rel="stylesheet"><link href="/dist/da53c52/main.css" rel="stylesheet"></head>

<body>
    <div id="synchweb-app"></div>
</body>

Access on https://10.30.62.223:8082/client/api/authenticate this url,php The authenticate function in the Dummpy.php file should be called, but from the above, is it not called?

My config.json is as follows
{
"apiurl": "/client/api",

"appurl": "/client",

"production": false,

"pucks": {
    "i02": 10,
    "i03": 23,
    "i04": 37,
    "i04-1": 9,
    "i24": 9,
    "i23": 4
},

"_gsMajorAxisOrientation" : "Determines whether the major grid scan axis determines the orientation of the view",
"gsMajorAxisOrientation"  : true,

"maintenance_message": "This is the maintenance message",
"maintenance": false,

"ga_ident": "",

"_data_catalogue_comment": " Remove the data_catalogue object if you don't want a link on the landing page",
"data_catalogue": {
            "name": "Data Catalogue",
            "url": "http://10.30.62.223:8080"
},

"site_name": "my site",
"site_link": "https://www.mysite.com"

}

@ndg63276
Copy link
Collaborator

If I add your code to my Dummy.php file, I get the /tmp/log.txt file as expected. You can use the error_log('debug message') function to add debug messages.

Dont worry about the config.json file for now, only the config.php file.

@Baokangwen
Copy link
Author

@ndg63276
Dear Mark
I have two questions for me below,
1.The git commit record says that the dockerfile supports php7.x, but the actual dockerfile content does not support php7.x.

2.Also, in the dockerfile deployment, how do we need to change the api_url field in the config_sample.json, whether its value is /client/api, or /api?

Looking forward to hearing from you.

Best Regards
Kangwen

@ndg63276
Copy link
Collaborator

The Dockerfile is not up to date, have a look at https://github.com/DiamondLightSource/synchweb-devel-env for PHP7 versions.

My apiurl field is just /api.

@Baokangwen
Copy link
Author

@ndg63276
Dear Mark

Now we visit /Clint/Login and will redirect directly to authorise, showing Status: Reddy Rectin Tokas. The authentication mode of the background configuration is dummy.Do you know what the reason is?

My front-end code also tried to block the authentication judgment of login routes in src/js/app/router/router.js, but it didn't work, I don't know why.

The change I made was to delete the 62 lines, change value for false,the original value is true.
019e4d78854140d77ccf7b8334a35f1

@ndg63276
Copy link
Collaborator

I don't know what "Reddy Rectin Tokas" means, can you double check the spelling?

@Baokangwen
Copy link
Author

@ndg63276
sorry mark.
That is status:redirecting to cas

@ndg63276
Copy link
Collaborator

Make sure to set $cas_sso = false;.

Also delete $summarydbconfig and $ifsummary if you haven't already.

@Baokangwen
Copy link
Author

@ndg63276
Thank you mark.
I've been able to display all the data in the database.
I'd like to ask you what kind of software you use to collect data and data processing software.
73221aa5b56fed768799e203c042828

@ndg63276
Copy link
Collaborator

ndg63276 commented Aug 22, 2024

The 2 boxes you have outlined in red are meant to show a snapshot of the crystal, and a jpeg version of the diffraction image.

The crystal snapshot is just the contents of the xtalSnapshotFullPath1 field of the DataCollection table, just set it to a .jpg or .png file.

The diffraction jpeg you have to generate yourself outside of Synchweb, using something like mosflm. You can also generate a thumbnail version if you like. Then you have to set these values in the config.php file to say where the jpegs are stored:

$jpeg_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.jpeg';
$jpeg_thumb_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.thumb.jpeg';

If you don't want a separate thumbnail, just set them to the same thing. For your example screenshot above, it would expect a file at /path/to/data/cm14451-1/jpegs/20160122/gw/ins2/001/ins2_2_0001.jpeg

@Baokangwen
Copy link
Author

@ndg63276
Dear mark,

The pipeline2ispyb.py scripts in this library(https://github.com/DiamondLightSource/ispyb-api) are too old, and there will be a lot of database table-related errors when I run them, I don't know if you have a relatively new case script, thank you.

Best regards,
Kangwen

@ndg63276
Copy link
Collaborator

I only deal with Synchweb, not how data is inserted into the database. But I'm sure there are many ways to insert data into a database, the MariaDB docs (eg https://mariadb.com/kb/en/insert/) can help you.

@Baokangwen
Copy link
Author

@ndg63276
Dear Mark,
image
When I encounter a problem, I would like to ask how to upload the data of the CBF file to the database and display it on the web, do we need to write and parse the content of the CBF file by ourselves, or ISPYB-API can parse it by other ways.

The same problem as the CBF file is that our data processing software, such as: autoproc, xia2-xds, are run alone, and the data will be processed after getting the data, I can get the file of the result of data processing, or the log file, the results in the file also need to be parsed by myself, is there any script that can help me parse and automatically upload to the database, this problem has been bothering me for a long time, my own data has not been able to upload to the database, Can you give me some advice?

Thank you mark.

@ndg63276
Copy link
Collaborator

ndg63276 commented Oct 9, 2024

Hello, when you generate the CBF image, you can use a separate program like mosflm to generate a jpeg version of the diffraction image. You then need to set these variables in config.php:

$jpeg_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.jpeg';
$jpeg_thumb_location = '<%=VISITDIR%>/jpegs/<%=IMDIRECTORY%>/<%=IMFILE%>.thumb.jpeg';

Then it would expect your jpegs to be in some location like:

/wherever/you/put/your/data/cm14451-2/jpegs/gw/20160418/thau/edna_test/thau_2_0001.jpeg
/wherever/you/put/your/data/cm14451-2/jpegs/gw/20160418/thau/edna_test/thau_2_0001.thumb.jpeg

based on your example screenshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants