Sorry for the short notice but we would like to announce a plugin developers meeting for EDNA on the 18-20th of January at Diamond.
The schedule of the workshop will be more of an introductory level for new developers on the Monday then more of a code camp style where tasks for kernel developments and specific plugin developments (e.g. MXv1 and V2) will be undertaken in breakout groups for the remainder of the time.
We anticipate that the workshop will be very popular so expression of interest/commitment to attend early is recommended! We will try and organise accommodation at http://www.stfc.ac.uk/About/Conts/Find/Coseners/Introduction.aspx though all travel and accommodation expenses will be the responsibility of the attendee and their home institution.
Early expressions of interest would be very much appreciated! I hope you haven't all gone away for Christmas already, if you have, I hope your having a good time!
Alun
Friday, December 18, 2009
Tuesday, December 1, 2009
EDNA in Java
The idea of being able to call EDNA from within java has been discussed many times, and Olof has recently produced an eclipse plug-in which can make these calls. However the way this is done is by using the command line based launchers for EDNA, and hence there is some overhead involved.
The work that I have been doing extends this idea to using java embedded python (jepp) to make the calls to the EDNA plugins directly. This allows for the plugins to be called independently without the general overhead of the framework, and also removes the need to start up a separate shell for each plug-in.
Jep has some issues with being able to pass information between java and python, however as EDNA relies on XML to pass information between different plugins, Jep can be used quite effectively.
Initial work on this topic has now been completed, and this is now working well in a test environment. The next step is to create a full API which allows the Execution of various plugins easily in the Java environment.
However there were a few points which are worth pointing out, as they took a long time to fully get working.
1. Environment variables, It is important to set both the environment variables stated, if only the LD_LIBRARY_PATH is set, most things work completely fine however there are very obscure errors with imports which take a while to track down, a small typo in the LD_PRELOAD path can cause a lot of trouble. So, if there are any issues with import statements such as "import random" this is where to check first.
2. The biggest problem is with imports, if a problem occurs in an import, then jepp will always report the error as being that the first import that was called from the jepp environment cannot be found, even if the import that failed was way down a long list of dependencies, and the error was actually something completely different. Unfortunately there seems to be no way of retrieving the actual exception, either through Python or java, so you’re down to lots of print statements to find where the problem is.
3. The last issue which caused a lot of problems, especially as it was used in a class definition of one particular import, so the error was swallowed, is that as jepp is running on the embedded platform there is no sys.argv. This means that PySys.argv also doesn't exist. This however can be worked round by as one of the first things we do in Jep is to set sys.argv to basically a sensible set of values. This just needs to be a list of strings.
Basically that's then it. It’s then quite easy to call the plugins to be run, and give it an xml string input. And the main thing is that no changes are required in the EDNA code itself. This should be easily combined with the work that Olof has done to make it very easy to use the plugins. Plus the speed of execution is very high in comparison to launching the process externally. More updates to follow, when the full API is finished, and properly integrated into the Eclipse EDNA plugin.
The work that I have been doing extends this idea to using java embedded python (jepp) to make the calls to the EDNA plugins directly. This allows for the plugins to be called independently without the general overhead of the framework, and also removes the need to start up a separate shell for each plug-in.
Jep has some issues with being able to pass information between java and python, however as EDNA relies on XML to pass information between different plugins, Jep can be used quite effectively.
Initial work on this topic has now been completed, and this is now working well in a test environment. The next step is to create a full API which allows the Execution of various plugins easily in the Java environment.
However there were a few points which are worth pointing out, as they took a long time to fully get working.
1. Environment variables, It is important to set both the environment variables stated, if only the LD_LIBRARY_PATH is set, most things work completely fine however there are very obscure errors with imports which take a while to track down, a small typo in the LD_PRELOAD path can cause a lot of trouble. So, if there are any issues with import statements such as "import random" this is where to check first.
2. The biggest problem is with imports, if a problem occurs in an import, then jepp will always report the error as being that the first import that was called from the jepp environment cannot be found, even if the import that failed was way down a long list of dependencies, and the error was actually something completely different. Unfortunately there seems to be no way of retrieving the actual exception, either through Python or java, so you’re down to lots of print statements to find where the problem is.
3. The last issue which caused a lot of problems, especially as it was used in a class definition of one particular import, so the error was swallowed, is that as jepp is running on the embedded platform there is no sys.argv. This means that PySys.argv also doesn't exist. This however can be worked round by as one of the first things we do in Jep is to set sys.argv to basically a sensible set of values. This just needs to be a list of strings.
Basically that's then it. It’s then quite easy to call the plugins to be run, and give it an xml string input. And the main thing is that no changes are required in the EDNA code itself. This should be easily combined with the work that Olof has done to make it very easy to use the plugins. Plus the speed of execution is very high in comparison to launching the process externally. More updates to follow, when the full API is finished, and properly integrated into the Eclipse EDNA plugin.
Friday, November 20, 2009
EDNA logo
The EDNA project has not yet any logo. The one used until recently on the EDNA Wiki was rapidly put together and meant to be a temporary logo - which has remained in its place for more than two years...
In the last developers' meeting in Grenoble I suggested a new logo in the meeting introduction :
The idea behind this logo is the following picture, which was presented for the first time in the EDNA developers' meeting at the DLS in February this year:
This is the EDNA project in a nutshell : We are trying to build a framework and applications that are modular, that share data models, implements workflows and are robust thanks to the testing framework. The base of the EDNA project / collaboration is the project management with the executive committee, project manager, project agreement, coding conventions, code reviews etc.
The proposed logo is not yet approved, so if you have any comments please add them to the blog or send them to me!
In the last developers' meeting in Grenoble I suggested a new logo in the meeting introduction :
The idea behind this logo is the following picture, which was presented for the first time in the EDNA developers' meeting at the DLS in February this year:
This is the EDNA project in a nutshell : We are trying to build a framework and applications that are modular, that share data models, implements workflows and are robust thanks to the testing framework. The base of the EDNA project / collaboration is the project management with the executive committee, project manager, project agreement, coding conventions, code reviews etc.The proposed logo is not yet approved, so if you have any comments please add them to the blog or send them to me!
Estimation of sensitivity to radiation damage by sacrificing a crystal
The goal of this new development is to enable the estimation of radiation damage by sacrificing a crystal. The proposal of Sasha Popov and Ricardo Leal is to this it in four steps:
1. Collect one or several reference images from the crystal to be sacrificed.
2. Use EDNA characterisation and a new version of BEST (3.3) to calculate a strategy that both "fries" the crystal and collects some reference frames at regular intervals. The proposal is that this strategy consists of 10 data collection sets, where each data set consists of:
- A couple of reference frames collected at e.g. 5 % transmission
- A data set for burning the crystal with 100 % transmission, the idea is to give a certain dose to the crystal. The images of this data set are not needed in the analysis.
3. Collect this strategy using the BCM (e.g. mxCuBE)
4. Analyse the reference frames with new EDNA plugins. What I understand being the most important parameter to monitor is the increase of B-factor with dose/exposure time. The angle of the slope of this line is a measure of the crystal life time on the beamline.
I will work together with Ricardo Leal to implement step 2. The work in EDNA for step 2 should be straightforward:
- Modification of the MXv1 and BESTv1_1 data model, add "strategyOption" [XSDataString] to XSDataDiffractionPlan (bug #379).
- Update the BEST plugin to version 1.2 in order to use the strategyOption attribute for running BEST with the correct command line option (bug #384).
This new attribute obsoletes the current attribute "anomalousData" [XSDataBoolean] since the anomalousData could be passed as a string to "strategyOption". I suggest though to keep "anomalousData" in the data models in order to not break any existing plugins.
Step 4 will be developed later...
1. Collect one or several reference images from the crystal to be sacrificed.
2. Use EDNA characterisation and a new version of BEST (3.3) to calculate a strategy that both "fries" the crystal and collects some reference frames at regular intervals. The proposal is that this strategy consists of 10 data collection sets, where each data set consists of:
- A couple of reference frames collected at e.g. 5 % transmission
- A data set for burning the crystal with 100 % transmission, the idea is to give a certain dose to the crystal. The images of this data set are not needed in the analysis.
3. Collect this strategy using the BCM (e.g. mxCuBE)
4. Analyse the reference frames with new EDNA plugins. What I understand being the most important parameter to monitor is the increase of B-factor with dose/exposure time. The angle of the slope of this line is a measure of the crystal life time on the beamline.
I will work together with Ricardo Leal to implement step 2. The work in EDNA for step 2 should be straightforward:
- Modification of the MXv1 and BESTv1_1 data model, add "strategyOption" [XSDataString] to XSDataDiffractionPlan (bug #379).
- Update the BEST plugin to version 1.2 in order to use the strategyOption attribute for running BEST with the correct command line option (bug #384).
This new attribute obsoletes the current attribute "anomalousData" [XSDataBoolean] since the anomalousData could be passed as a string to "strategyOption". I suggest though to keep "anomalousData" in the data models in order to not break any existing plugins.
Step 4 will be developed later...
Thursday, October 22, 2009
Presentations and conclusions from EDNA developers' meeting now online
The presentations from and the conclusions of the EDNA developers' meeting held at the ESRF between October 12th and 13th are now available online:
http://www.edna-site.org/wiki/index.php/EDNA_Documents
http://www.edna-site.org/wiki/index.php/EDNA_Documents
EDNA JSR publication online
As already announced on the edna mailing list, the EDNA framework article written by Marie-Françoise submitted to Journal of Synchrotron Radiation is now available online:
http://dx.doi.org/10.1107/S0909049509036681
On behalf of the EDNA collaboration I thank Marie-Françoise for all the hard work she did for the EDNA project in general and in particular for writing this article! We hope that Marie-Françoise can soon join the EDNA development team again.
http://dx.doi.org/10.1107/S0909049509036681
On behalf of the EDNA collaboration I thank Marie-Françoise for all the hard work she did for the EDNA project in general and in particular for writing this article! We hope that Marie-Françoise can soon join the EDNA development team again.
Friday, October 9, 2009
Minutes from the VC of October 9th
Alun suggested in the VC today to create an EDNA Newsletter. I suggested to create a blog, however as Alun pointed out that many people don't follow blogs we agreed that the postings on the blog should also be sent to the EDNA mailing lists. So, this is the first entry in this new blog!
We agreed also in the VC today to do the following amendments to the developers' meeting agenda:
Andrew asked which were the goals of the workflow discussions of the meeting next week, and what the impacts would be on the MX developments. I answered that the goals were to set up milestones for any eventual implementation of workflows in EDNA, and that the impacts on MX developments could only be assessed after the meeting next week.
We decided to meet again in two weeks (October 23rd) in order to revise the MX development plan as agreed in the June meeting in Hamburg.
We agreed also in the VC today to do the following amendments to the developers' meeting agenda:
- Alun will give an overview of the possible requirements / use cases in general for workflow tools at synchrotron radiation facilities (Monday October 12th 11:40 - 12:00)
- There will be no parallel sessions on Tuesday October 13th, however the proposed agenda might be changed as a result of the discussions on Monday.
- In order to give some more time for a refreshment in town prior to the meeting dinner we decided to change the departure of the bus from the guest house to 18:30.
Andrew asked which were the goals of the workflow discussions of the meeting next week, and what the impacts would be on the MX developments. I answered that the goals were to set up milestones for any eventual implementation of workflows in EDNA, and that the impacts on MX developments could only be assessed after the meeting next week.
We decided to meet again in two weeks (October 23rd) in order to revise the MX development plan as agreed in the June meeting in Hamburg.
Subscribe to:
Posts (Atom)
Posts
