Good-bye Enterprise Architect - Hello Topcased!

Thanks to recent developments Karsten Thoms (Itemis, financed by the ESRF) we are now switching the EDNA UML modeler tool from Enterprise Architect to Topcased.

Enterprise Architect (EA) has played an important role in the EDNA project from its start till now. EA is a very powerful tool for UML modelling and has served it purpose well. However, the fact that EA is a commercial tool, only runs (well) on Windows and does not support the XMI standard very well has made us to look elsewhere for an EDNA UML modeller.

Since the majority of the EDNA developers use the Eclipse IDE for their developments it was felt that the ideal situation would be to have an Eclipse-based UML modeller for the EDNA developments. Peter Keller (Global Phasing), who has investigated many different solutions, recommended us to try the Topcased UML Editor. This Eclipse-based modeller is open-source and can easily be incorporated into an Eclipse installation.

Unlike EA, the Topcased UML modeller does not create XSD (XML Schema Definition) files from an UML model. These XSD files are necessary for automatically produce the data-binding objects used by the EDNA plugins. On the request of the ESRF Karsten Thoms has produced an Eclipse plugin "UML2XSD" with which it is straight forward to transform an UML model into an XSD file.

Both the kernel and the MXv1 data models have been fully transformed into Topcased UML diagrams and the EA data models are deprecated. The idea is to then slowly migrate all existing EA UML models to Topcased UML models. This migration will soon be simplified thanks to further developments by Karsten Thoms.

For more information on how practically the Topcased UML modeller is used in EDNA see: Data Modeling with the Topcased UML Editor

Minutes from the EDNA full meeting, EDNA training slides and the Kernel Code Camp summary are now available online

The minutes from the last EDNA full meeting held at the ESRF on September 23rd can be found here: EDNA Full Minutes (PDF)

On November 15th-16th an EDNA training was organised at the ESRF. The slides from this training are available: EDNA Introduction (PDF) and Edna Tutorial (PDF).

The summary from the last code camp which was dedicated to kernel developers is also available online: EDNA Kernel Code Camp November 2010 (PDF).

Announcing DIMPLE 1.0 Beta

We are pleased to announce the 1.0 Beta release of DIMPLE (DIfference Map PipeLinE).


Using the EDNA framework the DIMPLE pipeline takes a PDB file and an MTZ file and uses software included in the CCP4 suite to automatically calculate a difference density map. The results of the pipeline are a map file, a set or refined coordinates and a small script to automatically start COOT.

The pipeline has been based on extensive discussions with a number of users and an initial proof-of-concept application developed by Graeme Winter, solve-o-matic. In collaboration with CCP4 (namely Ronan Keegan and formerly George Pelios) and with grateful guidance from Mark Basham, the proof of principle application has been developed into an EDNA framework application. We have deployed this pipeline on Diamond beamlines where we will hopefully iron out any further problems but we would be delighted to hear if anybody else tries to use the pipeline and how you get on! Work is already underway to improve some of the reporting and usability aspects.

The software is made available under the LGPL v3 license and is available from the EDNA SVN or drop us an email. We will distribute with future releases of the CCP4 package. More details can be found from an early presentation at: http://www.edna-site.org/wiki/images/3/30/DIMPLE_-_Difference_Map_Pipeline.pdf

Signed
Graeme, Ronan and Alun

EDNA Full Meeting - September 23, 2010, at the ESRF

The next EDNA full meeting will take place at the ESRF on September 23rd. It will be preceded by a mxCuBE meeting and succeeded by an ISPyB meeting. For more information and registration to the EDNA meeting (and also the mxCuBE/ISPyB meetings) see:

http://www.esrf.fr/events/conferences/mxcube_edna_ispyb_meetings

EDNA Dev Kernel code camp at DLS 26-27th April

Diamond will be hosting a virtual code camp on April 26th and 27th with the aim to add some kernel upgrades and generally improve the documentation.

The areas which will be covered in detail are as follows, they are based on CCP4's first use of EDNA and also comments from DLS from the use of MXv1. This is not an exhaustive list, and any other kernel issues which could do with addressing will be welcomed and added before the meeting.

1. Documentation
2. Automatic unpacking of input objects
3. Definition of methods inside plugins
4. Allow use of state in plugins
5. Define plugin types in data model
6. Should input XML be validated
7. EDNA run folder location (Possibly an MXv1 issue, but may run deeper)

The Agenda to try to address these issues is as follows (Times are UK) :

Monday

• 09:00 - Coffee + Setup Maratech etc.
• 09:30 - Discuss Kernel changes as stated above, and divide the documentation writing tasks as Olofs plan http://www.edna-site.org/joomla/index.php?option=com_wrapper&view=wrapper&Itemid=6 Should include a code review of some DIMPLE plugins as examples of where issues came from.
• 12:00 - Lunch
• 13:00 - Open session for kernel development
• 4:00 - Code Reviews
• 14:30 - Coffee
• 15:00 - Documentathon - try to get as much doc in place in 30 mins.
• 15:30 - Open Session for kernel development
• 16:30 - Code Reviews
• 17:00 - Close

Tuesday

• 09:00 - Plugin Development (i.e. try to use the new kernel changes in your own code, also the new docs)
• 10:00 - Documentathon - try to fill in any missed bits of doc
• 10:30 - Coffee
• 11:00 - Discussion over kernel changes made, and there implementations.
• 12:00 - Lunch
• 13:00 - Final Session for Kernel Developments
• 14:30 - Code reviews
• 15:00 - Coffee
• 15:30 - Closing discussions etc plus workshop outcomes.

The whole thing will be done using Maratech, and is really aimed for seasoned Python programmers who have written EDNA plugins and are happy to help with making kernel changes.

EDNA Bugzilla server now open for anonymous read-only access

As of today the EDNA bugzilla server is open for anonymous read-only access. Note that you still have to have an account for creating new bugs and posting messages on the Bugzilla server.

Developers workshop/code camp 18 January 2010 DLS

Sorry for the short notice but we would like to announce a plugin developers meeting for EDNA on the 18-20th of January at Diamond.

The schedule of the workshop will be more of an introductory level for new developers on the Monday then more of a code camp style where tasks for kernel developments and specific plugin developments (e.g. MXv1 and V2) will be undertaken in breakout groups for the remainder of the time.

We anticipate that the workshop will be very popular so expression of interest/commitment to attend early is recommended! We will try and organise accommodation at http://www.stfc.ac.uk/About/Conts/Find/Coseners/Introduction.aspx though all travel and accommodation expenses will be the responsibility of the attendee and their home institution.

Early expressions of interest would be very much appreciated! I hope you haven't all gone away for Christmas already, if you have, I hope your having a good time!

Alun

EDNA in Java

The idea of being able to call EDNA from within java has been discussed many times, and Olof has recently produced an eclipse plug-in which can make these calls. However the way this is done is by using the command line based launchers for EDNA, and hence there is some overhead involved.

The work that I have been doing extends this idea to using java embedded python (jepp) to make the calls to the EDNA plugins directly. This allows for the plugins to be called independently without the general overhead of the framework, and also removes the need to start up a separate shell for each plug-in.

Jep has some issues with being able to pass information between java and python, however as EDNA relies on XML to pass information between different plugins, Jep can be used quite effectively.

Initial work on this topic has now been completed, and this is now working well in a test environment. The next step is to create a full API which allows the Execution of various plugins easily in the Java environment.

However there were a few points which are worth pointing out, as they took a long time to fully get working.

1. Environment variables, It is important to set both the environment variables stated, if only the LD_LIBRARY_PATH is set, most things work completely fine however there are very obscure errors with imports which take a while to track down, a small typo in the LD_PRELOAD path can cause a lot of trouble. So, if there are any issues with import statements such as "import random" this is where to check first.

2. The biggest problem is with imports, if a problem occurs in an import, then jepp will always report the error as being that the first import that was called from the jepp environment cannot be found, even if the import that failed was way down a long list of dependencies, and the error was actually something completely different. Unfortunately there seems to be no way of retrieving the actual exception, either through Python or java, so you’re down to lots of print statements to find where the problem is.

3. The last issue which caused a lot of problems, especially as it was used in a class definition of one particular import, so the error was swallowed, is that as jepp is running on the embedded platform there is no sys.argv. This means that PySys.argv also doesn't exist. This however can be worked round by as one of the first things we do in Jep is to set sys.argv to basically a sensible set of values. This just needs to be a list of strings.

Basically that's then it. It’s then quite easy to call the plugins to be run, and give it an xml string input. And the main thing is that no changes are required in the EDNA code itself. This should be easily combined with the work that Olof has done to make it very easy to use the plugins. Plus the speed of execution is very high in comparison to launching the process externally. More updates to follow, when the full API is finished, and properly integrated into the Eclipse EDNA plugin.

EDNA logo

The EDNA project has not yet any logo. The one used until recently on the EDNA Wiki was rapidly put together and meant to be a temporary logo - which has remained in its place for more than two years...

In the last developers' meeting in Grenoble I suggested a new logo in the meeting introduction :

The idea behind this logo is the following picture, which was presented for the first time in the EDNA developers' meeting at the DLS in February this year:

This is the EDNA project in a nutshell : We are trying to build a framework and applications that are modular, that share data models, implements workflows and are robust thanks to the testing framework. The base of the EDNA project / collaboration is the project management with the executive committee, project manager, project agreement, coding conventions, code reviews etc.

The proposed logo is not yet approved, so if you have any comments please add them to the blog or send them to me!

Estimation of sensitivity to radiation damage by sacrificing a crystal

The goal of this new development is to enable the estimation of radiation damage by sacrificing a crystal. The proposal of Sasha Popov and Ricardo Leal is to this it in four steps:

1. Collect one or several reference images from the crystal to be sacrificed.

2. Use EDNA characterisation and a new version of BEST (3.3) to calculate a strategy that both "fries" the crystal and collects some reference frames at regular intervals. The proposal is that this strategy consists of 10 data collection sets, where each data set consists of:

- A couple of reference frames collected at e.g. 5 % transmission

- A data set for burning the crystal with 100 % transmission, the idea is to give a certain dose to the crystal. The images of this data set are not needed in the analysis.

3. Collect this strategy using the BCM (e.g. mxCuBE)

4. Analyse the reference frames with new EDNA plugins. What I understand being the most important parameter to monitor is the increase of B-factor with dose/exposure time. The angle of the slope of this line is a measure of the crystal life time on the beamline.

I will work together with Ricardo Leal to implement step 2. The work in EDNA for step 2 should be straightforward:

- Modification of the MXv1 and BESTv1_1 data model, add "strategyOption" [XSDataString] to XSDataDiffractionPlan (bug #379).

- Update the BEST plugin to version 1.2 in order to use the strategyOption attribute for running BEST with the correct command line option (bug #384).

This new attribute obsoletes the current attribute "anomalousData" [XSDataBoolean] since the anomalousData could be passed as a string to "strategyOption". I suggest though to keep "anomalousData" in the data models in order to not break any existing plugins.

Step 4 will be developed later...

Presentations and conclusions from EDNA developers' meeting now online

The presentations from and the conclusions of the EDNA developers' meeting held at the ESRF between October 12th and 13th are now available online:

http://www.edna-site.org/wiki/index.php/EDNA_Documents

EDNA JSR publication online

As already announced on the edna mailing list, the EDNA framework article written by Marie-Françoise submitted to Journal of Synchrotron Radiation is now available online:

http://dx.doi.org/10.1107/S0909049509036681

On behalf of the EDNA collaboration I thank Marie-Françoise for all the hard work she did for the EDNA project in general and in particular for writing this article! We hope that Marie-Françoise can soon join the EDNA development team again.

Minutes from the VC of October 9th

Alun suggested in the VC today to create an EDNA Newsletter. I suggested to create a blog, however as Alun pointed out that many people don't follow blogs we agreed that the postings on the blog should also be sent to the EDNA mailing lists. So, this is the first entry in this new blog!

We agreed also in the VC today to do the following amendments to the developers' meeting agenda:

• Alun will give an overview of the possible requirements / use cases in general for workflow tools at synchrotron radiation facilities (Monday October 12th 11:40 - 12:00)
• There will be no parallel sessions on Tuesday October 13th, however the proposed agenda might be changed as a result of the discussions on Monday.
• In order to give some more time for a refreshment in town prior to the meeting dinner we decided to change the departure of the bus from the guest house to 18:30.

I will add detailed descriptions on the agenda were to find the seminar rooms.

Andrew asked which were the goals of the workflow discussions of the meeting next week, and what the impacts would be on the MX developments. I answered that the goals were to set up milestones for any eventual implementation of workflows in EDNA, and that the impacts on MX developments could only be assessed after the meeting next week.

We decided to meet again in two weeks (October 23rd) in order to revise the MX development plan as agreed in the June meeting in Hamburg.