Aug
09

IUCR 2011

I will be at the IUCR meeting in Madrid if anybody want to chat with me about computing/packaging/whatever, they are welcome :)

I’ll be presenting a poster on Linear transformations of variance/covariance matrices. abstract nº 1593 in MS62 Crystallographic Software: the Library Approach on August 27-28.

Permanent link to this article: http://blog.debroglie.net/2011/08/09/iucr-2011/

Aug
06

Playing with postfix/dovecot

I am hijacking my own blog…

I need to add spamassassin now and therefore I need some junk, here is for you little bad bots: test@genea4p.net

Permanent link to this article: http://blog.debroglie.net/2011/08/06/playing-with-postfixdovecot/

Jul
27

Platon nightly

New package!

Platon has a very fast releases which make things difficult to keep the package up to date. There is now an automatic build (build at 3am GMT+1) of platon for fedora 13,14 and 15 and for centos 5 and 6. The build on centos 6 is broken at the moment though.

The package name is platon-nightly in debroglie-nightly repository and does not interfere with the platon rpm. The usual platon rpm will remain.

To activate the new repository, you need to update debroglie-release. It will also update the repo link to the new server.

Debroglie repository

Permanent link to this article: http://blog.debroglie.net/2011/07/27/platon-nightly/

Jul
26

Code optimisation

A while ago, I discussed a bit about code optimisation using valgrind:
Spot bottlenecks. At the end I obtained something like 1sec for about 4000 reflections. Meaning about 250-500us/refl.

In the meantime, I needed a much faster version. I am using Monte Carlo simulation which is very demanding.

Here is the old code:

Original Fcalc subroutine


The part on the right about string was dealing with the symmetry operations. This was really ugly. A bad habbit from associated array in pythons/php :) Strings operations are very slow compared to number. I am using integer indices now, problem solved.

The matmul_r4 in the middle is dealing with the expansion of the coordinate into P1. But this expansion is constant for every reflection at this time I did not bother avoiding this expansion for each reflection although the result is exactly the same. Now I am caching the result of the first calculation for reuse. I also merged the matrices operations by using stacked vectors so I can use a fast blas/lapack library (acml) without too much overhead.

The Kronecker product was also quite demanding. I am now caching the result at the beginning of the program.

Now the result: 70ms for 5700 reflections. That’s 13us/refl, 20 times faster. On a quad core processor, so 52us on one thread. Furthermore the code is doing slightly more things than the old one using the matrix of constraints, variables allocations…

Here is the result in kcachegrind:

Fcalc subroutine fully optimised

This version is now fully optimised. The large majority of CPU is used computing fortran intrinsics like exponential and atan2. The rest is really minor. My only chance now would be to use faster trigonometric functions. I tried the fastexp in the acml library but I did not see any improvement. Anyway, this is now fast enough.

Fortran code:

! Code under the GPL v3 license
 
complex(kind=8) function structfact(h,k,l, cifatomiccoordinates, cifatomicuaniso, newcoordinatespool, newanisopool, f2)
use unitcellmod
use datamod, ONLY: cifatomnametable, f1f2table, wavelength, listsymm, pi, twopi2, twopi
use datamod, ONLY: numberofatoms, listsymsize, cachekronecker
use datamod, only: cachesymmatrix, cachetranslation,cifsiteatomicnumber
implicit none
 
integer, dimension(3,3) :: sym
double precision, dimension(3,1) :: translation
double precision, dimension(2,1) :: f1f2
integer i,j, atomindex
integer, intent(in) :: h,k,l
logical, intent(in) :: f2
complex(kind=8) oldaf, af
double precision, dimension(:,:), intent(in) :: cifatomiccoordinates
double precision, dimension(:,:), intent(in) :: cifatomicuaniso
! data to cache some results
real(kind=8), dimension(:,:,:), allocatable, intent(inout) :: newcoordinatespool, newanisopool
real(kind=8) :: tempr1, temph2cela2, tempk2celb2, templ2celc2, hkab, lkbc, hlac
real(kind=8) :: phi, z, anisoscale
real(kind=8) :: resolution
 
if(.not. (allocated(newcoordinatespool) .and. allocated(newanisopool))) then
    !$OMP CRITICAL
    if(.not. (allocated(newcoordinatespool) .and. allocated(newanisopool))) then
        allocate(newcoordinatespool(3,numberofatoms,listsymsize))
        allocate(newanisopool(9,numberofatoms,listsymsize))
        do i=1,listsymsize
            sym=cachesymmatrix(i,:,:)
            translation=cachetranslation(i,:,:)
 
            do j=1,numberofatoms
                newcoordinatespool(:,j,i)=translation(:,1)
            end do
            call DGEMM('N','N',3,numberofatoms,3,1.0D0,real(sym,8),3,&
            &   cifatomiccoordinates(1:3,1:numberofatoms),3,1.0D0,&
            &   newcoordinatespool(:,:,i),3)
            !print *, newcoordinatespool(1:3,1,i)
 
            !newaniso=matmul(cachekronecker(:,:,i),cifatomicuaniso(1:9,2:2))
            !print *, newaniso(1:3,1)
            call DGEMM('N','N',9,numberofatoms,9,1.0D0,real(cachekronecker(:,:,i),8),9,&
            &   cifatomicuaniso(1:9,1:numberofatoms),9,0.0D0,newanisopool(:,:,i),9)
            !print *, newanisopool(:,2,i)
        end do
    end if
    !$OMP END CRITICAL
end if
 
structfact=cmplx(0.0,0.0,8)
! temp variable to avoid unnecessary calculations
temph2cela2=h**2*rcela**2
tempk2celb2=k**2*rcelb**2
templ2celc2=l**2*rcelc**2
hkab=h*k*rcela*rcelb
lkbc=l*k*rcelb*rcelc
hlac=h*l*rcela*rcelc
resolution=dataresolution(dhkl(h,k,l))
 
Do j=1,numberofatoms
    atomindex=cifsiteatomicnumber(j)
!        f' and f''
    f1f2=f1f2table(1:2,atomindex:atomindex)
 
    tempr1=cifatomiccoordinates(6,j)*&
    &   exp(real(-4.0, 8)*twopi2*cifatomiccoordinates(5,j)*(resolution)**2)/&
    &   cifatomiccoordinates(4,j)
 
!       saving the atomic factor for later use
    oldaf=atomicfactor(atomindex, resolution, f1f2, f2)
 
    do i=1,listsymsize
        af = oldaf
 
!		#anisotropic scale factor
!		# 11 22 33 23 13 12
        anisoscale=exp(-twopi2*(newanisopool(1,j,i)*temph2cela2 +     &
&           newanisopool(5,j,i)*tempk2celb2 +     &
&           newanisopool(9,j,i)*templ2celc2 +     &
&           2.0d0*newanisopool(2,j,i)*hkab + &
&           2.0d0*newanisopool(6,j,i)*lkbc + &
&           2.0d0*newanisopool(3,j,i)*hlac))
 
        phi = twopi*(real(h,8)*newcoordinatespool(1,j,i)+  &
&           real(k,8)*newcoordinatespool(2,j,i)+          &
&           real(l,8)*newcoordinatespool(3,j,i))
 
        phi=phi+atan2(aimag(af),real(af,8))
        phi=mod(phi,real(twopi,8))
 
        z=abs(af)
        af=z*exp(cmplx(0.0,1.0,8)*phi)*tempr1*anisoscale
        structfact=structfact+af
    enddo
enddo
return
end function

Permanent link to this article: http://blog.debroglie.net/2011/07/26/code-optimisation/

Jul
16

New server

This blog is hosted on a new server. The other services will slowly migrate as well.

Permanent link to this article: http://blog.debroglie.net/2011/07/16/new-server/

Feb
21

Transformers

Transformers is a python script which can apply symmetry operations or linear maps on crystallographic parameters (coordinates, ADPs, eigenvalues and their esds). It uses the full variance/covariance matrix so the standard uncertainties is correctly calculated.

The calculation is exact, meaning that if you re-refine the structure in the transformed basis, you’ll get the same esds.

I used it to expand some structures to P1 for normal coordinate analysis with NKA. It can be very useful to transpose data between different models.

Source file

Dependencies:
numpy (http://numpy.scipy.org/)
CifFile (http://pycifrw.berlios.de/)
optparse
scipy (http://www.scipy.org/)

help (very basic)
transformers.py –help

Permanent link to this article: http://blog.debroglie.net/2011/02/21/transformers/

Jan
21

Crystallographic Software Fayre 2011

At the IUCr Congress 2011 in Madrid there will again be a Software Fayre, where authors of academic and/or open-source software can present their new developments. The Software Fayre will be held from Tuesday, 23-8-2011 until Sunday, 28-8-2011. The presentations should have a tutorial character with one or more practical examples. There will also be additional “install sessions”, where software authors will help with the installation of their software.

To help with the planning of the Software Fayre, we ask potential speakers to show their interest before 28-2-2011. Later applications are possible, but the early information indicates, how many time-slots we will need and which general topics are to be expected.

If you are interested, please send an e-mail with your name and a preliminary title of the presentation to Martin Lutz, m.lutz@uu.nl.

The website of the Software Fayre will be updated regularly:
http://www.cryst.chem.uu.nl/lutz/software_fayre.html

For further information on the IUCr congress, see:
http://www.iucr2011madrid.es

With kind regards,
Martin Lutz (Organizer Software Fayre)
Lourdes Infantes (Organizer Software Fayre)

Permanent link to this article: http://blog.debroglie.net/2011/01/21/crystallographic-software-fayre-2011/

Jan
09

Update Olex2 and Olex2-GUI

After a recent test of Olex2-GUI rpm it became clear that the update mechanism I had used was not working correctly causing new installs to appear broken. This was exacerbated by a path typo in the Olex2 startup script installed from the Olex2-binaries.

I have now fixed both issues and moved the usettings.dat file to the GUI.

The current olex2-gui is pulled from the branch/1.1 svn and updated from a working version. It also includes all of the script codes which with the appropriate entries in the usettings.dat and modification of etc/scripts/customScripts.py to read:
#import example
import batch_overlay
import OlexSir
import OlexPlaton
import OlexMail
import OlexCheckCIF
import OlexCDS

(This will be made default in the next update).
Will activate these following commands:
spy.OlexCDS()
spy.OlexMail()
spy.OlexPlaton()
spy.OlexSir()
spy.OlexCheckCIF()
spy.flipsmall()

This update however does break the sync between olex2 and the repo we are neither a tag nor their beta or alpha release. Because of this I have disabled the internal olex2 userspace update mechanism updates will come via the GUI.rpm for now until they catch up with us.

Permanent link to this article: http://blog.debroglie.net/2011/01/09/update-olex2-and-olex2-gui/

Jan
09

Platon, CCTBX, Olex2, Olex2-GUI

New rpms for CCTBX, Olex2 and Olex2-GUI have been uploaded to the testing repo (only takes 7 hours to do this) only to find that the CCTBX had a bug so yet another CCTBX was uploaded yesterday (6 hours) to resolve it.

All Centos 5 build was using the Epel 5 extras python 2.6 environment (Centos 6 via native 2.7).

The new CCTBX and Olex2 binaries bring us pretty close to the official 1.1.4 release which is imminent but for the GUI. The GUI is proving to be contentious so I am planning on changing how the GUI rpm is created and how it updates the user space (wish me luck on this). As it stands Olex2 will work but the native refinement package in the 1.1.4 bins and CCTBX is too advanced for the GUI (go figure) it is a rather big mess!

On the bright side there is a new version of Platon in the release area.

J
(and yes I have been very rubbish at posting news updates here – mainly because I do not want to move Pascal’s lovely posts down the list – how nice are they?)

Permanent link to this article: http://blog.debroglie.net/2011/01/09/platon-cctbx-olex2-olex2-gui/

Nov
14

How to cut a slice in a 3D grid

Let define a 3D-grid, the pixels are not necessarily cubic. This grid contains the electron density resulting from a fourier transform. On pixel define a basis A(a1,a2,a3) of the grid. the length of the grid cover the unit cell in 3 directions.

The task is to cut a 2D-grid plane in this 3D object. Then, a contour plot can be drawn using the plane. A figure showing the transform between two 2D grids shows the dificulty of the task.

We need first to define the plane with 3 points P, Q and R (atoms) lying in this plane. The first thing to do is to choose an origin and define an orthonormal basis S(s1,s2,s3) for it. Let be P the origin of the basis. The A basis is not orthonormal, the ortohonormalisation matrix O is first use to get the coordinates of P, Q and R in an orthonormal coordinate system AO (ao1,ao2,ao3). The reason is the use of the cross product to define the basis of the plane.

The basis (s1,s2,s3) of the plane is define as:
\(\overrightarrow{s1}=\overrightarrow{Q}-\overrightarrow{P}\)
\(\overrightarrow{s2}=(\overrightarrow{Q}-\overrightarrow{P})\times (\overrightarrow{R}-\overrightarrow{P}) \)
\(\overrightarrow{s3}=\overrightarrow{s1} \times \overrightarrow{s2} \)

Now, we can define a tensor than going to transform orthonormal crystal coordinates into the new basis of the plane. This tensor operates on from the basis AO to S.

\( T = \begin{bmatrix}
s1_x^{ao} & s2_x^{ao} & s3_x^{ao} \\
s1_y^{ao} & s2_y^{ao} & s3_y^{ao} \\
s1_z^{ao} & s2_z^{ao} & s3_z^{ao} \\
\end{bmatrix} \)

The two tensors T and O are going to map the pixels from the 3D-grid to the 2D-grid. The next step is to define the grid fot the 2D-plane. Let define a plane width a width w of 16 ang and a ratio width/height r of 16/10. The resolution of the grid will be 0.1 ang, meaning a 160×100 grid plane. The pixel (i,j) will have the coordinates (x,y,0) in S with:

\( x = i \cdot w/(160/\|\overrightarrow{s1^{ao}}\| ) \)
\( y = j \cdot (w/r)/(100/\|\overrightarrow{s2^{ao}}\| ) \)

Now we map the coordinates of each (i,j) pixel plane to the coordinates (u,v,w) in the 3D-grid.

\( (u,v,w) = \mathbf{O}^{-1} ( ( \mathbf{T}^{-t} \begin{bmatrix}x \\ y \\ 0 \end{bmatrix} ) + \overrightarrow{P^{ao}} ) \)

This was the easy part. The (u,v,w) coordinates need to be matched to a pixel in the 3D-grid. To go from fractional coordinates to the pixel index, it is just necessary to multiply by the length in grid size in each direction. The problem is that the result is not an integer and an interpolation is needed. The easiest interpolation is to the nearest pixel, practically, we round the result (u,v,w) to the nearest integer. An other easy solution is to use tri-linear interpolation. Platon is using a more complex quadratic or cubic interpolation.

The resolution of the grid plane is arbitrary and have to be chosen carefully according to the original grid resolution. In case of wide angle and a cut through the diagonal, there is a loss of resolution. For example with a 0.3 ang grid resolution in the 3D-grid, the maximum resolution will be about 0.7 ang with a diagonal cut in the trigonal space group. 0.3 ang is the resolution used by platon. Hence the use of heavy interpolation in order to smooth the contour plot. Using a linear interpolation, a minimum resolution of 0.1 ang is necessary, 0.05 is perfect. It represents a 3D-grid of about 1M-8M pixels. The fftw library for the fourier transform handle this quite easily (0.2s-1.5s).

Interpolation to the nearest integer of a 0.3 ang 3D-grid :

High resolution 3D-grid of 0.05 ang. The 2D-grid resolution is kept at 0.01 ang to avoid losing any information and a linear interpolation has been used.

Permanent link to this article: http://blog.debroglie.net/2010/11/14/how-to-cut-a-slice-in-a-3d-grid/

Older posts «

» Newer posts

Performance Optimization WordPress Plugins by W3 EDGE