Continue reading »]]> New subject for this post. Since my new position at the crm2 in Nancy, I am working with optics.

We set up an experiment on interferences and optical diffraction for demonstration. I am looking for a free open source software to sketch them. Nothing dedicating to optics exists so I must use something else. I have seen two options so far: a drawing software like inkscape or a geometry software like geogebra.

I tried geogebra for the diffraction of an optical disc. The main disavantage is taht there is no easy access to proportionality like in a refraction and must use equivalent geometry construction.

For example, the refraction law is : \(n_1 \sin(\theta_1) = n_2 \sin(\theta_2) \)

To draw it you need to use cirles to get access to the sine of the angle. It complicates the drawing a bit.

For teaching purpose, geogebra can be really nice because of its interactive drawing. You can introduce variables and allow points to be moved. It is also possible to export the drawing as a java applet to publish it on the Internet. I have done it for optical diffraction through a polycarbonate medium (i.e. a CD or DVD). Click on the picture below to open the applet.

For an optical set up, it is difficult to see if it is efficient, there should not be too much of the problem above, most of the times, only reflections and transmissions are used. But the laser has a certain width and I don’t know if there is any easy way to handle focusing elements.

For a conclusion, I would say that geogebra or a similar software is worth to have a look for optics. It may not be useable for every situation but it can have some use.

Continue reading »]]> Part of a study on the standard deviation of the electron density using Monte Carlo simulations, I was needing a fast way to calculate structure factors. I achieve this goal a while ago but I continued to improve things each time when I got a new idea to try.

I already talked about it a few times:

• How to spot bottlenecks
• Code optimisation
• CPU starvation
• Loop tiling
• Auto vectorization

My algorithm is no longer memory bounded and the cpu is not starving anymore. Since I am calculating reflections by batches the memory access must be more efficient. This is a nice side effect of the modifications I have made. I checked it by running the program at 2 different cpu frequency and with different cores used. It all scale up nicely. I tried up to 4 threads, the limit of my actual cpu.

At this date, I am reaching 60-75M reflections*atoms per second and per core. On a 200 atoms structure with 100k reflections the result comes in less than 600ms.

The valgrind profile is shown below.

Compared to cctbx it’s almos 4 times faster. However cctbx is much more versatile, that’s probably the reason it’s slower. This efficiency is only possible with optimised trigonometric and BLAS functions. Without these, the calculations are a bit slower than cctbx. BLAS is only efficient on big vectors or matrices so I had to increase the number of reflections processed by batches from 128 to 1024.

This interesting thing about this batch process with BLAS functions is that it should be compatible for a GPU implementation. Though Fortran binding are not widely accessible yet and BLAS functions might not be as efficient as with the cpu. The number of reflections processed simultaneously would probably need to be increased. In the end it should be beneficial only on several hundred of thousands reflections. This range can only be found in protein crystallography.

It turns out that it can be useful to someone else and I am in the process of releasing only this part as a library to allow more people to play with it. This high level of perfomance relies on the AMD core mathematical library both for the vectorized trigonometric function (vrsa_sincos – 64bit only!) and the BLAS functions (SAXPY mainly). It should be possible to use mkl as well.

Actually the interface is very crude and uses several arrays. Object oriented data has not been implemented, data won’t be contigous anymore and would affect the efficiency. Difficult to say if the result would be very different or not.

The project is here:
libFc repository

The code produce both single precision and double precision routines accessible via generic interfaces. Via conditional compilation it is possible to select the vector math library and switch to blas functions.

Continue reading »]]> Vectorization is a nice concept in optimisation. It allows simulaneous calculations using special units in the processor. It is working only on special cases. I tried to apply to structure factors calculations. Fortran does not allow you to write vector code as you would do in C. You can only rely on autovectorization from the compiler by exposing known patterns.

Actually, each structure factor is calculated one by one by each thread. I modified the function to process them by batch allowing the compiler to do a better job.

1. Current version. 3 threads, structure factors calculated one by one.

Source code

[pascal@vinci ediff]$ perf stat -v -d  ../../trunk-r607/edensgrid --nopeaks  xu5015_shelxl
 Calculating structure factors
100.00 % done, remaining:   0:00min, elapsed:   0:01min, rate:   13.3 us/loop
101507 reflections processed in   1.4 s (201 atoms)
 Performance counter stats for '../../trunk-r607/edensgrid --nopeaks xu5015_shelxl':
 
       7210,750407 task-clock                #    1,664 CPUs utilized
               926 context-switches          #    0,000 M/sec
                18 CPU-migrations            #    0,000 M/sec
             7 404 page-faults               #    0,001 M/sec
    20 294 956 191 cycles                    #    2,815 GHz                     [24,85%]
      stalled-cycles-frontend
      stalled-cycles-backend
    27 473 509 707 instructions              #    1,35  insns per cycle         [37,42%]
     4 759 056 611 branches                  #  659,995 M/sec                   [37,49%]
        96 886 241 branch-misses             #    2,04% of all branches         [37,76%]
     8 209 247 003 L1-dcache-loads           # 1138,473 M/sec                   [25,22%]
       166 461 565 L1-dcache-load-misses     #    2,03% of all L1-dcache hits   [25,24%]
        88 992 738 LLC-loads                 #   12,342 M/sec                   [25,09%]
           710 262 LLC-load-misses           #    0,80% of all LL-cache hits    [24,87%]
 
       4,333275750 seconds time elapsed

1. New version. 3 threads, structure factors calculated by batches of 32 of them.

Source code

[pascal@vinci ediff]$ perf stat -v -d  ../../trunk/edensgrid --nopeaks  xu5015_shelxl
 Calculating structure factors
100.00 % done, remaining:   0:00min, elapsed:   0:01min, rate:   11.1 us/loop
101507 reflections processed in   1.1 s (201 atoms)
        3.6175E+07 reflections*atoms*s^-1
 
 Performance counter stats for '../../trunk/edensgrid --nopeaks xu5015_shelxl':
 
       6510,431679 task-clock                #    1,584 CPUs utilized
               904 context-switches          #    0,000 M/sec
                20 CPU-migrations            #    0,000 M/sec
             6 750 page-faults               #    0,001 M/sec
    18 147 024 815 cycles                    #    2,787 GHz                     [25,20%]
      stalled-cycles-frontend
      stalled-cycles-backend
    26 969 494 651 instructions              #    1,49  insns per cycle         [37,78%]
     4 751 540 537 branches                  #  729,835 M/sec                   [37,76%]
        76 868 617 branch-misses             #    1,62% of all branches         [37,75%]
     7 161 160 309 L1-dcache-loads           # 1099,952 M/sec                   [25,11%]
        36 037 316 L1-dcache-load-misses     #    0,50% of all L1-dcache hits   [25,07%]
        16 708 632 LLC-loads                 #    2,566 M/sec                   [24,88%]
           568 664 LLC-load-misses           #    3,40% of all LL-cache hits    [24,99%]
 
       4,110610561 seconds time elapsed

The result is an increase of around 15-25% in speed. L1 data cache misses dropped a lot and I have more instructions per cycle executed. The pressure is pushed back on the L2 cache. Not all portions have been vectorized as I would have expect, some more insight using the flag -ftree-vectorizer-verbose will help.

Best results have been obtained with batches of 16 or 32 structure factors.

Result from the compiler analysis confirmed that more parts have been vectorized. However, the same code compiled without vectorization is still 10% faster. Meaning that a bit more is going on.

Before:

Source code

modules/functions.f90:922: note: LOOP VECTORIZED.
modules/functions.f90:922: note: LOOP VECTORIZED.
modules/functions.f90:905: note: vectorized 2 loops in function.

After:

Source code

modules/functions.f90:852: note: LOOP VECTORIZED.
modules/functions.f90:842: note: LOOP VECTORIZED.
modules/functions.f90:834: note: LOOP VECTORIZED.
modules/functions.f90:839: note: LOOP VECTORIZED.
modules/functions.f90:825: note: LOOP VECTORIZED.
modules/functions.f90:827: note: LOOP VECTORIZED.
modules/functions.f90:782: note: vectorized 6 loops in function.
 
edensgrid.F90:226: note: LOOP VECTORIZED.
edensgrid.F90:226: note: LOOP VECTORIZED.
edensgrid.F90:207: note: vectorized 2 loops in function.

I am now able to process nearly 50 millions reflections*atoms per second. On a structure with about 200 atoms that’s more than 100k reflections per seconds. The results I obtained are very close to the ones obtained by Vincent Favre-Nicolin (pdf) but I am using a much more complicated formula suitable for small molecule crystallography.

I don’t know if anyone would need such a fast calculation but they can always get in touch with me.

Continue reading »]]> This post is the continuation of the previous one on: CPU starvation

I needed to rewrite a part of the code and it was a good opportunity to change the algorithm of the variance calculation. I was using the naive implementation: the first pass is the calculation of the mean, the second pass the calculation of the variance. Because I cannot store the data, they were calculated twice. Yes, doing twice more Fourier transforms is not efficient…

I found an online algorithm that can gives me both the mean and variance at the same in one pass (online variance calculation). The result is of course a big improvement but the program was still suffering from heavy starvation:

Source code

if(locali/=1) then
    delta=fftinout-average/(locali-1)
else
    delta=fftinout
end if
average=average+fftinout
sigmas = sigmas + delta*(fftinout - average/locali)

The problem here is that data are reused and do not stay in cache. delta, fftinout, average and sigmas are big 3D-arrays. When delta is computed, it does not fit into the cache, fftinout is send away to make room which is just used a line below and have to be pulled back from RAM…

What I did is doing some manual tiling. Usually the compiler do it automatically but in this case, gfortran was not smart enough even with -floop-block -floop-strip-mine. The result is that each stride is staying in cache which avoid moving data back and forth:

Source code

do kk=1,ubound(fftinout,3)
    do jj=1,ubound(fftinout,2)
        delta=fftinout(:,jj,kk)-tempfact*average(:,jj,kk)
        average(:,jj,kk)=average(:,jj,kk)+fftinout(:,jj,kk)
        sigmas(:,jj,kk) = sigmas(:,jj,kk) +delta*&
        &   (fftinout(:,jj,kk) - average(:,jj,kk)/locali)
    end do
end do

The result is amazing, I had a 20% speedup just with this change. You just need to be smarter than the compiler. And I save a bit of memory, delta is just a stride now.

New result on my 2.7GHz, 4 cores computer with 4GB@800MHz memory:

Previous results were for half the work, I was skipping the average. The current code is now faster than half the job in the previous version.

Continue reading »]]> A few weeks ago, I discovered that one of my programs has some cpu starvation issues. A cpu is starved when is waiting for the data. The most common cases are due high load on the disk but it can happen that even the RAM can be too slow. As I am just using a few MB on a 4GB system, that could not be the case.

My first hint was that the computing speed was not scaling up with the number of cores used on the newest version of the program. An old version did not have these problem. I even tested on 48 cores. But the new version version is much more efficient. Here is the results of the program running on a different numbers of cores.

Cores: number of cores used
real: real clock time in seconds
user: user clock time in seconds
loop: average time to execute a cycle of a do loop in my program in ms. In parenthesis, the one core equivalent loop time

Of course these results alone are not enough, problems from parallelism execution can have many sources. These results were obtained (home computer) on a intel q9505 (2800MHz) cpu equipped with 667MHz DDR2 memory (dual channel).

I have a similar computer at work: a intel q9400 (2666MHz) cpu equipped with 800MHz DDR2 memory (dual channel). The fact is my code is running slower at home (ie with the faster cpu) than at work. This was clearly the indication that my cpu was not running at full speed.

After searching a bit, I found two tools that would help me:

• perf, from perf-util (http://www.kernel.org/)
• oprofile (http://oprofile.sourceforge.net/news/)

Here is the result of the command # perf top:

As already noted in previous posts (http://blog.debroglie.net/2011/07/26/code-optimisation/), complex exponentials are quite numerous and can be seen here as sincos. They represent 6.7% of the time. The clear_page_c function is more worrying, it is a kernel function related to the control of memory. A web search did not reveal any more information.

The command # perf stat -p 425 -v -d is given more detailed informations about the program execution (425 was the pid of the program).

It revealed that the LLC-load-misses is 9.89%. It means that 10% of the time, the cpu won’t find any data in the L2 cache forcing him to wait for the data to come from the ram memory. I also run perf when the program is running on a different number of cores. In the case above, the 4 cores on my cpu were used. The missed cache rate was increasing with the number of cores (openmp has an environment variable: OMP_NUM_THREADS to do this). The workload was exactly the same but by doing simultaneously different jobs, the stress on the memory got more and more important. It would be interesting to run the same test on the same architecture but with a q9500 cpu. It has 12MB of L2 cache instead of 6MB for the q9505.

The next tool is oprofile. Similar to perf, it is based on recent kernel features about tracing performance. It comes with a nice gui where you can choose the parameter to follow, I choose the “LLc_*” events. The gui is launched via # oprof_start gui.

The opreport command gives you the results, honestly quite difficult to interpret. But I still manage to see that two “functions” got missed cache: the main program edensgrid (I really should put the cpu intensive part in its own subroutine…) and libfftw3f.

I feed the result into kcachegrind using the command: $ opreport -gdf | op2calltree. It creates a bunch of files, each one can loaded into kcachegrind. I loaded the one I was interested in: oprof.out.edensgrid. No need to look to the fftw3 one, this library is highly optimised, there is no room to improvement here. When compiled with the debugging symbols and with the source code available, kcachegrind will link the results to the source code.

The locations of missed cache are: sigmas=sigmas+(fftinout-datadiff)**2 and the initialisation of the array for a minor part: fftinout=0.0_fftkind with the inversion fftinout=-fftinout.

About the solution, it’s another story… I have at the moment no idea. I can probably work out a solution for the fftinout assignation by tweaking the fillfftarray subroutine. I guess I can remove the zero initalisation and the inversion. The sigmas business would need a better understanding of the cpu internals to find a better way to calculate if a solution exist.

A note about oprofile and perf vs valgrind:
I think that valgrind can also give this kind of information but as everything is virtualised, it’s slow and I am not sure if openmp code goes well with it. oprofile and perf have very little overhead and just probe the program while it’s running at full speed without any artefacts.