reproducing figure 3

[psteinb]

Hello,

I'd would like to reproduce the results reported in figure 3 of your paper, but I am a bit lost how to do it. Could you please indicate the command line parameters that you used to generate the time results? On another thought, how did you obtain the timings, just with the time utility?

Best

[solonas13]

Hi there,

Using "$ ./maw" from the pmaw directory you can get the help menu.

A typical run is then

./maw -a DNA -i ../data/1.fasta -o 1.fasta.out -k 2 -K 10 -t 4

which computes all maws of length between 2 and 10 with 4 threads of the DNA sequence in ../data/1.fasta.

If you would like to measure the time or peak memory then use

$ /usr/bin/time -v ./maw -a DNA -i ../data/1.fasta -o 1.fasta.out -k 2 -K 10 -t 4

As for Figure 3, synthetic data randomly generated have been used. You should be able to reproduce the results via generating random sequences of length 10Mbp, 100Mbp, and 1Gbp, respectively, by using a
uniform frequency distribution of letters of the DNA alphabet.

Best, Solon.

[psteinb]

interesting, the output of the ./maw binary doesn't yield any information about -t:

$ ./maw                                                                                                                              
 Usage: maw <options>                                                                                                                                      
 Standard (Mandatory):                                                                                                                                     
  -a, --alphabet            <str>     `DNA' for nucleotide  sequences or `PROT'                                                                            
                                      for protein  sequences.                                                                                              
  -i, --input-file          <str>     (Multi)FASTA input filename.                                                                                         
  -o, --output-file         <str>     Output filename.                                                                                                     
  -k, --min-length          <int>     The min length for maws.                                                                                             
  -K, --max-length          <int>     The max length for maws.                                                                                             
 Optional:                                                                                                                                                 
  -r, --reverse             <int>     `1' to check for the reverse complement or                                                                           
                                      `0' otherwise (default: 0). 

[solonas13]

Please use "$ ./maw" from the pmaw directory!

[psteinb]

sorry, yes ... I just read your earlier comment another time. Great, thanks for the info. I'll look into the pmaw runtimes now.
Best,
Peter

[psteinb]

I took a 34 MB fasta file and ran the pmaw binary with the flags from above, the walltime results reported below discourage me a bit

I tried under a lustre volume as well as with a ram disk (devshm) with mostly no difference. I would have expected that the results turn out between your 10MB and 100MB example. but they don't.

[solonas13]

Please note that Figure 3 refers to synthetic data with uniformly random distribution. Hence it is not expected to get the same results with a 34 MB human (thus REAL) data.

To get something meaningful with real data you need much bigger inputs (see Figure 4 that refers to FULL genomes).

Best, Solon.

[psteinb]

I am a bit lost with all the versions of the human genome floating around on the NCBI and the ensembl page. Which file did you use for the paper?

I see either ftp://ftp.ncbi.nlm.nih.gov/genomes/Homo_sapiens/CHR_01/hs_ref_GRCh38.p7_chr1.fa.gz or ftp://ftp.ensembl.org/pub/release-88/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz one as viable inputs. how do you feel?

[psteinb]

alright, I hacked together a small script to produce the synthetic data. I hope that this comes close to what you used for the paper and at the end of the day, I am mostly interested in the scaling behavior and how it plays out on our hardware. It would be nice, if you could share the script you used for the synthetic data as I don't want to run this app for longer than necessary and having the same input data makes my results more comparable to yours.

[psteinb]

I am done doing the tests we discussed. The results do not show the scaling reported in the paper using synthetic data.

It would be nice to have the same input data as you used for your tests, so I can be 100% sure, that we are on the same page. Maybe the results do not hold for Haswell and newer architectures? Related to this, did you research why you observe very good scaling only for large data sets?

for performing the experiments which the paper describes, did you pin the process or fix the CPU clock boost or anything like that? Were energy saving options enabled in the BIOS?

[psteinb]

from #4:

I’ve just check how the version on github scales on my computer, and it scales fine.
I’ve tested only on 1, 2 and 4 cores as my desktop computer only have two and thus it is not going to scale well above 4.

I observed this too on my laptop. But note that this does not contradict entirely the plot from above. the machine the job was running on for the plot above, had HT turned off and 2x12 cores (as you can tell if you look up E5-2680v3 on ark). So what you see is pure physical performance given the memory bandwidth contraints on that socket. On a laptop, you typically have 2 physical cores + 2 HT and maybe one or 2 memory channels (that depends on the model). But again, I am not interested in single socket performance. I'd like to know, why your code doesn't scale above (say) 4 cores on a dual socket system.

What is the red and the green lines on your results ?

I am very sorry, I should have been more explicit. The green line indicates the speed-up obtained with running your app like so:

$ ./maw <more flags to come>

The red line refers to the case running your app like:

$ numactl -m0 -N0 ./maw <more flags to come>

So everything above 12 cores will suffer NUMA penalties for sure ... which you can clearly see as the green line approaches the red one.

For what I see it seems that you used, 1, 6, 12, 16 and 24 threads, I am not expert in parallelism, but this seems quite unusual. If you want to redo our experiment, you should probably try to do it on a 10-core computer, using 1, 2, 4, 8 and 16 threads, as we did.
How many cores does your computer have ?

What you say surprises me. You published a paper on parallelising an algorithm and claim in public that you are not an expert in parallelism? ;)

If you want, I can rerun the code with your thread count, but given the cache layout of Intel E5 chips (AFAIK 2 physical cores share one L2 cache line, right?), I'd expect not much of a difference in runtimes. But maybe I am wrong - I am happy to learn. I am currently running some tests on an IvyBridge E5 dual socket system similar to what you used for the paper IIRC. So that will tell. Still, I am not using the same input data as you did. I asked above, if you can share it, but received no answer.

[psteinb]

Hi, the runs on our IvyBridge server are complete and here are the results:

I now ran your code on 3 different servers (Haswell, IvyBridge and SandyBridge which I didn't publish), I start to believe it's not our hardware that exposes a scalability problem. But I am happy to review further independent data.