Hello, I am having some issues trying to understand why I am not using aligned arrays in one of my subroutines. The code is shown below. I checked the alignment using
ifort -align array64byte -vec-report6 global.f90 mod_force.f90
module mod_force
use global
contains
subroutine force_vlist_MIC(step)
implicit none
double precision :: force,forcelj
double precision :: x1,y1,z1,x2,y2,z2
double precision :: dx,dy,dz,dr,dr2,dr2i,dr6i,dr12i
double precision :: filter
double precision :: fxl,fyl,fzl
integer :: i,j,step
integer :: itype,jtype,neigh
integer :: tid,num
integer :: offset
integer :: T1,T2,clock_rate,clock_max,rep
integer :: h
potential = 0.0d0
ff(:) = 0.0d0
!dir$ offload begin target(mic:0) in(r,numneigh,vlist) inout(ff,potential)
!$omp parallel do reduction(+: fxl,fyl,fzl,potential) schedule(dynamic),&
!$omp& default(private) shared(r,numneigh,neigh_alloc,vlist,rcut2,box,np)
do i = 1 ,np
x1 = r(3*i-2); y1 = r(3*i-1); z1 = r(3*i);
fxl = 0.0d0; fyl = 0.0d0; fzl = 0.0d0
!dir$ simd
!dir$ vector aligned
do j=1,numneigh(i)
!---look up neighboring particle
neigh = vlist(j+neigh_alloc*(i-1))
!---load particle data
x2 = r(3*neigh-2); y2 = r(3*neigh-1); z2 = r(3*neigh);
dx = x1-x2; dy = y1-y2; dz = z1-z2
dx = dx-box*nint(dx/box)
dy = dy-box*nint(dy/box)
dz = dz-box*nint(dz/box)
dr2 = dx*dx + dy*dy + dz*dz
filter = 0.0d0
if(dr2.lt.rcut2)filter = 1.0d0
dr2i = 1.0d0/dr2
dr6i = dr2i*dr2i*dr2i
force = 48.0d0*dr6i*(dr6i-0.50d0)
force = forcelj*dr2i
fxl = fxl + dx*force
fyl = fyl + dy*force
fzl = fzl + dz*force
potential = potential + dr6i*(dr6i-1.0d0)*filter
enddo
ff(3*i-2) = fxl; ff(3*i-1) = fyl; ff(3*i) = fzl
enddo
!$omp end parallel do
!dir$ end offload
call system_clock(T2,clock_rate,clock_max)
potential = 4.0d0 *potential
print*,'mic TIME',rep,real(T2-T1)/real(clock_rate),potential
end subroutine force_vlist_MIC
end module mod_forceimplicit none !--------------------------------------------------------------------- !*** simulation parameters !*** read at run time !dir$ attributes offload:mic::r double precision, allocatable :: r(:) double precision, allocatable :: rs(:) !dir$ attributes offload:mic::ff double precision, allocatable :: ff(:) double precision, allocatable :: ffs(:) double precision, allocatable :: drx(:),dry(:),drz(:) !dir$ attributes offload:mic::potential double precision :: potential double precision :: rcut !dir$ attributes offload:mic::rcut2 double precision :: rcut2 !dir$ attributes offload:mic::box double precision :: vol,box,hbox,ibox !dir$ attributes offload:mic::np integer :: np !----------------------------------------------------------------------- !*** LL and Vlist !*** !dir$ attributes offload:mic::vlist integer,allocatable :: vlist(:) !dir$ attributes offload:mic::numneigh integer,allocatable :: numneigh(:) !dir$ attributes offload:mic::neigh_alloc integer :: neigh_alloc end module global
the compiler output is
ogin1.stampede(152)$ ifort -align array64byte -vec-report6 global.f90 mod_force.f90 mod_force.f90(23): (col. 5) remark: vectorization support: unroll factor set to 2. mod_force.f90(23): (col. 5) remark: LOOP WAS VECTORIZED. mod_force.f90(23): (col. 5) remark: loop was not vectorized: not inner loop. mod_force.f90(39): (col. 8) remark: SIMD LOOP WAS VECTORIZED. mod_force.f90(63): (col. 11) remark: loop was not vectorized: not inner loop. mod_force.f90(46): (col. 11) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: indirect access . mod_force.f90(46): (col. 30) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: indirect access . mod_force.f90(46): (col. 49) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: indirect access . mod_force.f90(43): (col. 11) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: unit-strided, element aligned. mod_force.f90(46): (col. 11) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: indirect access . mod_force.f90(46): (col. 30) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: indirect access . mod_force.f90(46): (col. 49) remark: *MIC* vectorization support: gather was generated for the variable global_mp_r_: indirect access . mod_force.f90(43): (col. 11) remark: *MIC* vectorization support: reference global_mp_vlist_ has unaligned access. mod_force.f90(43): (col. 11) remark: *MIC* vectorization support: unaligned access used inside loop body. mod_force.f90(39): (col. 8) remark: *MIC* SIMD LOOP WAS VECTORIZED. mod_force.f90(43): (col. 11) remark: *MIC* vectorization support: reference global_mp_vlist_ has unaligned access. mod_force.f90(43): (col. 11) remark: *MIC* vectorization support: unaligned access used inside loop body. mod_force.f90(39): (col. 8) remark: *MIC* REMAINDER LOOP WAS VECTORIZED. mod_force.f90(31): (col. 5) remark: *MIC* loop was not vectorized: not inner loop.
I am having a hard time understanding what this is telling me. My first place of confusion is why line 43, which is using vlist, has unaligned access. Does the -align array64byte not work for global dynamic arrays? I couldn't quite figure out what the consensus was for aligning global dynamic arrays from the data alignment to assist vectorization article intel has available. Also, why is the !dir$ vector aligned not telling the compiler vlist is aligned? When it is telling me indirect access for global_mp_r, does that mean that the data is not aligned, or is that only in reference to non sequential array access being performed? The compiler then tells me that line 43 is unit stride and element aligned. Does that mean r is indeed aligned? Finally I noticed that if instead of using the !dir$ vector aligned, I switched to !dir$ assume_aligned r: 64 that I would get an error saying I could not do this for an array in a common block. However, I do not get such an error with the !dir$ vector aligned. Could someone explain this to me?