Dear friends
I am going to run a hybrid MPI+OPENMP code.
As you can see in the below I played with the number of threads in various cases while running my main code with (Openmpi,Mpich,Mpich2).
As you can see in OpenMPi and Mpich it seems that openmp did not work at all as Total time did not change considerably. But in Mpich2 Total computational time increased with increasing the number of threads. It could be because of using virtual threads instead of physical threads(or you said that over-subscribing).

Hybrid code result (MPI + OpenMP) :
Your suggestions: 
mvapich
mpirun -np 4 -genv OMP_NUM_THREADS 1 --bind-to hwthread:1 ./pjet.gfortran > output.txt
Total time = 7.290E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 ./pjet.gfortran > output.txt
Total time =  4.940E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8  -map-by hwthread:8 ./pjet.gfortran > output.txt
Total time =  4.960E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 --bind-to hwthread:16 ./pjet.gfortran > output.txt
Total time = 4.502E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to core:16 -map-by core:16 ./pjet.gfortran > output.txt
Total time =  4.628E+02

Pervios commands 
OpenMPI 1.8.1
mpirun -np 4 -x OMP_NUM_THREADS=1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time =  4.475E+02
mpirun -np 4 -x OMP_NUM_THREADS=8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time =  4.525E+02
mpirun -np 4 -x OMP_NUM_THREADS=16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time =  4.611E+02

mvapich 
mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.441E+02
mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.535E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time =  4.552E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time =  4.591E+02

mvapich2 
mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.935E+02
mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 5.562E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 6.392E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 8.170E+02

Then I used a simple "hybrid.f90" code and its result which I used to check whether the computer can recognize correct value of cores and threads or not. It showed that the correct values in all three in different cases.
here is its result: 

 Starting omp_dotprod_hybrid. Using           4  Cores...
 Core           3  using          16  threads
 Core           0  using          16  threads
 Core           2  using          16  threads
 Core           1  using          16  threads
 Core  1 thread  0  partial sum =   0.0000000000000000     
 Core  3 thread  0  partial sum =   0.0000000000000000    
 Core  1 thread  4  partial sum =   0.0000000000000000    
 Core  3 thread  7  partial sum =   200.00000000000000    
 Core  1 thread  8  partial sum =   200.00000000000000    
 Core  3 thread  9  partial sum =   200.00000000000000    
 Core  1 thread  11 partial sum =   200.00000000000000    
 Core  3 thread  3  partial sum =   200.00000000000000    
 Core  1 thread  2  partial sum =   0.0000000000000000    
 Core  3 thread  5  partial sum =   0.0000000000000000    
 Core  1 thread  3  partial sum =   200.00000000000000    
 Core  3 thread  2  partial sum =   0.0000000000000000    
 Core  1 thread  13 partial sum =   200.00000000000000    
 Core  3 thread  12 partial sum =   200.00000000000000    
 Core  1 thread  1  partial sum =   200.00000000000000    
 Core  3 thread  1  partial sum =   200.00000000000000    
 Core  3 thread  8  partial sum =   0.0000000000000000    
 Core  1 thread  7  partial sum =   0.0000000000000000    
 Core  3 thread  11 partial sum =   200.00000000000000    
 Core  1 thread  15 partial sum =   0.0000000000000000    
 Core  3 thread  15 partial sum =   0.0000000000000000    
 Core  1 thread  10 partial sum =   200.00000000000000    
 Core  1 thread  9  partial sum =   200.00000000000000    
 Core  3 thread  13 partial sum =   0.0000000000000000    
 Core  1 thread  5  partial sum =   200.00000000000000    
 Core  3 thread  6  partial sum =   0.0000000000000000    
 Core  3 thread  4  partial sum =   0.0000000000000000    
 Core  1 thread  6  partial sum =   0.0000000000000000    
 Core  3 thread  14 partial sum =   200.00000000000000    
 Core  1 thread  12 partial sum =   0.0000000000000000    
 Core  3 thread  10 partial sum =   200.00000000000000    
 Core  0 thread  0  partial sum =   0.0000000000000000    
 Core  0 thread  14 partial sum =   200.00000000000000    
 Core  0 thread  8  partial sum =   200.00000000000000    
 Core  0 thread  7  partial sum =   0.0000000000000000    
 Core  0 thread  15 partial sum =   200.00000000000000    
 Core  0 thread  5  partial sum =   200.00000000000000    
 Core  0 thread  9  partial sum =   200.00000000000000    
 Core  0 thread  11 partial sum =   0.0000000000000000    
 Core  0 thread  10 partial sum =   200.00000000000000    
 Core  0 thread  6  partial sum =   200.00000000000000    
 Core  0 thread  3  partial sum =   200.00000000000000    
 Core  0 thread  4  partial sum =   0.0000000000000000    
 Core  0 thread  2  partial sum =   0.0000000000000000    
 Core  0 thread  13 partial sum =   0.0000000000000000    
 Core  0 thread  12 partial sum =   0.0000000000000000    
 Core  0 thread  1  partial sum =   0.0000000000000000    
 Core  0 partial sum =   1600.0000000000000    
 Core  2 thread  3  partial sum =   0.0000000000000000    
 Core  2 thread  15 partial sum =   0.0000000000000000    
 Core  2 thread  0  partial sum =   0.0000000000000000    
 Core  2 thread  2  partial sum =   200.00000000000000    
 Core  2 thread  4  partial sum =   0.0000000000000000     
 Core  2 thread  5  partial sum =   0.0000000000000000    
 Core  2 thread  9  partial sum =   200.00000000000000    
 Core  2 thread  7  partial sum =   0.0000000000000000    
 Core  2 thread  14 partial sum =   200.00000000000000    
 Core  2 thread  8  partial sum =   200.00000000000000    
 Core  2 thread  12 partial sum =   200.00000000000000    
 Core  2 thread  10 partial sum =   200.00000000000000    
 Core  2 thread  6  partial sum =   200.00000000000000    
 Core  2 thread  1  partial sum =   0.0000000000000000    
 Core  2 thread  13 partial sum =   0.0000000000000000    
 Core  2 thread  11 partial sum =   200.00000000000000    
 Core  2 partial sum =   1600.0000000000000    
 Core  3 partial sum =   1600.0000000000000    
 Core  1 thread  14  partial sum =   0.0000000000000000    
 Core  1 partial sum =   1600.0000000000000    
 Done. Hybrid version: global sum  = 6400.0000000000000    
 



Please tell me If I should check something. I still getting nowhere.
Best regards

Pasha Pashaei