1.10. Isambard¶
The requirements for the machine learning project on the blade servers:
Current purpose - re-run just the steady state case without optimisation. Only require:
Anaconda 3
Python 3.x
PyFOAM 0.6.6
OpenFOAM 4.x
Pointwise 18.0 R2
1.10.1. Proposed Run Script (no run functions - check if everything is installed)¶
module load python/anaconda3
source activate py3
of4x
pyFoamClearCase.py . --processors-remove
/usr/local/Pointwise/PointwiseV18.0R2/pointwise -b ./Hydro_V18_wider.glf
rm -r 0
cp -r 0_orig 0
transformPoints -scale '(0.0254 0.0254 0.0254)'
renumberMesh -overwrite
cp system/controlDict_simple system/controlDict
decomposePar
mpirun -np 28 potentialFoam -parallel > log.potentialFoam 2>&1
mpirun -np 28 simpleFoam -parallel > log.simpleFoam 2>&1
reconstructPar -latestTime
pyFoamCopyLastToFirst.py . .
pyFoamClearCase.py . --processors-remove
cp system/controlDict_parcel system/controlDict
decomposePar
mpirun -np 28 icoUncoupledKinematicParcelFoam -parallel > log.icoUncoupledKinematicParcelFoam 2>&1
With run functions:
module load python/anaconda3
source activate py3
module load mpi/openmpi-x86_64
source /usr/local/OpenFOAM/OpenFOAM-4.x/etc/bashrc
export PS1="(OF:\$WM_PROJECT_VERSION) $PS1"
. $WM_PROJECT_DIR/bin/tools/RunFunctions
echo "Environment setup complete"
rm -rf log.*
pyFoamClearCase.py . --processors-remove
echo "Case cleared"
renumberMesh -overwrite
echo "Renumbered mesh"
cp system/controlDict_simple system/controlDict
echo "Copied controlDict"
decomposePar
echo "Decomposed case"
runParallel potentialFoam
echo "PotentialFoam finished"
runParallel simpleFoam
echo "SimpleFoam finished"
reconstructPar -latestTime
echo "Case reconstructed"
pyFoamCopyLastToFirst.py . .
echo "Copied first to last"
pyFoamClearCase.py . --processors-remove
echo "Cleared case"
cp system/controlDict_parcel system/controlDict
echo "Copied controlDict"
decomposePar
echo "Decomposed case"
runParallel icoUncoupledKinematicParcelFoam
echo "IcoUncoupledKinematicParcelFoam finished"
1.10.2. OS - Phase 1 Isambard¶
uname -a
Linux login-01 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
1.10.3. Anaconda 3.5.4¶
source /opt/anaconda3/latest/bin/activate
1.10.4. Python 3.x¶
Change to python 3 environment
conda search "python"
conda create -n py3 python=3.5.4 anaconda
Proceed ([y]/n)?
y
conda activate py3
environment location: /home/ex-aroberts/.conda/envs/py3
1.10.4.1. pyFoam 0.6.6¶
pip install --user PyFoam==0.6.6
pip install --user PyFoam==0.6.6
Collecting PyFoam==0.6.6
Using cached https://files.pythonhosted.org/packages/aa/be/47eb7b9b0533af692a494d0bb3a6820f00d01f3b7ee8d3743b07475a5885/PyFoam-0.6.6-py2.py3-none-any.whl
Requirement already satisfied: numpy in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from PyFoam==0.6.6) (1.14.5)
jupyter 1.0.0 requires qtconsole, which is not installed.
grpcio 1.10.0 has requirement protobuf>=3.5.0.post1, but you'll have protobuf 3.5.0 which is incompatible.
Installing collected packages: PyFoam
Successfully installed PyFoam-0.6.6
You are using pip version 10.0.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
1.10.4.2. Numpy¶
pip install --user numpy
Requirement already satisfied: numpy in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (1.14.5)
jupyter 1.0.0 requires qtconsole, which is not installed.
grpcio 1.10.0 has requirement protobuf>=3.5.0.post1, but you'll have protobuf 3.5.0 which is incompatible.
You are using pip version 10.0.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
1.10.4.3. Matplotlib¶
pip install --user matplotlib
Requirement already satisfied: matplotlib in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (2.2.3)
Requirement already satisfied: numpy>=1.7.1 in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (1.14.5)
Requirement already satisfied: cycler>=0.10 in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (2.2.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (2.7.3)
Requirement already satisfied: pytz in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (2018.5)
Requirement already satisfied: six>=1.10 in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (1.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from matplotlib) (1.0.1)
Requirement already satisfied: setuptools in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (from kiwisolver>=1.0.1->matplotlib) (40.2.0)
jupyter 1.0.0 requires qtconsole, which is not installed.
grpcio 1.10.0 has requirement protobuf>=3.5.0.post1, but you'll have protobuf 3.5.0 which is incompatible.
You are using pip version 10.0.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
1.10.4.4. scipy¶
pip install --user scipy
Requirement already satisfied: scipy in /opt/anaconda3/5.1.0/lib/python3.6/site-packages (1.1.0)
jupyter 1.0.0 requires qtconsole, which is not installed.
grpcio 1.10.0 has requirement protobuf>=3.5.0.post1, but you'll have protobuf 3.5.0 which is incompatible.
You are using pip version 10.0.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
1.10.4.5. deap¶
pip install --user deap
Collecting deap
Downloading https://files.pythonhosted.org/packages/f7/d2/ca7c4ebbd2c30a017c188b72da98dab3ab54648c3b0aaa7a2efa772d6147/deap-1.3.0.tar.gz (1.1MB)
100% |████████████████████████████████| 1.1MB 621kB/s
Building wheels for collected packages: deap
Running setup.py bdist_wheel for deap ... done
Stored in directory: /home/ex-aroberts/.cache/pip/wheels/95/46/10/82f914655ecf6ff1dc7d42a47ba704a9c9181f24f8edfbe75d
Successfully built deap
jupyter 1.0.0 requires qtconsole, which is not installed.
grpcio 1.10.0 has requirement protobuf>=3.5.0.post1, but you'll have protobuf 3.5.0 which is incompatible.
Installing collected packages: deap
Successfully installed deap-1.3.0
You are using pip version 10.0.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
1.10.4.6. PyDOE¶
pip install pyDOE
1.10.4.7. evoalgos¶
pip install evoalgos
1.10.4.8. GPy¶
pip install GPy
1.10.4.9. CMA¶
pip install cma
1.10.5. OpenFOAM 4.x¶
Download packages:
git clone https://github.com/OpenFOAM/OpenFOAM-4.x.git
git clone https://github.com/OpenFOAM/ThirdParty-4.x.git
Don’t need to install any software for compilation
Set environment variables:
cd $𝐻𝑂𝑀𝐸
alias of4x='module load mpi/openmpi-x86_64; source $HOME/OpenFOAM/OpenFOAM-4.x/etc/bashrc; export PS1="(APR OF:\$WM_PROJECT_VERSION) $PS1"'
After logging out and logging in again, install Third Party Scotch and PT Scotch
of4x
cd $WM_THIRD_PARTY_DIR
./Allwmake > log.make 2>&1 &
Install OpenFOAM
cd ../OpenFOAM-4.x/
./Allwmake -j 16 > log.make 2>&1 &
1.10.6. Output of PATH¶
(APR OF:4.x) (py3) $ echo $PATH
/misc/OpenFOAM/ThirdParty-4.x/platforms/linux64Gcc/gperftools-svn/bin:
/misc/OpenFOAM/ThirdParty-4.x/platforms/linux64Gcc/ParaView-5.0.1/bin:
/home/links/apr207/OpenFOAM/apr207-4.x/platforms/linux64GccDPInt32Opt/bin:
/misc/OpenFOAM/site/4.x/platforms/linux64GccDPInt32Opt/bin:
/misc/OpenFOAM/OpenFOAM-4.x/platforms/linux64GccDPInt32Opt/bin:
/misc/OpenFOAM/OpenFOAM-4.x/bin:
/misc/OpenFOAM/OpenFOAM-4.x/wmake:
/usr/lib64/openmpi/bin:
/home/links/apr207/.conda/envs/py3/bin:
/usr/local/anaconda3/bin:
/usr/local/texlive/2017/bin/x86_64-linux:
/usr/lib64/qt-3.3/bin:
/usr/local/bin:
/usr/bin:
/usr/local/sbin:
/usr/sbin:
/opt/puppetlabs/bin:
/home/links/apr207/.local/bin:
/home/links/apr207/bin
1.10.7. Pointwise 18.0R2¶
sh pw-V18.0R2-linux_x86_64-jre.sh
Next
Accept
/home/links/apr207/Pointwise/PointwiseV18.0R2 is installation directory
Run Pointwise
(Won't find license)
Specify license server
server: emps-pointwise
port: 2385
Add this to .bashrc:
# Pointwise alias for this version
alias pointwise="/home/links/apr207/Pointwise/PointwiseV18.0R2/pointwise"
1.10.8. Local Install of OpenFOAM on Isambard (Phase 2)¶
Download OpenFOAM-v1812 and ThirdParty-v1812
BUILD_DIR=$HOME/OpenFOAM
mkdir $BUILD_DIR
cd $BUILD_DIR
wget https://sourceforge.net/projects/openfoamplus/files/v1812/OpenFOAM-v1812.tgz
wget https://sourceforge.net/projects/openfoamplus/files/v1812/ThirdParty-v1812.tgz
tar -xvf OpenFOAM-v1812.tgz -C $BUILD_DIR
tar -xvf ThirdParty-v1812.tgz -C $BUILD_DIR
Load ARM compilers
# load ARM compiler if no compilers are loaded
module load PrgEnv-allinea/6.0.5
# or swap current cray compiler with arm compiler
module switch PrgEnv-cray/6.0.5 PrgEnv-allinea/6.0.5
Make modifications to the configuration files required to build OF
# Download the two patch files and apply the patch running the below commands
wget https://dl.dropboxusercontent.com/s/jnpjop7kll5f3ph/OpenFOAM-v1812.patch
wget https://dl.dropboxusercontent.com/s/q0iwu2qf7pv9s6n/ThirdParty-v1812.patch
(cd OpenFOAM-v1812/; patch -p1 < $BUILD_DIR/OpenFOAM-v1812.patch)
(cd ThirdParty-v1812/; patch -p1 < $BUILD_DIR/ThirdParty-v1812.patch)
# create prefs.sh file with compiler options
cd $BUILD_DIR/OpenFOAM-v1812
echo -e "WM_COMPILER=Arm \nWM_MPLIB=CRAY-MPICH \nWM_LABEL_SIZE=64" > etc/prefs.sh
# add the following to $HOME/.bashrc:
module swap PrgEnv-cray PrgEnv-allinea
alias localof1812='module load PrgEnv-allinea; export OPENFOAM_DIR=$HOME/OpenFOAM/OpenFOAM-v1812; export PATH=$PATH:$OPENFOAM_DIR/bin/; source $OPENFOAM_DIR/etc/bashrc; export PS1="(LOCAL OF:\$WM_PROJECT_VERSION) $PS1"'
# reload bashrc (exit and relogin)
Compile OpenFOAM-v1812
# This step is optional to check if everything is okay to build OpenFOAM
foamSystemCheck
# Compile openfoam using all available processors (-j) with reduced output (-s) and log the output (-l) to a file so that we can examine any compilation issues later.
./Allwmake -j -s -l
Example run script
#!/bin/bash
#PBS -N openfoam_job
# Select 1 nodes (maximum of 64 cores)
#PBS -l select=1:ncpus=4
# Select wall time to 1 hour
#PBS -l walltime=1:00:00
# Use the arm nodes
#PBS -q arm
# Load modules for currently recommended OpenFOAM v1812 build
module load PrgEnv-allinea
# Change to directory that script was submitted from
export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)
export OMP_NUM_THREADS=1
cd $PBS_O_WORKDIR
# Load the environment variables for OpenFOAM v1812 build
export OPENFOAM_DIR=$HOME/OpenFOAM/OpenFOAM-v1812
export PATH=$PATH:$OPENFOAM_DIR/bin/
source $OPENFOAM_DIR/etc/bashrc
# Clear folder
rm -rf log
rm -rf log.*
rm -rf openfoam*
rm -rf processor*
rm -rf 0.*
rm -rf postProcessing
# Launch OpenFoam solvers
blockMesh > log.blockMesh 2>&1
checkMesh > log.checkMesh 2>&1
decomposePar > log.decomposePar 2>&1
aprun -n 4 icoFoam -parallel > log.icoFoam 2>&1
reconstructPar -time 0: > log.reconstructPar 2>&1