Hands-On 6: Tune the temporary storage locations (SASWORK)
In this hands-on, we will see how to change the default SASWORK location (emptyDir). Instead we will configure the SASWORK location to point to a specific host directory. In a customer scenario, this directory would point to a high performance storage system location (for example in the Cloud, it could be the path of the Cloud instance NVME/SSD temporary drive or the mountpoint of a RAID stack of multiple SSD Drives). (Tip #5)
Objectives of the hands-on
-
Create the directories for the SASWORK on each Kubernetes node used for SAS Compute servers.
- Create a patchTransformer manifest to:
- Mount the Host directories to the SAS Compute pods.
-
Re-Run the deployment playbook to pick up and implement the configuration changes.
- Check the configuration change.
Step-by-step instructions
Create a directory on the Compute node(s) for the SASWORK
-
Open the Root Terminal
Note: We know that the SAS Compute Server pod can start on one of the 3 Kubernetes “Compute” nodes (which in this specific environment correspond to the containers k3d-edu-agent3, 5 and 6).
-
So, let’s run the following command to create the /saswork directory on each of the 3 Compute nodes (copy and paste the command in the Root Terminal).
for i in 3 5 6;do docker exec -it k3d-edu-agent-$i mkdir -p /saswork;done
-
Then set the permissions on the /saswork directory.
for i in 3 5 6;do docker exec -it k3d-edu-agent-$i chmod 777 /saswork;done
-
Make sure we have the /saswork directory on the 3 compute nodes:
for i in 3 5 6;do docker exec -it k3d-edu-agent-$i ls -l / | grep saswork;done
-
You should see the 3 folders as below with the correct permissions (
drwxrwxrwx
).
Create the PatchTransformer for the SASWORK configuration.
-
Now that the
/saswork
directory exists on each Kubernetes Compute node, we create a PatchTransformer manifest to remove the currentviya
volume usingemptyDir
add to add a new one using ahostPath
volume type with our new/saswork
directory path. -
Copy and paste the command below to create the PatchTransformer manifest file and populate its content:
tee /root/edu_k3dv5/edu/site-config/edu/compute-manage-saswork.yaml > /dev/null << EOF --- apiVersion: builtin kind: PatchTransformer metadata: name: delete-viya-volume patch: |- apiVersion: v1 kind: PodTemplate metadata: name: change-viya-volume-storage-class template: spec: volumes: - \$patch: delete name: viya target: kind: PodTemplate labelSelector: "sas.com/template-intent=sas-launcher" --- apiVersion: builtin kind: PatchTransformer metadata: name: add-viya-volume patch: |- - op: add path: /template/spec/volumes/- value: name: viya hostPath: path: /saswork type: Directory target: kind: PodTemplate labelSelector: "sas.com/template-intent=sas-launcher" EOF
Redeploy with the configuration change
-
Copy and paste the following command in the Root Terminal window, to re-run the Viya deployment playbook (so the configuration changes will be picked up):
/root/edu_k3dv5/scripts/redeploy.sh
-
It should take between 1 and 2 minutes to complete. Optionally, while the redeployment is running you can switch to OpenLens to look at potential changes in the Pods workload.
-
If the redeployment is successful you should see something like this in the Terminal window:
-
Since this SAS Viya deployment is managed by the SAS Deployment operator, you need to wait for the SAS Deployment operator’s reconciliation job to complete.
-
You can monitor the progress of the job by running the command below:
watch 'kubectl -n edu get SASDeployment'
-
You should see something like this:
-
The reconciling step should take around 5 minutes.
-
When you see that the SUCCEEDED for the STATE, type
<Ctrl+C>
to exit the watch command. -
Make sure the change has been taken into account:
kubectl -n edu get podtemplate sas-compute-job-config -o json | jq '.template.spec.volumes[] | select(.name=="viya")'
-
You should see that there is a
hostPath
definition for theviya
volume which maps to the/saswork
path.
Check the configuration changes and new location for SASWORK
- Now, it is time to test our change !
- If you were already logged in SAS Studio, go in the Options menu and click Reset SAS Session.
-
Otherwhise open Google Chrome and click the SAS Studio bookmark or if you use the SAS App, click the “hamburger” icon (top left) and select Develop Code and Flows.
-
Login as student/Metadata0.
-
Choose No when asked to opt in to all of your assumable groups.
-
Click Program in SAS.
-
Copy and paste the following SAS code in the “Code” window:
data work.top10tuning; set sashelp.cars; do i=1 to 100; output; end; run;
-
Run the code (Hit the F3 key or click the button).
-
Back to the Root Terminal, determine where your pod is running by running this command:
kubectl -n edu get pods -l launcher.sas.com/username=student -o wide
-
Let’s grab the pod name.
PODNAME=$(kubectl -n edu get pods -l launcher.sas.com/username=student -o=custom-columns=PODNAME:.metadata.name --no-headers) echo "The SAS Compute server is running in the pod $PODNAME"
-
and the pod’s node:
NODENAME=$(kubectl -n edu get pods -l launcher.sas.com/username=student -o=custom-columns=NODENAME:.spec.nodeName --no-headers) echo "The SAS Compute server pod $PODNAME is running on node $NODENAME"
-
Now, let’s look at the paths inside the pod:
kubectl -n edu exec -it ${PODNAME} -c sas-programming-environment -- bash -c "ls -alR /opt/sas/viya/config/var/tmp/compsrv/default"
-
You should see a file that corresponds to the
top10tuning
temporary SAS dataset. -
Finally, let’s look on the node:
kubectl debug node/${NODENAME} -it --image=server.demo.sas.com:5000/busybox
This command will open an interactive session on the node where the SAS Compute pod is running. You will see something like this:
-
Once you are connected let’s see if we can see some SASWORK temporary files:
ls -alR /host/saswork
-
You should see a lot of files under our custom SASWORK location:
-
If the output includes the
top10tuning
temporary SAS dataset, then it means that your configuration was successful! -
DO NOT FORGET to exit from the debug pod, by typing:
exit
What have we learned ?
-
Changing the default location of the temporary storage areas for CAS and Compute requires changes in the kustomize configuration and a redeployment.
-
However, it is strongly recommended to provision and configure sufficient and highly-performant storage for them. The default location and volume types for CAS Disk Cache and SASWORK (Node’s root directories with the “emptyDir” volume should never be used in production environments).
-
In this example, to keep things simple we have used the
hostPath
volume type. However this option may not always be acceptable for security reasons. Alternative options exist, such as the Kubernetes generic ephemeral volume (with dynamic provisioning) or with the Cloud provider’s specific storage capabilities (for example AKS ephemeral OS disks in Azure).
- hands on / HO1 Enable Progressive App
- hands on / HO2 Allow more analytics jobs to run
- hands on / HO3 Create a pool of reusable compute servers
- hands on / HO4 Increase Compute pod CPU limit
- hands on / HO5 Tune temporary storage locations CAS DISK CACHE
- hands on / HO6 Tune temporary storage locations SASWORK<-- you are here