Hands-On 6: Tune the temporary storage locations (SASWORK)

Objectives of the hands-on
Step-by-step instructions
What have we learned ?

In this hands-on, we will see how to change the default SASWORK location (emptyDir). Instead we will configure the SASWORK location to point to a specific host directory. In a customer scenario, this directory would point to a high performance storage system location (for example in the Cloud, it could be the path of the Cloud instance NVME/SSD temporary drive or the mountpoint of a RAID stack of multiple SSD Drives). (Tip #5)

Objectives of the hands-on

Create the directories for the SASWORK on each Kubernetes node used for SAS Compute servers.
Create a patchTransformer manifest to:
- Mount the Host directories to the SAS Compute pods.
Re-Run the deployment playbook to pick up and implement the configuration changes.
Check the configuration change.

Step-by-step instructions

Create a directory on the Compute node(s) for the SASWORK

Open the Root Terminal

Note: We know that the SAS Compute Server pod can start on one of the 3 Kubernetes “Compute” nodes (which in this specific environment correspond to the containers k3d-edu-agent3, 5 and 6).
So, let’s run the following command to create the /saswork directory on each of the 3 Compute nodes (copy and paste the command in the Root Terminal).
```
for i in 3 5 6;do docker exec -it k3d-edu-agent-$i mkdir -p /saswork;done
```

Then set the permissions on the /saswork directory.

for i in 3 5 6;do docker exec -it k3d-edu-agent-$i chmod 777 /saswork;done

Make sure we have the /saswork directory on the 3 compute nodes:

for i in 3 5 6;do docker exec -it k3d-edu-agent-$i ls -l / | grep saswork;done

You should see the 3 folders as below with the correct permissions (drwxrwxrwx).

Create the PatchTransformer for the SASWORK configuration.

Now that the /saswork directory exists on each Kubernetes Compute node, we create a PatchTransformer manifest to remove the current viya volume using emptyDir add to add a new one using a hostPath volume type with our new /saswork directory path.

Copy and paste the command below to create the PatchTransformer manifest file and populate its content:

tee /root/edu_k3dv5/edu/site-config/edu/compute-manage-saswork.yaml  > /dev/null << EOF
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: delete-viya-volume
patch: |-
  apiVersion: v1
  kind: PodTemplate
  metadata:
    name: change-viya-volume-storage-class
  template:
    spec:
      volumes:
        - \$patch: delete
          name: viya
target:
  kind: PodTemplate
  labelSelector: "sas.com/template-intent=sas-launcher"
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: add-viya-volume
patch: |-
  - op: add
    path: /template/spec/volumes/-
    value:
      name: viya
      hostPath:
        path: /saswork
        type: Directory
target:
  kind: PodTemplate
  labelSelector: "sas.com/template-intent=sas-launcher"
EOF

Redeploy with the configuration change

Copy and paste the following command in the Root Terminal window, to re-run the Viya deployment playbook (so the configuration changes will be picked up):
```
/root/edu_k3dv5/scripts/redeploy.sh
```
It should take between 1 and 2 minutes to complete. Optionally, while the redeployment is running you can switch to OpenLens to look at potential changes in the Pods workload.
If the redeployment is successful you should see something like this in the Terminal window:
Since this SAS Viya deployment is managed by the SAS Deployment operator, you need to wait for the SAS Deployment operator’s reconciliation job to complete.
You can monitor the progress of the job by running the command below:
```
watch 'kubectl -n edu get SASDeployment'
```
You should see something like this:
The reconciling step should take around 5 minutes.
When you see that the SUCCEEDED for the STATE, type <Ctrl+C> to exit the watch command.

Make sure the change has been taken into account:

kubectl -n edu get podtemplate sas-compute-job-config -o json | jq '.template.spec.volumes[] | select(.name=="viya")'

You should see that there is a hostPath definition for the viya volume which maps to the /saswork path.

Check the configuration changes and new location for SASWORK

Now, it is time to test our change !
If you were already logged in SAS Studio, go in the Options menu and click Reset SAS Session.
Otherwhise open Google Chrome and click the SAS Studio bookmark or if you use the SAS App, click the “hamburger” icon (top left) and select Develop Code and Flows.
Login as student/Metadata0.
Choose No when asked to opt in to all of your assumable groups.
Click Program in SAS.

Copy and paste the following SAS code in the “Code” window:

data work.top10tuning;  
set sashelp.cars;  
do i=1 to 100;  
output;  
end;  
run;

Run the code (Hit the F3 key or click the button).
Back to the Root Terminal, determine where your pod is running by running this command:
```
kubectl -n edu get pods -l launcher.sas.com/username=student -o wide
```

Let’s grab the pod name.

PODNAME=$(kubectl -n edu get pods -l launcher.sas.com/username=student -o=custom-columns=PODNAME:.metadata.name --no-headers)
echo "The SAS Compute server is running in the pod $PODNAME"

and the pod’s node:

NODENAME=$(kubectl -n edu get pods -l launcher.sas.com/username=student -o=custom-columns=NODENAME:.spec.nodeName --no-headers)
echo "The SAS Compute server pod $PODNAME is running on node $NODENAME"

Now, let’s look at the paths inside the pod:

kubectl -n edu exec -it ${PODNAME} -c sas-programming-environment -- bash -c "ls -alR /opt/sas/viya/config/var/tmp/compsrv/default"

You should see a file that corresponds to the top10tuning temporary SAS dataset.
Finally, let’s look on the node:
```
kubectl debug node/${NODENAME} -it --image=server.demo.sas.com:5000/busybox
```
This command will open an interactive session on the node where the SAS Compute pod is running. You will see something like this:
Once you are connected let’s see if we can see some SASWORK temporary files:
```
ls -alR /host/saswork
```
You should see a lot of files under our custom SASWORK location:
If the output includes the top10tuning temporary SAS dataset, then it means that your configuration was successful!
DO NOT FORGET to exit from the debug pod, by typing:
```
exit
```

What have we learned ?

Changing the default location of the temporary storage areas for CAS and Compute requires changes in the kustomize configuration and a redeployment.
However, it is strongly recommended to provision and configure sufficient and highly-performant storage for them. The default location and volume types for CAS Disk Cache and SASWORK (Node’s root directories with the “emptyDir” volume should never be used in production environments).
In this example, to keep things simple we have used the hostPath volume type. However this option may not always be acceptable for security reasons. Alternative options exist, such as the Kubernetes generic ephemeral volume (with dynamic provisioning) or with the Cloud provider’s specific storage capabilities (for example AKS ephemeral OS disks in Azure ).