Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Hands-On 6: Tune the temporary storage locations (SASWORK)

In this hands-on, we will see how to change the default SASWORK location (emptyDir). Instead we will configure the SASWORK location to point to a specific host directory. In a customer scenario, this directory would point to a high performance storage system location (for example in the Cloud, it could be the path of the Cloud instance NVME/SSD temporary drive or the mountpoint of a RAID stack of multiple SSD Drives). (Tip #5)

Objectives of the hands-on

  • Create the directories for the SASWORK on each Kubernetes node used for SAS Compute servers.

  • Create a patchTransformer manifest to:
    • Mount the Host directories to the SAS Compute pods.
  • Re-Run the deployment playbook to pick up and implement the configuration changes.

  • Check the configuration change.

Step-by-step instructions

Create a directory on the Compute node(s) for the SASWORK

  1. Open the Root Terminal

    Note: We know that the SAS Compute Server pod can start on one of the 3 Kubernetes “Compute” nodes (which in this specific environment correspond to the containers k3d-edu-agent3, 5 and 6).

  2. So, let’s run the following command to create the /saswork directory on each of the 3 Compute nodes (copy and paste the command in the Root Terminal).

    for i in 3 5 6;do docker exec -it k3d-edu-agent-$i mkdir -p /saswork;done
    
  3. Then set the permissions on the /saswork directory.

    for i in 3 5 6;do docker exec -it k3d-edu-agent-$i chmod 777 /saswork;done
    
  4. Make sure we have the /saswork directory on the 3 compute nodes:

    for i in 3 5 6;do docker exec -it k3d-edu-agent-$i ls -l / | grep saswork;done
    
  5. You should see the 3 folders as below with the correct permissions (drwxrwxrwx).

    image74

Create the PatchTransformer for the SASWORK configuration.

  1. Now that the /saswork directory exists on each Kubernetes Compute node, we create a PatchTransformer manifest to remove the current viya volume using emptyDir add to add a new one using a hostPath volume type with our new /saswork directory path.

  2. Copy and paste the command below to create the PatchTransformer manifest file and populate its content:

    tee /root/edu_k3dv5/edu/site-config/edu/compute-manage-saswork.yaml  > /dev/null << EOF
    ---
    apiVersion: builtin
    kind: PatchTransformer
    metadata:
      name: delete-viya-volume
    patch: |-
      apiVersion: v1
      kind: PodTemplate
      metadata:
        name: change-viya-volume-storage-class
      template:
        spec:
          volumes:
            - \$patch: delete
              name: viya
    target:
      kind: PodTemplate
      labelSelector: "sas.com/template-intent=sas-launcher"
    ---
    apiVersion: builtin
    kind: PatchTransformer
    metadata:
      name: add-viya-volume
    patch: |-
      - op: add
        path: /template/spec/volumes/-
        value:
          name: viya
          hostPath:
            path: /saswork
            type: Directory
    target:
      kind: PodTemplate
      labelSelector: "sas.com/template-intent=sas-launcher"
    EOF
    

Redeploy with the configuration change

  1. Copy and paste the following command in the Root Terminal window, to re-run the Viya deployment playbook (so the configuration changes will be picked up):

    /root/edu_k3dv5/scripts/redeploy.sh
    
  2. It should take between 1 and 2 minutes to complete. Optionally, while the redeployment is running you can switch to OpenLens to look at potential changes in the Pods workload.

  3. If the redeployment is successful you should see something like this in the Terminal window:

  4. Since this SAS Viya deployment is managed by the SAS Deployment operator, you need to wait for the SAS Deployment operator’s reconciliation job to complete.

  5. You can monitor the progress of the job by running the command below:

    watch 'kubectl -n edu get SASDeployment'
    
  6. You should see something like this:

  7. The reconciling step should take around 5 minutes.

  8. When you see that the SUCCEEDED for the STATE, type <Ctrl+C> to exit the watch command.

  9. Make sure the change has been taken into account:

    kubectl -n edu get podtemplate sas-compute-job-config -o json | jq '.template.spec.volumes[] | select(.name=="viya")'
    
  10. You should see that there is a hostPath definition for the viya volume which maps to the /saswork path.

    image75

Check the configuration changes and new location for SASWORK

  1. Now, it is time to test our change !
  2. If you were already logged in SAS Studio, go in the Options menu and click Reset SAS Session.
  3. Otherwhise open Google Chrome and click the SAS Studio bookmark or if you use the SAS App, click the “hamburger” icon (top left) and select Develop Code and Flows.

  4. Login as student/Metadata0.

  5. Choose No when asked to opt in to all of your assumable groups.

  6. Click Program in SAS.

  7. Copy and paste the following SAS code in the “Code” window:

    data work.top10tuning;  
    set sashelp.cars;  
    do i=1 to 100;  
    output;  
    end;  
    run;
    
  8. Run the code (Hit the F3 key or click the button).

  9. Back to the Root Terminal, determine where your pod is running by running this command:

    kubectl -n edu get pods -l launcher.sas.com/username=student -o wide
    
  10. Let’s grab the pod name.

    PODNAME=$(kubectl -n edu get pods -l launcher.sas.com/username=student -o=custom-columns=PODNAME:.metadata.name --no-headers)
    echo "The SAS Compute server is running in the pod $PODNAME"
    
  11. and the pod’s node:

    NODENAME=$(kubectl -n edu get pods -l launcher.sas.com/username=student -o=custom-columns=NODENAME:.spec.nodeName --no-headers)
    echo "The SAS Compute server pod $PODNAME is running on node $NODENAME"
    
  12. Now, let’s look at the paths inside the pod:

    kubectl -n edu exec -it ${PODNAME} -c sas-programming-environment -- bash -c "ls -alR /opt/sas/viya/config/var/tmp/compsrv/default"
    
  13. You should see a file that corresponds to the top10tuning temporary SAS dataset.

  14. Finally, let’s look on the node:

    kubectl debug node/${NODENAME} -it --image=server.demo.sas.com:5000/busybox
    

    This command will open an interactive session on the node where the SAS Compute pod is running. You will see something like this:

  15. Once you are connected let’s see if we can see some SASWORK temporary files:

    ls -alR /host/saswork
    
  16. You should see a lot of files under our custom SASWORK location:

  17. If the output includes the top10tuning temporary SAS dataset, then it means that your configuration was successful!

  18. DO NOT FORGET to exit from the debug pod, by typing:

    exit
    

What have we learned ?

  • Changing the default location of the temporary storage areas for CAS and Compute requires changes in the kustomize configuration and a redeployment.

  • However, it is strongly recommended to provision and configure sufficient and highly-performant storage for them. The default location and volume types for CAS Disk Cache and SASWORK (Node’s root directories with the “emptyDir” volume should never be used in production environments).

  • In this example, to keep things simple we have used the hostPath volume type. However this option may not always be acceptable for security reasons. Alternative options exist, such as the Kubernetes generic ephemeral volume (with dynamic provisioning) or with the Cloud provider’s specific storage capabilities (for example AKS ephemeral OS disks in Azure).



Back to top

Copyright © SAS Institute Inc. All Rights Reserved.