Hands-On 7: Increase Compute pod’s CPU limit
In this hands-on, we will first look at the default CPU and memory limits and request for the SAS Compute server pods, and also monitor the CPU and memory consumption of the SAS Compute Server pod while a SAS program is running. Then we will change the max and default values for the CPU limits and memory and run the program again to see if the processing times are changing. (Tip #3)
Step-by-step instructions
Check the default CPU and Memory limits of your SAS Compute session
-
Open Google Chrome and click the SAS Studio bookmark or if you use the SAS App, click the hamburger icon (top left) and select Develop Code and Flows.
-
Login as student/Metadata0.
-
Choose Yes when asked to opt in to all of your assumable groups.
-
Click Program in SAS.
-
Switch from the SAS Studio Compute Context_SHARED Context to the standard SAS Studio Compute Context, confirm by clicking on Change.
-
Wait for the Studio Compute context to be ready, then check the current CPU and Memory requests and limits for the Compute container of the pod that you have just started.
- Run the following command in the Root Terminal window:
kubectl -n edu get pod -l 'launcher.sas.com/username=student' -o json | jq -r '.items[0].spec.containers[] | select (.name=="sas-programming-environment") | .name, .resources '
-
You should now see the default CPU and memory limits of the pod: 2 cores and 2Gi of memory.
-
Now let’s put a watch on the CPU and memory utilization for the pod, by running this command in the Root Terminal window:
watch 'kubectl -n edu top pod -l launcher.sas.com/username=student'
- With this command you can see the real time consumption of CPU and memory of your SAS Compute session.
Run a sample SAS Program
-
Come back to your SAS App (or Google Chrome) and the SAS Studio programming interface.
-
Click Program in SAS.
-
Copy and paste the following SAS Code that creates a sample dataset and perform a “high-performance” procedure named HPGENSELECT:
options fullstimer; /*display resource usage in the log*/ /*generate a sample dataset*/ %let numCopies=400000; data getStarted; input C1-C5 Y Total; do i=1 to &numCopies; output; end; drop i; datalines; 0 3 1 1 3 2 28.361 2 3 0 3 1 2 39.831 1 3 2 2 2 1 17.133 1 2 0 0 3 2 12.769 0 2 1 0 1 1 29.464 0 2 1 0 2 1 4.152 1 2 1 0 1 0 0.000 0 2 1 1 2 1 20.199 1 2 0 0 1 0 0.000 0 1 1 3 3 2 53.376 2 2 2 2 1 1 31.923 0 3 2 0 3 2 37.987 2 2 2 0 0 1 1.082 0 2 0 2 0 1 6.323 1 3 0 0 0 0 0.000 1 2 1 2 3 2 4.217 0 1 2 3 1 1 26.084 1 1 0 0 1 0 0.000 1 3 2 2 2 0 0.000 2 1 3 1 1 2 52.640 1 3 0 1 2 1 3.257 2 0 2 3 0 5 88.066 2 2 2 1 0 1 15.196 3 1 3 1 0 1 11.955 3 1 3 1 2 3 91.790 3 1 1 2 3 7 232.417 3 1 1 1 0 1 2.124 3 1 0 0 0 2 32.762 3 1 2 3 0 1 25.415 2 2 0 1 2 1 42.753 3 3 2 2 3 1 23.854 2 0 0 2 3 2 49.438 1 0 0 2 3 4 105.449 0 0 2 3 0 6 101.536 0 3 1 0 0 0 0.000 3 0 1 0 1 1 5.937 2 0 0 0 3 2 53.952 1 0 1 0 3 2 23.686 1 1 3 1 1 1 0.287 2 1 3 0 3 7 281.551 1 3 2 1 1 0 0.000 2 1 0 0 1 0 0.000 0 0 1 1 2 3 93.009 0 1 0 1 0 2 25.055 1 2 2 2 3 1 1.691 0 3 2 3 1 1 10.719 3 3 0 3 3 1 19.279 2 0 0 2 1 2 40.802 2 2 3 0 3 3 72.924 0 2 0 3 0 1 10.216 3 0 1 2 2 2 87.773 2 1 2 3 1 0 0.000 3 2 0 3 1 0 0.000 3 0 3 0 0 2 62.016 1 3 2 2 1 3 36.355 2 3 2 0 3 1 23.190 1 0 1 2 1 1 11.784 2 1 2 2 2 5 204.527 3 0 1 1 2 5 115.937 0 1 1 3 2 1 44.028 2 2 1 3 1 4 52.247 1 1 0 0 1 1 17.621 3 3 1 2 1 2 10.706 2 2 0 2 3 3 81.506 0 1 0 0 2 2 81.835 0 1 2 0 1 2 20.647 3 2 2 2 0 1 3.110 2 2 3 0 0 1 13.679 1 2 2 3 2 1 6.486 3 3 2 2 1 2 30.025 0 0 3 1 3 6 202.172 3 2 3 1 2 3 44.221 0 3 0 0 0 1 27.645 3 3 3 0 3 2 22.470 2 3 2 0 2 0 0.000 1 3 0 2 0 1 1.628 1 3 1 0 2 0 0.000 3 2 3 3 0 1 20.684 3 1 0 2 0 4 108.000 0 1 2 2 1 1 4.615 0 2 3 2 2 1 12.461 0 3 2 0 1 3 53.798 2 1 1 2 0 1 36.320 1 0 3 0 0 0 0.000 0 0 3 2 0 1 19.902 0 2 3 1 0 0 0.000 2 2 2 1 3 2 31.815 3 3 3 0 0 0 0.000 2 2 1 3 3 2 17.915 0 2 3 2 3 2 69.315 1 3 1 2 1 0 0.000 3 0 1 1 1 4 94.050 2 1 1 1 3 6 242.266 0 2 0 3 2 1 40.885 2 0 1 1 2 2 74.708 2 2 2 2 3 2 50.734 1 0 2 2 1 3 35.950 1 3 3 1 1 1 2.777 3 1 2 1 3 5 118.065 0 3 2 1 2 0 0.000 ; proc hpgenselect data=getStarted; class C1-C5; model Y = C1-C5 / Distribution=Poisson Link=Log; run;
-
Run the SAS program (Hit the F3 key or click the button).
-
While your program runs, bring up the Terminal window where you have set the watch on the CPU and memory usage, you should see the CPU and memory used value change.
-
At the end, click the Log tab and take a note of the “real time” and “user cpu time” for the last operation (PROC HPGENSELECT).
-
In the Root Terminal window, you can now interrupt the “watch” command by typing
<Ctrl+C>
.
Change the default CPU and Memory limits for the Compute Server
There are two configuration tasks to perform:
- Increase the maximum possible CPU and RAM limits for SAS Compute Server pod.
- Inrease default CPU and RAM limit for the SAS Compute Server pod.
--- Expand this ONLY if you want to use the SAS Environment Manager UI to do it ---
-
Switch to Environment Manager (select Manage Environment) and open the configuration page .
-
Using the drop-down list at the top, choose All services.
-
Select Launcher service (You can type “Launcher” in the filter to quickly locate the service).
-
The instances are expanded by default. Collapse all the items by clicking on the icon .
-
Now, beside the sas.launcher.max, click the Edit icon to modify the maximum values for CPU requests, CPU limits, memory requests, and memory limits.
-
Set the CPU limit to 4 (instead of 2 by default).
-
Set the Memory limit to 16Gi (instead of 2Gi by default).
-
-
Click Save.
-
Now, for the sas.launcher.default instance, click the Edit icon beside to modify the default values for the CPU and memory limit.
-
If we increase the CPU limit, we should also increase the memory limit so each CPU have more memory to work with.
-
Set the cpu.limit value to 4.
-
Set the Memory limit to 8Gi (instead of 2Gi).
-
-
Click Save.
Otherwhise follow the instructions below to do it from the command line (faster):
Increase the max limits for CPU and memory
-
If we increase the CPU limit, we should also increase the memory limit so each CPU have more memory to work with.
-
Get the identifier for
sas.launcher.max
definition:ITEM="sas.launcher.max" MEDIATYPE=$(sas-viya configuration configurations download -d $ITEM | jq -r '.items[]["metadata"]["mediaType"] ')
-
Create a JSON file (increase maximum “cpu.limit” from
2
to4
and increase maximum “memory.limit” from2Gi
to16Gi
):tee ./update-$ITEM.json > /dev/null << EOF { "items": [ { "metadata": { "isDefault": false, "mediaType": "${MEDIATYPE}" }, "cpu.limit": "4", "cpu.request": "100m", "memory.limit": "16Gi", "memory.request": "500M" } ] } EOF
-
Apply the change:
sas-viya configuration configurations update --file ./update-$ITEM.json
With result similar to:
"PATCH" "/configuration/configurations" complete from "./update-sas.launcher.max.json".
-
(Optional) Validate the change in SAS Environment Manager.
Navigate to Configuration (left-menu) > Definitions (View pull-down) > sas.launcher.max (from list) > Launcher service (parameter group) > cpu.limit (item) should have value of
4
(originally2
) and memory.limit (item) should have a value of16Gi
(originally2Gi
).
Increase the default CPU and memory limit
-
Get the identifier for
sas.launcher.default
definition:ITEM="sas.launcher.default" MEDIATYPE=$(sas-viya configuration configurations download -d $ITEM | jq -r '.items[]["metadata"]["mediaType"] ')
-
Create a JSON file (increase default “cpu.limit” from
2
to4
cores and the memory from2Gi
to8Gi
):tee ./update-$ITEM.json > /dev/null << EOF { "items": [ { "metadata": { "isDefault": false, "mediaType": "${MEDIATYPE}" }, "cpu.limit": "4", "cpu.request": "100m", "memory.limit": "8Gi", "memory.request": "500M" } ] } EOF
-
You might have noticed that we also increase the memory limits. If we increase the CPU limit, we should also increase the memory limit so each CPU have more memory to work with.
-
Apply the change:
sas-viya configuration configurations update --file ./update-$ITEM.json
With result similar to:
"PATCH" "/configuration/configurations" complete from "./update-sas.launcher.default.json".
-
Now wait about 20 seconds for the configuration change to propagate,
Run the sample program after the configuration changes
-
Now, come back to SAS Studio (Develop Code and Flows), and click Options in the menu, then select Reset SAS Session.
-
In the dialog box, confirm you want to reset your SAS Session (it will restart a Compute server pod with the new memory and CPU limits settings).
- Let’s check the CPU and Memory requests and limits for the Compute container you just started.
- Back to the Root Terminal window, interrupt the watch command with
<Ctrl+C>
. - Run the following command in the Root Terminal:
kubectl -n edu get pod -l 'launcher.sas.com/username=student' -o json | jq -r '.items[0].spec.containers[] | select (.name=="sas-programming-environment") | .name, .resources '
- Back to the Root Terminal window, interrupt the watch command with
-
You should now see the updated CPU limit value of 4 and memory limit value of 8Gi set for this pod.
Note: if you don’t see the new values set for the memory limit, try to do another reset, it can take a bit of time for the configuration change to be picked up
-
You can run the watch command to monitor the CPU and memory utilization again.
watch 'kubectl -n edu top pod -l launcher.sas.com/username=student'
-
Finally we can resubmit our SAS program (“hpgenselect-test.sas”).
-
Run the code, look at the real time and CPU time for the proc HPGENSELECT operation.
- Take a note of the “real time” and “user cpu time”.
- Did it improve the execution times ?
What have we learned ?
- Changing the default limits for the SAS Compute pods can improve the performances.
- The SAS High Performance procedure (such as the HPGENSELECT procedure used in this example) can really benefit from additional CPU processing.
- Similarly, The PROC SORT could take advantage of available memory to temporarily store data during the sort operations. If the amount of memory is too low for the amount of data, it can become a bottleneck. In such cases, allowing the utilization of more memory for the pod can reduce the execution times.
- Tools like the Grafana Dashboards (provided as part of the SAS Viya Monitoring tools) can help to determine if a lack of CPU or memory becomes a bottleneck for a given SAS procedure or data step.
References
- hands on / HO1 Enable Progressive App
- hands on / HO2 Allow more analytics jobs to run
- hands on / HO3 Create a pool of reusable compute servers
- hands on / HO4 Increase Compute pod CPU limit<-- you are here
- hands on / HO5 Tune temporary storage locations CAS DISK CACHE
- hands on / HO6 Tune temporary storage locations SASWORK