Scaling Memory for High-Performance Workloads: Huge Pages and Transparent Huge Pages in OpenShift

As applications and workloads become increasingly complex and demanding, optimizing system resources for maximum performance is essential. One way to do this is by utilizing Huge Pages and Transparent Huge Pages. These memory management techniques can significantly improve performance by reducing memory fragmentation and improving memory access times.
Managing memory is critical in an OpenShift cluster, where multiple containers are running simultaneously. Let's look closer at Huge Pages and Transparent Huge Pages and their role in an OpenShift cluster.

What are Huge Pages?

Huge Pages are a memory management technique allowing the operating system to allocate memory pages larger than the standard size of 4KB. Instead, Huge Pages can be 2MB, or even larger, depending on the system configuration. Using larger memory pages, the operating system can reduce memory fragmentation and improve performance by reducing the number of page table entries needed to manage the memory.
Huge Pages are beneficial for workloads that require large amounts of memory, such as databases or in-memory analytics applications. In these cases, using Huge Pages can significantly improve performance by reducing the number of page faults and page table entries needed to manage the memory.

Why Huge Pages?

Huge pages, also known as large pages or super pages, are a technique used by modern CPUs and operating systems to optimize the use of physical memory. When a process uses memory, the CPU and operating system must track which pages belong to which method and where each page is stored. This can become relatively inefficient when dealing with large amounts of memory. For instance, when a process uses 1GB of memory, the CPU and operating system must manage 262,144 individual 4K pages (1GB / 4K).

The CPU and operating system can optimize memory usage and reduce overhead by reducing the number of pages that need to be managed. Huge pages allow the CPU and operating system to manage larger memory pages, typically 2MB or 1GB, instead of the default 4K pages. With 2MB huge pages, for example, the same 1GB memory allocation only requires 256 entries to be tracked (1GB / 2MB). This can result in significant performance improvements for memory-intensive applications.

How do Huge Pages work?

The architecture of Huge Pages involves allocating large memory pages from the system memory pool. The operating system maintains a Huge Page Table that maps the virtual memory address space to the physical memory address space. When a process requests a large amount of memory, the operating system can allocate the memory as Huge Pages, which reduces the number of page table entries needed to manage the memory.
The main advantage of Huge Pages is that they reduce memory fragmentation, which can occur when a process frequently requests and releases small amounts of memory. With Huge Pages, the process can request and release large amounts of memory, which reduces the overhead of managing the memory and improves performance.

What are Transparent Huge Pages?

Transparent Huge Pages (THP) are a variation of Huge Pages that allow the operating system to automatically allocate and manage large memory pages without requiring any changes to the application code. THP is available on most modern Linux systems and is enabled by default in many distributions.
THP works by monitoring the memory access patterns of the application and automatically promoting small memory pages to large memory pages if it detects that the application is accessing a large contiguous block of memory. This can significantly improve performance by reducing the number of page faults and page table entries needed to manage the memory.

How do Transparent Huge Pages work?

The architecture of THP involves the operating system monitoring the memory access patterns of the application and automatically promoting small memory pages to large memory pages if it detects that the application is accessing a large contiguous block of memory. The THP Page Table maps the virtual memory address space to the physical memory address space, just like the Huge Page Table.

The main advantage of THP is that it allows the operating system to automatically manage memory allocation without requiring any changes to the application code. This can make optimizing the performance of applications not designed to use Huge Pages easier.

When to use Huge Pages and Transparent Huge Pages?

Huge Pages and Transparent Huge Pages are useful for workloads that require large amounts of memory and that access memory in a contiguous manner. Examples include databases, in-memory analytics applications, and high-performance computing applications.

It's important to note that not all applications benefit from using Huge Pages or Transparent Huge Pages. Workloads that frequently allocate and deallocate small amounts of memory may experience decreased performance due to increased memory fragmentation.

Enable 1G Huge Page using grub

To enable 1GB of huge pages and configure the number of huge pages on a Linux system, you can use the following steps:
Check if your system supports 1GB huge pages:

grep -i huge /proc/meminfo

This command will display information about the huge pages supported by your system. Look for the line that starts with Hugepagesize:. If it shows 2048 kB, your system supports 2MB huge pages; if it displays 1048576 kB, then your system supports 1GB huge pages.
Edit the /etc/default/grub file:

sudo vi /etc/default/grub

Add the following line to the file:

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=128"

This line specifies that the default huge page size and size of the huge pages to be used should be 1GB. The hugepages=128 option specifies the number of 1GB huge pages that should be allocated for use by the system.

Update the GRUB configuration:

sudo update-grub

This command updates the GRUB configuration file with the changes made in the previous step.

Reboot the system:

sudo reboot

This will reboot the system with the new GRUB configuration.
Verify the huge page configuration:

grep -i huge /proc/meminfo

This command will display information about the huge pages supported by your system. Look for the lines that start with Hugepagesize: and HugePages_Total:. The former should display 1048576 kB, and the latter should display the 1GB huge pages you configured in step 2.

Output:

AnonHugePages:     26624 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:     128
HugePages_Free:      124
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        163577856 kB

Enabling Huge Page using sysctl

You can also use sysctl command to enable/disable Huge Pages as follows,

sudo sysctl -w vm.nr_hugepages=<number of Huge Pages>

Replace <number of Huge Pages> with the desired number of Huge Pages. For example, to set 2048 2MB Huge Pages, run the following commands:

sudo sysctl -w vm.nr_hugepages=2048

To make these changes persistent, add the following lines to the /etc/sysctl.conf file:

vm.nr_hugepages=2048

Save and close the file.

To apply the changes, run the following command:

sudo sysctl -p /etc/sysctl.conf

This command will read the settings from the /etc/sysctl.conf file and apply them to the system.

To check the updated settings, run the following command:

cat /proc/cmdline

The output should show the updated Huge Pages settings. If the settings have not been applied, restart the system or execute the command sudo sysctl -p /etc/sysctl.conf again.

To refresh the kernel parameters without rebooting the system, run the following command:

sudo sysctl --system

This command will reload the /etc/sysctl.conf file and apply the changes to the system.

Disable Huge Pages

Following command will disable huge pages completely.

echo 0 | sudo tee /proc/sys/vm/nr_hugepages

Disable 2MB hugepages

You can use following command to disable 2MB hugepages.

echo 0 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 0 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

Enable/Disable transparent huge pages

To enable transparent huge pages, run the following command with root privileges:

echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

This command writes the value always to the file /sys/kernel/mm/transparent_hugepage/enabled, which will enable transparent huge pages.

To disable transparent huge pages, run the following command with root privileges:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

This command writes the value never to the file /sys/kernel/mm/transparent_hugepage/enabled, which will disable transparent huge pages.

Note that disabling transparent huge pages may negatively impact system performance, so it should only be done if necessary for certain workloads.

Enabling Huge Pages in OpenShift

Enabling Huge Pages in OpenShift is a straightforward process that involves modifying the kernel boot parameters for the machine. You can use following machine config configuration to achieve this easily.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-openshift-machineconfig-worker-hugepage
spec:
  kernelArguments:
    - default_hugepagesz=1G
    - hugepagesz=1G
    - hugepages=221
    - hugepagesz=2M
    - hugepages=4096
    - transparent_hugepage=madvise

In this method, we are configuring the Linux kernel on each worker node to allocate Huge Pages using kernel boot parameters. The kernelArguments field in the YAML definition specifies the values to be set for these kernel boot parameters.

First, we set the default Huge Page size to 1GB using the default_hugepagesz parameter. Any application that requests Huge Pages without specifying a size will automatically receive 1GB of Huge Pages.

Next, we specify 221 1GB Huge Pages using the huge pages parameter. This reserves a contiguous memory block on each worker node for Huge Pages. This memory block is then divided into the number of Huge Pages requested, in this case, 221.

In addition to 1GB Huge Pages, we are setting the Huge Page size to 2MB using the hugepagesz parameter. This allows applications to request 2MB Huge Pages if needed.

Finally, we are setting the Transparent Huge Pages to use the madvise method using the transparent_hugepage parameter. The madvise method allows the kernel to automatically split huge pages into smaller pages if required by the application. This can help to reduce memory waste and improve performance.

By setting these kernel boot parameters using MachineConfigs, we can ensure that Huge Pages are consistently and automatically configured across all worker nodes in an OpenShift cluster. This provides a more scalable and efficient way to manage Huge Pages than manually configuring each node individually.