Enterprise-grade GPUs can be a significant investment, and they require substantial power and cooling capabilities to maintain. For this reason, many organizations are turning to GPU as a service to process their large data workloads.
As more organizations embrace AI and advanced analytics, there has been a growing demand for high-performance computing that relies on GPUs. IT teams, though, must figure out how they’re going to implement and maintain the storage necessary to support GPUaaS initiatives.
GPUaaS is a cloud-based offering that virtualizes GPUs and makes them available as on-demand, scalable services that users can consume over the internet or a private network, similar to other types of cloud services. GPUaaS provides a way for organizations to carry out high-performance computing without needing to invest in costly physical infrastructure. Customers can also implement GPUaaS in their private clouds.
Many public cloud providers now offer GPUaaS capabilities, which come with several advantages. Customers can access computational power without the administrative overhead or extended delays. The service provider manages the systems, keeps them running and updates them as necessary. Users can scale the services up and down and connect to the services from anywhere they have an internet connection, leading to greater productivity and overall flexibility.
Despite the benefits, IT teams must safely store the data; make sure it is available to the GPU service whenever it is needed, regardless of the amount of data; and optimize performance while minimizing risk to sensitive information.
Best practices for GPUaaS storage
When setting up storage for GPUaaS, IT teams often work with data that is in multiple locations. This complexity can make it difficult for the teams to know how to proceed when planning their GPUaaS storage and data strategies. Here we provide seven best practices.
When setting up storage for GPUaaS, IT teams often work with data that is in multiple locations. This complexity can make it difficult for the teams to know how to proceed when planning their GPUaaS storage and data strategies.
1. Understand your storage and data requirements
Before you start repurposing hardware or signing up for new cloud services, you should have a clear understanding of your project’s goals and objectives as they relate to storage for your GPUaaS initiative. Know what problems you’re trying to address and what the overall expectations are for the project. You might be required to define service-level agreements that guarantee performance, availability, security or other deliverables.
Be as specific as possible when defining your storage and data requirements, particularly in terms of performance. You need to know how much throughput your storage and network should provide and the level of latency your organization can tolerate. You’ll also need to know how much data you’ll be expected to store in the short and long term, as well as how your organization will access and use that data.
2. Identify and assess existing systems and operations
Gather details about the storage systems and services you currently use, their capacities and performance capabilities, how they’re configured to facilitate data transfers and cross-system communications, and the networks that support these transfers and communications. You should also know how those systems and services are being maintained, including the tools that facilitate management and interoperability.
Determine whether you can use any of these resources to support the GPUaaS effort. If so, you’ll need to know what it will take to repurpose them for GPUaaS and what additional resources might be needed to augment the existing infrastructure. Assess the impact the repurposing might have on existing systems or operations.
3. Identify and assess existing data workflows
Identify your data sources, how your data pipelines operate, your storage locations and the amount and type of data. Examine your extract, transform and load operations so you understand if and how the data is being modified. If you already have a solid data management and governance strategy in place, much of this information should be readily available.
Once you have a complete picture of your data workflows, you can then assess whether you can modify or use them in some way to accommodate your GPUaaS effort. Assess how those modifications could potentially impact your storage infrastructure, as well as other systems or operations.
4. Plan the GPUaaS storage and data strategy
Develop a comprehensive plan that identifies which systems and services should be added or repurposed to accommodate the GPUaaS workflows. To this end, you must identify the type of storage, storage format and where data will reside. Consider bandwidth and latency requirements, as well as issues related to scalability, availability and fault tolerance.
Study various ways you can optimize your environment. For example, you might use a distributed or parallel file system or implement tiering or caching. Optimize your operations in other ways as well, such as with Nvidia’s GPUDirect Storage, adjusting your PCIe settings or using storage drives that support NVMe or NVMe-oF. The planning process should include careful cost assessments that analyze the total cost of ownership and return on investment. Finally, create a detailed roll-out strategy that you can follow when you’re ready to move forward.
5. Plan the data workflow strategy
Determine where to locate your data, considering factors such as security, privacy, cost and performance, as it relates to the data’s proximity to the GPUaaS platform. Decide whether to preprocess any data and, if so, when and where that should occur.
Planning your workflow can help you have the right storage in place and better prepare for data migration and synchronization operations. Once you’ve solidified a workflow strategy, thoroughly test your operations to verify they’re achieving optimal results.
6. Integrate data management and governance
Newly acquired or repurposed storage and new sources of data should not disrupt your ability to manage the data throughout its entire lifecycle, no matter where the data resides or whether it’s stored in a central repository such as a data lake. You should be able to catalog your data and track its lineage even when supporting GPUaaS.
Proper data safeguarding measures are essential. These might include encrypting data at rest and in motion, enforcing granular access controls, implementing identity management or deploying network segmentation. At the same time, ensure that your storage and data management strategies comply with applicable regulations and that you’ve implemented a disaster recovery plan to ensure business continuity.
7. Continuously monitor and optimize your systems
Your management tools should provide you with complete visibility into your storage and data environments. They should also support real-time alerting and notifications and make it possible to generate comprehensive reports that can be easily shared with key players.
Your organization should be able to act quickly upon the information gathered by your tools so you can troubleshoot and address security threats, anomalous behaviors, performance issues, service disruptions or other problems as quickly and efficiently as possible. Continuous monitoring can also help you track and optimize costs, as well as ensure conformance with applicable regulations.
Storage challenges with GPUaaS
Despite the advantages, setting up storage for GPUaaS can come with a number of challenges, including the following:
Operational complexity. Addressing the differences in tools, standards, platforms and operations can be difficult and time-consuming, often requiring advanced skills and expertise. Lack of complete visibility into public cloud environments can increase this complexity.
Performance and reliability. Storage systems and networks can sometimes struggle to keep up with the demands of a GPUaaS platform and must be continuously monitored and optimized to maintain the required performance.
Data locality and movement. Storing data at a geographic location different from the GPU servers increases the likelihood of slower data transfers and higher latency. Carefully plan the location and movement of data to reduce the likelihood of bottlenecks. In some cases, you might need to co-locate the data on the same cloud platform as the GPU servers to ensure the necessary performance.
Data management and governance. Maintaining large amounts of data across multiple environments can make it difficult to properly manage the data and ensure the necessary governance. Poor governance can increase the risk of data integrity, security or compliance issues.
Scalability and capacity management. If your GPUaaS workloads grow and fluctuate in unexpected ways, the storage systems might not be able to deliver the necessary performance and capacity, especially when dealing with large data sets. Frequent migrations and synchronizations can exacerbate this issue even further. Your storage systems and connecting networks must be designed for scalability.
System and data visibility. Without the proper visibility, an organization might experience performance issues or disruptions in operations or put its data at risk for security breaches and compliance violations. IT teams need to have the tools necessary to monitor their systems and gain real-time insights that help them track and address issues before they affect operations.
Skills and expertise. Failure to invest in skills — whether through training, bringing in experts or other means — can potentially affect performance, operations, security, compliance and data accessibility. Implementing storage for GPUaaS and the data infrastructure to support this initiative can be a complex undertaking that must balance a variety of moving parts.
Cost management. The more distributed the data, the more difficult it becomes to manage and optimize costs. The problem becomes even more of a challenge when your organization works with growing data volumes and needs to manage and store it in different ways. At the same time, IT teams must continuously deliver the high throughput and low latency that GPUaaS requires. Factor cost optimization into the project’s early planning, when considering options such as tiered storage, drive types and storage formats.
How vendors are responding
Nvidia has been at the forefront of the GPUaaS movement with its enterprise-grade GPUs, which are used extensively in enterprise data centers and large hyperscalers. Many of the cloud providers that offer GPUaaS use Nvidia GPUs as the backbone for their services. Google Cloud, Rackspace, Hyperstack, Liquid Web and Lambda Labs are just a few of the providers offering virtual GPU services based on Nvidia infrastructure.
To facilitate GPU connectivity, Nvidia provides GPUDirect Storage, a data technology that enables local and remote storage systems to communicate directly with GPU memory, using industry-standard protocols such as NVMe or NVMe-oF. GPUDirect Storage enables a direct memory access engine to move data into or out of GPU memory, without requiring the CPU.
Nvidia also offers virtual GPU (vGPU) software for virtualizing Nvidia GPUs and sharing them across multiple VMs. For example, an IT team can use VMware Cloud Director in conjunction with the vSphere platform and Nvidia’s vGPU software to create a GPUaaS environment for their organization.
Lenovo offers GPUaaS services as part of its TruScale infrastructure model, making it possible to provide customers with GPU services on demand. TruScale uses Nvidia GPUs such as the H100 and L40S. Dell and HPE have also partnered with Nvidia to provide similar services, enabling organizations to include GPUaaS in their private clouds. Like Lenovo, Dell and HPE are promoting their efforts to bring AI computing to the enterprise.
Nvidia GPUs also play a role in the latest products coming from Scan Computers, which offers GPU-accelerated AI compute products. The company recently announced that it has collaborated with Peak:AIO and Micron Technology to provide a range of AI data servers.
Peak:AIO is a software-defined storage platform that is optimized for GPU utilization. The software uses Nvidia’s GPUDirect Storage technology to support up to 10 data servers configured with Nvidia GPUs. Scan Computers has extensively tested its Peak:AIO systems with a variety of Nvidia-certified server architectures, including DGX, HGX and EGX.
Robert Sheldon is a freelance technology writer. He has written numerous books, articles and training materials on a wide range of topics, including big data, generative AI, 5D memory crystals, the dark web and the 11th dimension.
