diff --git a/.readthedocs.yaml b/.readthedocs.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..dea9d45a14232cb76f663bccaaff1968d807a060
--- /dev/null
+++ b/.readthedocs.yaml
@@ -0,0 +1,18 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+version: 2
+
+sphinx:
+ configuration: docs/conf.py
+
+formats: [htmlzip]
+
+python:
+ install:
+ - requirements: docs/sphinx/requirements.txt
+
+build:
+ os: ubuntu-22.04
+ tools:
+ python: "3.10"
diff --git a/README.md b/README.md
index fbcaf8f5ffffff4d323f3eba4c38fe5b61c98e3e..c798f1a9844d7ff025e3a43d5356678b8ba67900 100644
--- a/README.md
+++ b/README.md
@@ -1,29 +1,35 @@
# AMD GPU device plugin for Kubernetes
+
[](https://goreportcard.com/report/github.com/ROCm/k8s-device-plugin)
## Introduction
+
This is a [Kubernetes][k8s] [device plugin][dp] implementation that enables the registration of AMD GPU in a container cluster for compute workload. With the appropriate hardware and this plugin deployed in your Kubernetes cluster, you will be able to run jobs that require AMD GPU.
More information about [ROCm][rocm].
-
## Prerequisites
+
* [ROCm capable machines][sysreq]
* [kubeadm capable machines][kubeadm] (if you are using kubeadm to deploy your k8s cluster)
* [ROCm kernel][rock] ([Installation guide][rocminstall]) or latest AMD GPU Linux driver ([Installation guide][amdgpuinstall])
* A [Kubernetes deployment][k8sinstall]
* If device health checks are enabled, the pods must be allowed to run in privileged mode (for example the `--allow-privileged=true` flag for kube-apiserver), in order to access `/dev/kfd`
-
## Limitations
+
* This plugin targets Kubernetes v1.18+.
## Deployment
+
The device plugin needs to be run on all the nodes that are equipped with AMD GPU. The simplest way of doing so is to create a Kubernetes [DaemonSet][ds], which runs a copy of a pod on all (or some) Nodes in the cluster. We have a pre-built Docker image on [DockerHub][dhk8samdgpudp] that you can use for your DaemonSet. This repository also has a pre-defined yaml file named `k8s-ds-amdgpu-dp.yaml`. You can create a DaemonSet in your Kubernetes cluster by running this command:
+
```
-$ kubectl create -f k8s-ds-amdgpu-dp.yaml
+kubectl create -f k8s-ds-amdgpu-dp.yaml
```
+
or directly pull from the web using
+
```
kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml
```
@@ -35,25 +41,29 @@ If you want to enable the experimental device health check, please use `k8s-ds-a
If you want to deploy this device plugin using Helm, a [Helm Chart][helmamdgpu] is available via [Artifact Hub][artifacthub].
## Example workload
+
You can restrict workloads to a node with a GPU by adding `resources.limits` to the pod definition. An example pod definition is provided in `example/pod/alexnet-gpu.yaml`. This pod runs the timing benchmark for AlexNet on AMD GPU and then goes to sleep. You can create the pod by running:
+
```
-$ kubectl create -f alexnet-gpu.yaml
+kubectl create -f alexnet-gpu.yaml
```
or
```
-$ kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/example/pod/alexnet-gpu.yaml
+kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/example/pod/alexnet-gpu.yaml
```
and then check the pod status by running
+
```
-$ kubectl describe pods
+kubectl describe pods
```
After the pod is created and running, you can see the benchmark result by running:
+
```
-$ kubectl logs alexnet-tf-gpu-pod alexnet-tf-gpu-container
+kubectl logs alexnet-tf-gpu-pod alexnet-tf-gpu-container
```
For comparison, an example pod definition of running the same benchmark with CPU is provided in `example/pod/alexnet-cpu.yaml`.
@@ -61,22 +71,24 @@ For comparison, an example pod definition of running the same benchmark with CPU
## Labelling node with additional GPU properties
Please see [AMD GPU Kubernetes Node Labeller](cmd/k8s-node-labeller/README.md) for details. An example configuration is in [k8s-ds-amdgpu-labeller.yaml](k8s-ds-amdgpu-labeller.yaml):
+
```
-$ kubectl create -f k8s-ds-amdgpu-labeller.yaml
+kubectl create -f k8s-ds-amdgpu-labeller.yaml
```
or
```
-$ kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml
+kubectl create -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-labeller.yaml
```
-
## Notes
+
* This plugin uses [`go modules`][gm] for dependencies management
* Please consult the `Dockerfile` on how to build and use this plugin independent of a docker image
## TODOs
+
* Add proper GPU health check (health check without `/dev/kfd` access.)
[artifacthub]: https://artifacthub.io/packages/helm/amd-gpu-helm/amd-gpu
diff --git a/docs/about/license.md b/docs/about/license.md
new file mode 100644
index 0000000000000000000000000000000000000000..4ef8847e035a6f6de26e97342e0e0cdab71f0fa3
--- /dev/null
+++ b/docs/about/license.md
@@ -0,0 +1,4 @@
+# License
+
+```{include} ../../LICENSE
+```
diff --git a/docs/conf.py b/docs/conf.py
new file mode 100644
index 0000000000000000000000000000000000000000..7131f59169261947d382657ca314bf0e46b84544
--- /dev/null
+++ b/docs/conf.py
@@ -0,0 +1,23 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# configurations for PDF output by Read the Docs
+project = "k8s-device-plugin Documentation"
+author = "Advanced Micro Devices, Inc."
+copyright = "Copyright (c) 2024 Advanced Micro Devices, Inc. All rights reserved."
+version = "6.2.0"
+release = "6.2.0"
+
+external_toc_path = "./sphinx/_toc.yml"
+
+extensions = ["rocm_docs"]
+
+external_projects_current_project = "rocm"
+
+html_theme = "rocm_docs_theme"
+html_theme_options = {"flavor": "rocm-docs-home"}
+
+html_title = project
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..12746b708597afec4714a3ba7e912a7541dd7073
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,4 @@
+# k8s-device-plugin
+
+```{include} ../README.md
+```
diff --git a/docs/sphinx/_toc.yml.in b/docs/sphinx/_toc.yml.in
new file mode 100644
index 0000000000000000000000000000000000000000..5d47b9527ad02f88596be413b23e79cc5cca1478
--- /dev/null
+++ b/docs/sphinx/_toc.yml.in
@@ -0,0 +1,8 @@
+defaults:
+ numbered: False
+ maxdepth: 6
+root: index
+subtrees:
+- caption: About
+ entries:
+ - file: about/license
diff --git a/docs/sphinx/requirements.in b/docs/sphinx/requirements.in
new file mode 100644
index 0000000000000000000000000000000000000000..9ad6c5d8be7979c748b302d4e3b5ced4a0695338
--- /dev/null
+++ b/docs/sphinx/requirements.in
@@ -0,0 +1 @@
+rocm-docs-core==1.6.2
diff --git a/docs/sphinx/requirements.txt b/docs/sphinx/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a842fed1e1e2051dc589e1613219ace07e4192b5
--- /dev/null
+++ b/docs/sphinx/requirements.txt
@@ -0,0 +1,147 @@
+#
+# This file is autogenerated by pip-compile with Python 3.10
+# by the following command:
+#
+# pip-compile requirements.in
+#
+accessible-pygments==0.0.5
+ # via pydata-sphinx-theme
+alabaster==1.0.0
+ # via sphinx
+babel==2.16.0
+ # via
+ # pydata-sphinx-theme
+ # sphinx
+beautifulsoup4==4.12.3
+ # via pydata-sphinx-theme
+breathe==4.35.0
+ # via rocm-docs-core
+certifi==2024.7.4
+ # via requests
+cffi==1.17.0
+ # via
+ # cryptography
+ # pynacl
+charset-normalizer==3.3.2
+ # via requests
+click==8.1.7
+ # via sphinx-external-toc
+cryptography==43.0.0
+ # via pyjwt
+deprecated==1.2.14
+ # via pygithub
+docutils==0.21.2
+ # via
+ # breathe
+ # myst-parser
+ # pydata-sphinx-theme
+ # sphinx
+fastjsonschema==2.20.0
+ # via rocm-docs-core
+gitdb==4.0.11
+ # via gitpython
+gitpython==3.1.43
+ # via rocm-docs-core
+idna==3.7
+ # via requests
+imagesize==1.4.1
+ # via sphinx
+jinja2==3.1.4
+ # via
+ # myst-parser
+ # sphinx
+markdown-it-py==3.0.0
+ # via
+ # mdit-py-plugins
+ # myst-parser
+markupsafe==2.1.5
+ # via jinja2
+mdit-py-plugins==0.4.1
+ # via myst-parser
+mdurl==0.1.2
+ # via markdown-it-py
+myst-parser==4.0.0
+ # via rocm-docs-core
+packaging==24.1
+ # via
+ # pydata-sphinx-theme
+ # sphinx
+pycparser==2.22
+ # via cffi
+pydata-sphinx-theme==0.15.4
+ # via
+ # rocm-docs-core
+ # sphinx-book-theme
+pygithub==2.3.0
+ # via rocm-docs-core
+pygments==2.18.0
+ # via
+ # accessible-pygments
+ # pydata-sphinx-theme
+ # sphinx
+pyjwt[crypto]==2.9.0
+ # via pygithub
+pynacl==1.5.0
+ # via pygithub
+pyyaml==6.0.2
+ # via
+ # myst-parser
+ # rocm-docs-core
+ # sphinx-external-toc
+requests==2.32.3
+ # via
+ # pygithub
+ # sphinx
+rocm-docs-core==1.6.2
+ # via -r requirements.in
+smmap==5.0.1
+ # via gitdb
+snowballstemmer==2.2.0
+ # via sphinx
+soupsieve==2.5
+ # via beautifulsoup4
+sphinx==8.0.2
+ # via
+ # breathe
+ # myst-parser
+ # pydata-sphinx-theme
+ # rocm-docs-core
+ # sphinx-book-theme
+ # sphinx-copybutton
+ # sphinx-design
+ # sphinx-external-toc
+ # sphinx-notfound-page
+sphinx-book-theme==1.1.3
+ # via rocm-docs-core
+sphinx-copybutton==0.5.2
+ # via rocm-docs-core
+sphinx-design==0.6.1
+ # via rocm-docs-core
+sphinx-external-toc==1.0.1
+ # via rocm-docs-core
+sphinx-notfound-page==1.0.4
+ # via rocm-docs-core
+sphinxcontrib-applehelp==2.0.0
+ # via sphinx
+sphinxcontrib-devhelp==2.0.0
+ # via sphinx
+sphinxcontrib-htmlhelp==2.1.0
+ # via sphinx
+sphinxcontrib-jsmath==1.0.1
+ # via sphinx
+sphinxcontrib-qthelp==2.0.0
+ # via sphinx
+sphinxcontrib-serializinghtml==2.0.0
+ # via sphinx
+tomli==2.0.1
+ # via sphinx
+typing-extensions==4.12.2
+ # via
+ # pydata-sphinx-theme
+ # pygithub
+urllib3==2.2.2
+ # via
+ # pygithub
+ # requests
+wrapt==1.16.0
+ # via deprecated