From fb499dcf4ada931586ce4e6e780182eb9622bd86 Mon Sep 17 00:00:00 2001 From: hhhfccz Date: Thu, 19 Aug 2021 17:17:40 +0800 Subject: [PATCH 1/8] add: RFC of add oneflow frontend --- rfcs/0024-add-oneflow-frontend.md | 101 ++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 rfcs/0024-add-oneflow-frontend.md diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md new file mode 100644 index 00000000..32e1ea51 --- /dev/null +++ b/rfcs/0024-add-oneflow-frontend.md @@ -0,0 +1,101 @@ +- Feature Name: (`add oneflow frontend`) +- Start Date: (2021-8-20) +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/) + +# Summary +[summary]: #summary + +To enhance the compatibility of TVM with deep learning frameworks, +we have created a frontend for TVM that targets [oneflow](https://github.com/Oneflow-Inc/oneflow) + +# Motivation +[motivation]: #motivation + +# TODO + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +We use a simple API to help users convert oneflow(eager) model to tvm relay + +```python +relay.frontend.from_oneflow(graph, model_dir_path, freeze_params=True, user_input=None) +``` + +- graph: +- model_dir_path: +- freeze_params: +- user_input: + +The following codes will show how to convert a oneflow model to a tvm relay + +```python +import tvm +import tvm.relay as relay + +import oneflow as flow + + +class Graph(flow.nn.Graph): + def __init__(self, module): + super().__init__() + self.m = module + + def build(self, x): + out = self.m(x) + return out + + +# load eager model +res50_module = resnet50() +pretrain_models = flow.load(model_path) +res50_module.load_state_dict(pretrain_models) +res50_module.eval().to("cuda") + +# load test image +image = load_image(image_path) +image_flow = flow.Tensor(image, device=flow.device("cuda")) + +# use Graph convert eager to lazy +res50_graph = Graph(res50_module) +_ = res50_graph._compile(image_flow) + +# get tvm relay +mod, params = relay.frontend.from_oneflow(res50_graph, model_path) +``` + +If user writes a model that contains an op that is not yet supported, the program will report an error. + +More demos could be seen at [this](https://github.com/Oneflow-Inc/oneflow_convert_tools/tree/tvm_oneflow/oneflow_tvm) + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +# TODO + +# Drawbacks +[drawbacks]: #drawbacks + +# TODO(hujiakui) + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +# TODO(hujiakui) + +# Prior art +[prior-art]: #prior-art + +It's the first time we have added a OneFlow frontend to an ML compiler. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any proslems, please let me know. + +# Future possibilities +[future-possibilities]: #future-possibilities + +For this RFC, we have made an established plan, +# TODO From 278d519f9748d52809475a5f81bb01dc352a23b6 Mon Sep 17 00:00:00 2001 From: hhhfccz Date: Fri, 20 Aug 2021 14:53:06 +0800 Subject: [PATCH 2/8] add: motivation and so on --- rfcs/0024-add-oneflow-frontend.md | 78 ++++++++++++++++++++++--------- 1 file changed, 55 insertions(+), 23 deletions(-) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index 32e1ea51..533610c6 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -12,28 +12,19 @@ we have created a frontend for TVM that targets [oneflow](https://github.com/One # Motivation [motivation]: #motivation -# TODO +OneFlow, an open source deep learning framework with whole new frame design and the world's leading technology for distributed system. Here are advantages of OneFlow: -# Guide-level explanation -[guide-level-explanation]: #guide-level-explanation +- Perfectly support container platforms(k8s & docker) +- Handle large models easily +- Almost zero runtime overhead & linear speedup +- Support automatic mixed precision +- ... -We use a simple API to help users convert oneflow(eager) model to tvm relay +We are proud that OneFlow can support basic CNNs as well as very large pre-trained models, such as [GPT3, BERT, etc](https://github.com/Oneflow-Inc/OneFlow-Benchmark/tree/master/LanguageModeling). They are built using an early version of OneFlow, based on lazy mode. -```python -relay.frontend.from_oneflow(graph, model_dir_path, freeze_params=True, user_input=None) -``` - -- graph: -- model_dir_path: -- freeze_params: -- user_input: - -The following codes will show how to convert a oneflow model to a tvm relay +Currently, oneflow(nightly) supports the conversion of eager models to lazy graphs. We can quickly convert the eager model built by OneFlow to a lazy graph with the following code. ```python -import tvm -import tvm.relay as relay - import oneflow as flow @@ -47,7 +38,41 @@ class Graph(flow.nn.Graph): return out -# load eager model +# module: eager model built by OneFlow +graph = Graph(module) +``` + +Because of these features we wrote this `from_oneflow` which is based on lazy mode. + +We also note that many developers have converted their models to ONNX format for compatibility with TVM, and this part of the work is ongoing, as you can see [here](https://github.com/Oneflow-Inc/oneflow_convert_tools). + +Based on this background, we proposed this RFC to add a OneFlow frontend for TVM, improving usability for OneFlow users and enhancing the compatibility between OneFlow and TVM. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +We use a simple API to help users convert oneflow to tvm relay. + +```python +relay.frontend.from_oneflow(graph, model_dir_path, freeze_params=True, user_input=None) +``` + +- graph: flow.nn.Graph, contains information about the nodes of the model +- model_dir_path: str, path of parameters +- freeze_params: bool, if this parameter is False, then the user can specify the input of the graph +- user_input: dict, information about the specified input of the model + +> NOTES: +> We prefer to let the user change the model node information by changing the model itself + +The following codes will show how to convert a oneflow model to a tvm relay + +```python +import tvm +import tvm.relay as relay + + +# load eager model(assuming that resnet50 has been built) res50_module = resnet50() pretrain_models = flow.load(model_path) res50_module.load_state_dict(pretrain_models) @@ -67,22 +92,27 @@ mod, params = relay.frontend.from_oneflow(res50_graph, model_path) If user writes a model that contains an op that is not yet supported, the program will report an error. -More demos could be seen at [this](https://github.com/Oneflow-Inc/oneflow_convert_tools/tree/tvm_oneflow/oneflow_tvm) +More demos could be seen at [this](https://github.com/Oneflow-Inc/oneflow_convert_tools/tree/tvm_oneflow/oneflow_tvm). # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -# TODO +Since the purpose of this RFC is only to add a new front-end for converting the OneFlow model to TVM Relay IR, other functions will not be affected. + +In this proposed RFC, the whole process of OneFlow frontend conversion can be divided into 2 steps: + +1. Parse Graph: Read the node information contained in the graph, extract the size, data type of each node and construct the computational graph by each computational node (in oneflow, these nodes are marked as user_conf). At the same time, we get the data information of the parameter nodes (in oneflow, these nodes are labeled as variable_conf) according to the paths where the parameters are provided by the user. +2. Convert operators: After the exported inference graph is loaded, we will extract its parameters and convert operators one by one. The order of all operator conversions will be based on the flow of the graph. # Drawbacks [drawbacks]: #drawbacks -# TODO(hujiakui) +Potential increase in time-cost of unit tests. # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -# TODO(hujiakui) +The frontend of OneFlow is similar to ONNX. For now, we consider only supporting conversion via `flow.nn.Graph`, as this API will be rolled out and recommended starting with OneFlow 0.5.0. In the meantime, we will develop the latest OneFlow eager model to ONNX format conversion script based on the existing OneFlow lazy model conversion to ONNX format to make it easy to work with TVM. # Prior art [prior-art]: #prior-art @@ -98,4 +128,6 @@ We will add new unit test cases that rely on OneFlow framework, and this may inc [future-possibilities]: #future-possibilities For this RFC, we have made an established plan, -# TODO + +- Support OneFlow Eager to ONNX +- Some operators will be supported in this quarter From 08ebce86e042b0ed41596d415fc8a4bdce069bdf Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Date: Fri, 20 Aug 2021 15:03:11 +0800 Subject: [PATCH 3/8] Update 0024-add-oneflow-frontend.md --- rfcs/0024-add-oneflow-frontend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index 533610c6..7f59f3bd 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -122,7 +122,7 @@ It's the first time we have added a OneFlow frontend to an ML compiler. # Unresolved questions [unresolved-questions]: #unresolved-questions -We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any proslems, please let me know. +We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any problems, please let me know. # Future possibilities [future-possibilities]: #future-possibilities From 968465d0173c8a489702e4936010f45ab705d984 Mon Sep 17 00:00:00 2001 From: hhhfccz Date: Fri, 20 Aug 2021 15:08:20 +0800 Subject: [PATCH 4/8] issue --- rfcs/0024-add-oneflow-frontend.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index 533610c6..87735e53 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -1,7 +1,7 @@ -- Feature Name: (`add oneflow frontend`) +- Feature Name: (add oneflow frontend) - Start Date: (2021-8-20) -- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/) -- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/) +- RFC PR: [apache/tvm-rfcs#0024](https://github.com/apache/tvm-rfcs/pull/0024) +- GitHub Issue: [apache/tvm#8804](https://github.com/apache/tvm/issues/8804) # Summary [summary]: #summary From 8a5e13d127639c548d2de0f8916f178c792eaf0e Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Date: Fri, 20 Aug 2021 15:10:28 +0800 Subject: [PATCH 5/8] Update 0024-add-oneflow-frontend.md --- rfcs/0024-add-oneflow-frontend.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index 52e95571..5ac6e129 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -1,7 +1,7 @@ -- Feature Name: (add oneflow frontend) +- Feature Name: (`add oneflow frontend`) - Start Date: (2021-8-20) -- RFC PR: [apache/tvm-rfcs#0024](https://github.com/apache/tvm-rfcs/pull/0024) -- GitHub Issue: [apache/tvm#8804](https://github.com/apache/tvm/issues/8804) +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/) # Summary [summary]: #summary @@ -38,7 +38,7 @@ class Graph(flow.nn.Graph): return out -# module: eager model built by OneFlow +# module: eager model built by OneFlow, align with Pytorch's python api. graph = Graph(module) ``` @@ -57,7 +57,7 @@ We use a simple API to help users convert oneflow to tvm relay. relay.frontend.from_oneflow(graph, model_dir_path, freeze_params=True, user_input=None) ``` -- graph: flow.nn.Graph, contains information about the nodes of the model +- graph: flow.nn.Graph, contains information about the nodes and edges of the model - model_dir_path: str, path of parameters - freeze_params: bool, if this parameter is False, then the user can specify the input of the graph - user_input: dict, information about the specified input of the model @@ -122,7 +122,7 @@ It's the first time we have added a OneFlow frontend to an ML compiler. # Unresolved questions [unresolved-questions]: #unresolved-questions -We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any problems, please let me know. +We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any proslems, please let me know. # Future possibilities [future-possibilities]: #future-possibilities From 6d11ca097a9d36cd037e897d4c01119e043dea97 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Date: Fri, 20 Aug 2021 15:11:43 +0800 Subject: [PATCH 6/8] Update 0024-add-oneflow-frontend.md --- rfcs/0024-add-oneflow-frontend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index 5ac6e129..3349e8a6 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -122,7 +122,7 @@ It's the first time we have added a OneFlow frontend to an ML compiler. # Unresolved questions [unresolved-questions]: #unresolved-questions -We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any proslems, please let me know. +We will add new unit test cases that rely on OneFlow framework, and this may increase time-cost of unit tests. If there are any problems, please let me know. # Future possibilities [future-possibilities]: #future-possibilities From b8cc388c305d53fbcc90421121cb1069c494c364 Mon Sep 17 00:00:00 2001 From: JiaKui Hu Date: Fri, 20 Aug 2021 15:16:07 +0800 Subject: [PATCH 7/8] Update 0024-add-oneflow-frontend.md --- rfcs/0024-add-oneflow-frontend.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index 3349e8a6..d99e8605 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -1,7 +1,7 @@ - Feature Name: (`add oneflow frontend`) - Start Date: (2021-8-20) -- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/) -- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/) +- RFC PR: [apache/tvm-rfcs#0024](https://github.com/apache/tvm-rfcs/pull/24) +- GitHub Issue: [apache/tvm#8804](https://github.com/apache/tvm/issues/8804) # Summary [summary]: #summary From 7df915fc34793c14f1632e8b180bbd6e10d51173 Mon Sep 17 00:00:00 2001 From: JiaKui Hu Date: Fri, 20 Aug 2021 17:29:40 +0800 Subject: [PATCH 8/8] tvmc --- rfcs/0024-add-oneflow-frontend.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0024-add-oneflow-frontend.md b/rfcs/0024-add-oneflow-frontend.md index d99e8605..3b6bbbc3 100644 --- a/rfcs/0024-add-oneflow-frontend.md +++ b/rfcs/0024-add-oneflow-frontend.md @@ -129,5 +129,6 @@ We will add new unit test cases that rely on OneFlow framework, and this may inc For this RFC, we have made an established plan, +- cover `tvmc`, like [this](https://github.com/apache/tvm/blob/e691c7f83892d7242d0992c78cec1e2f8953a9e3/python/tvm/driver/tvmc/frontends.py#L174-L198) - Support OneFlow Eager to ONNX - Some operators will be supported in this quarter