From eb445035a7bf2be3faf5c97dd2a8b3600c229acd Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Fri, 25 Oct 2024 14:24:21 +0800 Subject: [PATCH 1/9] doc: graph: create folder for complex fusions --- .../{ => complex_fusion}/images/sdpa-mask-1.png | Bin .../{ => complex_fusion}/images/sdpa-mask-2.png | Bin .../{ => complex_fusion}/images/sdpa-reorder.png | Bin doc/graph/{ => complex_fusion}/images/sdpa.png | Bin doc/graph/{ => complex_fusion}/sdpa.md | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename doc/graph/{ => complex_fusion}/images/sdpa-mask-1.png (100%) rename doc/graph/{ => complex_fusion}/images/sdpa-mask-2.png (100%) rename doc/graph/{ => complex_fusion}/images/sdpa-reorder.png (100%) rename doc/graph/{ => complex_fusion}/images/sdpa.png (100%) rename doc/graph/{ => complex_fusion}/sdpa.md (100%) diff --git a/doc/graph/images/sdpa-mask-1.png b/doc/graph/complex_fusion/images/sdpa-mask-1.png similarity index 100% rename from doc/graph/images/sdpa-mask-1.png rename to doc/graph/complex_fusion/images/sdpa-mask-1.png diff --git a/doc/graph/images/sdpa-mask-2.png b/doc/graph/complex_fusion/images/sdpa-mask-2.png similarity index 100% rename from doc/graph/images/sdpa-mask-2.png rename to doc/graph/complex_fusion/images/sdpa-mask-2.png diff --git a/doc/graph/images/sdpa-reorder.png b/doc/graph/complex_fusion/images/sdpa-reorder.png similarity index 100% rename from doc/graph/images/sdpa-reorder.png rename to doc/graph/complex_fusion/images/sdpa-reorder.png diff --git a/doc/graph/images/sdpa.png b/doc/graph/complex_fusion/images/sdpa.png similarity index 100% rename from doc/graph/images/sdpa.png rename to doc/graph/complex_fusion/images/sdpa.png diff --git a/doc/graph/sdpa.md b/doc/graph/complex_fusion/sdpa.md similarity index 100% rename from doc/graph/sdpa.md rename to doc/graph/complex_fusion/sdpa.md From b5b3d559dbbf790679c3795449be18b8e16045e3 Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Tue, 12 Nov 2024 10:40:28 +0800 Subject: [PATCH 2/9] doc: graph: sdpa: memory usage for ref implementation --- doc/graph/complex_fusion/sdpa.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/graph/complex_fusion/sdpa.md b/doc/graph/complex_fusion/sdpa.md index 1b0864a5c76..28ca84c8353 100644 --- a/doc/graph/complex_fusion/sdpa.md +++ b/doc/graph/complex_fusion/sdpa.md @@ -94,7 +94,11 @@ platforms follow the general description in @ref dev_guide_data_types. floating point SDPA patterns are usually implemented with f32/bf16/f16 matmul (with post-ops) and softmax primitives, while quantized SDPA patterns are implemented with int8 matmul (with post-ops) and f32/bf16/f16 softmax - primitives. + primitives. The reference implementation requires memory to store the + intermediate results of the dot products between Query and Key which takes + \f$O(S^2)\f$ memory. It may lead to Out-of-Memory when computing long + sequence length input on platforms with limited memory. + 2. The SDPA patterns functionally supports all input shapes meeting the shape requirements of each operation in the graph. For example, Add, Multiply, Divide, and Select operations require the input tensors to have the same From c762c086513d91e5b8426b97331da73ef162d8df Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Tue, 3 Sep 2024 13:36:21 +0800 Subject: [PATCH 3/9] doc: graph: sdpa: example and reference link for mqa --- doc/graph/complex_fusion/sdpa.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/graph/complex_fusion/sdpa.md b/doc/graph/complex_fusion/sdpa.md index 28ca84c8353..b5a37e383e9 100644 --- a/doc/graph/complex_fusion/sdpa.md +++ b/doc/graph/complex_fusion/sdpa.md @@ -125,8 +125,16 @@ example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/sdpa.cpp) demonstrating how to construct a typical floating point SDPA pattern with oneDNN Graph API on CPU and GPU with different runtimes. +oneDNN also provides a [MQA (Multi-Query Attention) +example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/mqa.cpp) [3] +demonstrating how to construct a floating point MQA pattern with the same +pattern structure as in the SDPA example but different head number in Key and +Value tensors. In MQA, the head number of Key and Value is always one. + ## References [1] Attention is all you need, https://arxiv.org/abs/1706.03762v7 [2] oneDNN Graph API documentation, https://oneapi-src.github.io/oneDNN/graph_extension.html + +[3] Fast Transformer Decoding: One Write-Head is All You Need, https://arxiv.org/abs/1911.02150 From 16b22c7899304c47144191e6f55dfb6fca339880 Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Thu, 21 Nov 2024 23:14:17 +0800 Subject: [PATCH 4/9] doc: graph: sdpa: fix wordings --- doc/graph/complex_fusion/sdpa.md | 35 ++++++++++++++++---------------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/doc/graph/complex_fusion/sdpa.md b/doc/graph/complex_fusion/sdpa.md index b5a37e383e9..bead0f80974 100644 --- a/doc/graph/complex_fusion/sdpa.md +++ b/doc/graph/complex_fusion/sdpa.md @@ -1,9 +1,9 @@ Scaled Dot-Product Attention (SDPA) {#dev_guide_graph_sdpa} =========================================================== -## Background +## Overview -Scaled Dot-Product Attention (SDPA) was introduced in [1] as the core operation +Scaled Dot-Product Attention (SDPA) is introduced in [1] as the core operation of Transformer block which now becomes the backbone of many language models and generative models (BERT, Stable Diffusion, GPT, etc.). @@ -30,9 +30,9 @@ SDPA graph, getting partition from the graph, and optimizing the kernels underneath. In general, an SDPA pattern is defined as a directional acyclic graph (DAG) using oneDNN Graph API. -### Floating point SDPA +### Floating-point SDPA -oneDNN defines floating point (f32, bf16, or f16) SDPA as follows. The blue +oneDNN defines floating-point (f32, bf16, or f16) SDPA as follows. The blue nodes are required when defining an SDPA pattern while the brown parts are optional. @@ -74,12 +74,12 @@ optional. ![SDPA-Reorder](images/sdpa-reorder.png) -## Data types +## Data Types -oneDNN supports the floating point SDPA pattern with data types f32, bf16, and -f16. oneDNN users can specify the data type via the input and output logical -tensors' data type fields for each operation. oneDNN does not support mixing -different floating data types in a floating point SDPA pattern. +oneDNN supports the floating-point SDPA pattern with data types f32, bf16, and +f16. You can specify the data type via the input and output logical tensors' +data type fields for each operation. oneDNN does not support mixing different +floating data types in a floating-point SDPA pattern. oneDNN supports the quantized SDPA pattern with int8-f32 mixed precision, int8-bf16 mixed precision, and int8-f16 mixed precision data types. @@ -91,14 +91,13 @@ platforms follow the general description in @ref dev_guide_data_types. 1. oneDNN primitive-based SDPA is implemented as the reference implementation on both Intel Architecture Processors and Intel Graphics Products. In this case, - floating point SDPA patterns are usually implemented with f32/bf16/f16 matmul - (with post-ops) and softmax primitives, while quantized SDPA patterns are - implemented with int8 matmul (with post-ops) and f32/bf16/f16 softmax - primitives. The reference implementation requires memory to store the + floating-point SDPA patterns are usually implemented with f32, bf16, or f16 + matmul (with post-ops) and softmax primitives, while quantized SDPA patterns + are implemented with int8 matmul (with post-ops) and f32, bf16, or f16 + softmax primitives. The reference implementation requires memory to store the intermediate results of the dot products between Query and Key which takes - \f$O(S^2)\f$ memory. It may lead to Out-of-Memory when computing long + \f$O(S^2)\f$ memory. It may lead to out-of-memory error when computing long sequence length input on platforms with limited memory. - 2. The SDPA patterns functionally supports all input shapes meeting the shape requirements of each operation in the graph. For example, Add, Multiply, Divide, and Select operations require the input tensors to have the same @@ -114,7 +113,7 @@ platforms follow the general description in @ref dev_guide_data_types. 4. GPU - Optimized implementation is available for 4D Q/K/V tensors with shape defined as (N, H, S, D). - - Optimized implementation is available for floating point SDPA with `f16` + - Optimized implementation is available for floating-point SDPA with `f16` data type and `D <= 256` on Intel Graphics Products with Intel(R) Xe Matrix Extensions (Intel(R) XMX) support. @@ -122,12 +121,12 @@ platforms follow the general description in @ref dev_guide_data_types. oneDNN provides an [SDPA example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/sdpa.cpp) -demonstrating how to construct a typical floating point SDPA pattern with oneDNN +demonstrating how to construct a typical floating-point SDPA pattern with oneDNN Graph API on CPU and GPU with different runtimes. oneDNN also provides a [MQA (Multi-Query Attention) example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/mqa.cpp) [3] -demonstrating how to construct a floating point MQA pattern with the same +demonstrating how to construct a floating-point MQA pattern with the same pattern structure as in the SDPA example but different head number in Key and Value tensors. In MQA, the head number of Key and Value is always one. From 26a2260ec6e3ad502bb5d040e7520c93c58d75c4 Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Wed, 4 Sep 2024 11:41:38 +0800 Subject: [PATCH 5/9] doc: graph: add gqa pattern document --- doc/graph/complex_fusion/gqa.md | 106 ++++++++++++++++++++++++ doc/graph/complex_fusion/images/gqa.png | Bin 0 -> 38288 bytes 2 files changed, 106 insertions(+) create mode 100644 doc/graph/complex_fusion/gqa.md create mode 100644 doc/graph/complex_fusion/images/gqa.png diff --git a/doc/graph/complex_fusion/gqa.md b/doc/graph/complex_fusion/gqa.md new file mode 100644 index 00000000000..ea48cb6c1ea --- /dev/null +++ b/doc/graph/complex_fusion/gqa.md @@ -0,0 +1,106 @@ +Grouped Query Attention (GQA) {#dev_guide_graph_gqa} +==================================================== + +## Overview + +In a typical Scaled Dot-Product Attention (SDPA) [1], the input Query, Key, and +Value tensors have the same head number. It becomes a performance bottleneck to +load the Key and Value tensors in each generation step, especially when the +sentence length gets longer. + +To reduce the memory bandwidth overhead of loading the Key and Value tensors, +Multi-Query Attention (MQA) [2] is created by reducing the head number of Key +and Value tensors to one which means multiple Queries will map to the same +single Key and Value tensor. However, MQA may lead to model quality degradation +and training instability. Therefore, Grouped-Query Attention (GQA) [3], an +interpolation between the typical SDPA and MQA, is proposed with single Key and +Value head per a subgroup of Query heads. The head number of Key and Value +equals to the group number of Query heads. + +The notations used in the document: + +- N: the mini-batch size. +- H_q: the head number of Query. +- H_kv: the head number of Key or Value. +- N_rep: H_q / H_kv, indicates how many Query heads are mapped to one Key head. +- S: the sequence length. +- D: the size of each head. + +## GQA Pattern + +Similar to how SDPA is supported, the GQA pattern is also defined as a +directional acyclic graph (DAG) using oneDNN Graph API. oneDNN extends the +[SDPA pattern](@ref dev_guide_graph_sdpa) to support floating-point (f32, bf16, +and f16) GQA as follows. The blue nodes are required when defining a GQA pattern +while the brown nodes are optional. + +![GQA pattern](images/gqa.png) + +Compared to a typical SDPA pattern, there are a few differences in the GQA +pattern: + +1. The input Query has shape (N, H_q, S, D). It will be reshaped to (N, H_kv, + N_rep, S, D) by splitting H_q dimension into H_kv and N_rep. The reshaping + can be constructed using the [StaticReshape](@ref dev_guide_op_staticreshape) + operation in Graph API. +2. Similarly, the input Key and Value have shape (N, H_kv, S, D). They will be + reshaped to (N, H_kv, 1, S, D) to meet the input shape requirement of + [MatMul](@ref dev_guide_op_matmul) operation. +3. The second MatMul calculates the dot products between the probabilities after + SoftMax and Value nodes and generates output with shape (N, H_kv, N_rep, S, D). +4. Another StaticReshape operation is applied to the output of the second MatMul + to convert the shape into (N, H_q, S, D) by combining H_kv and N_rep + dimensions. +5. The input scale factor and mask in the pattern also need to meet the + operations' shape requirement which can be achieved through StaticReshape + similarly. Besides that, they have the same definition as described in the + typical SDPA pattern. + +## Data Types + +oneDNN supports the floating-point GQA pattern with data types f32, bf16, and +f16. You can specify the data type via the input and output data type fields of +logical tensors for each operation. oneDNN does not support mixing different +floating data types in a floating-point GQA pattern. + +The definition of the data types and support status on different CPU and GPU +platforms follow the general description in @ref dev_guide_data_types. + +## Implementation Limitations + +1. oneDNN primitive-based GQA is implemented as the reference implementation on + both Intel Architecture Processors and Intel Graphics Products. The reference + implementation requires memory to store the intermediate results of the dot + products between Query and Key which takes \f$O(S^2)\f$ memory. It may lead + to Out-of-Memory error when computing long sequence length input on platforms with + limited memory. +2. The GQA patterns functionally support all input shapes meeting the shape + requirements of each operation in the graph. +3. CPU + - Optimized implementation is available for 4D Q/K/V tensors with shape + defined as (N, H_q, S, D) for Query and (N, H_kv, S, D) for Key and Value. + - Optimized implementation is available for OpenMP runtime and Threadpool + runtime on Intel Architecture Processors. + - Specifically for OpenMP runtime, the optimized implementation requires `N * + H_q > 2 * thread number` to get enough parallelism. +4. GPU + - Optimized implementation is available for 4D Q/K/V tensors with shape + defined as (N, H_q, S, D) for Query and (N, H_kv, S, D) for Key and Value. + - Optimized implementation is available for floating-point GQA with `f16` + data type and `D <= 256` on Intel Graphics Products with Intel(R) Xe Matrix + Extensions (Intel(R) XMX) support. + +## Example + +oneDNN provides a [GQA +example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/gqa.cpp) +demonstrating how to construct a floating-point GQA pattern with oneDNN Graph +API on CPU and GPU with different runtimes. + +## References + +[1] Attention is all you need, https://arxiv.org/abs/1706.03762v7 + +[2] Fast Transformer Decoding: One Write-Head is All You Need, https://arxiv.org/abs/1911.02150 + +[3] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, https://arxiv.org/abs/2305.13245 diff --git a/doc/graph/complex_fusion/images/gqa.png b/doc/graph/complex_fusion/images/gqa.png new file mode 100644 index 0000000000000000000000000000000000000000..0871903bcda007797af77ba5e72e4547bccc232c GIT binary patch literal 38288 zcmd>lWmlcc(k&23kl^kT+!HLg1-Ib7u;A`)!6iU~ySpq1?yd>$PLQArcfAkU=j?sX z9q&8t54fL_#elBv>Z)0@tL6&(C@+DGNPq|h1%)gn`9TQ^3Kk6f{{;T!^G`|&~^Cj zy(015)#PyLD6Q4;$aa+bO-B#{D>QWnY!VcvK8&p2e}CAC^Bdc3aMSp|fBqkT0mTTa zLjRA?|Mv#Nr|HJbC`m+dis~7*fg(j+w^>~}Y;8%VB^B$iUS;t=dpSJ(k z`hGSDm2UZq#lQUD{;vnEbnE5(Uwi;8lBEs92-{U(7I zHJ^XhObr+A)t)(Xkof0u-z|hJyq4$vG2FuJ9C3C)w~Oy>Ac6H@9Z#-}sZH!WF4dRfDi*;LMi*OHGFppHwiQ$FJ)d5nt(%iqu{(yF=W%)4YyaCeu&e)c zPDeCsP#I9ElI zF#yX}oxn;gTj@>+#=GS9LAcsbqz}1(apT~@h`a|AiU*1|p+g!Pa+FXLr(@({LIZyn zqGX5*+lGyCCA?r;D{Fo=Psb%C>>@0H1}=jVfpV{P0k3&c6aP-DF?47guoF8z zAJ$(EhWCNV2vcVjodY|zT4)l~QSTRQVi+~uut24kM5%_o_A?23`-bx(tBT!-)Gtn} zXGnzoj;vJo4oB$2bQOhMP9pLS1MKfv;rEM%(j56FpU%I_l$E~4L4H$Bot*IUsSNfC z+l>i?P!`a`4Yg8}muw@GZ2hQ+nD$CPy1S3aAb_U2QF<81WbE~bj+zNn0ey*yOiFQi zshctGWj|a^EoG9_!0h*lg%$iIwvJrWe=Z3K&{CmMsLag2P9{$WLlUmuuiGp<#Kv%7 z!2MEmOI|2JJ$S(b-cm&|w2W3MKHs}&sz!sn%8$?VwbEtDRGw}u0hK$df80i1jco-n zriwmUgFfIYl%$QJ~BZa5hnOkRG{1IZ1RX|YD!Xpd}>;FmOjf9 z#4Im0Oz5f~@3^;KZ=^*c!dnGmRrwu=tj7=IT;51oqiE%zU9&=1dINiTSq*c|Y@X;!)bFx3_1_-NWx1YEvI+z5qlH6V)1Eh;vI9AE z?BtKQC1^$MDb-X3dErV9uSm?*uEHLF4#eBT0uP++ed08&G>FjEPee-Ak^G6HcH;E= zOR>BBD1*AgC7NaPK^XDnegt)N@ZWIQWSnAe(Vzm^e{)UOOpNX+J$T}ldWdMrhGoTw>4t*17rLq9TimUG;*|K63jpD) z^gyAG$jdO5uf1^#OAV`DEbgGj9HUW&bbJidkbmt~2n|#Jgg%reI%MQ82)KQ-hIB&j zF5ibNGUN)qsV!Sv{_UMhy-VEDggyFzU9IFIM;>d^4 zdw<(f{HMNjiU9ecpG-$W^H>K`7&*ceCG8?r(JC1nyi}p8^+f0IN#q^+#qU5XernmGI*2+XSf$SR zL^_lDH_}PT5&<^{UtUet9%40$)lxgkPN?4fO##{{0bapk&U2U2)WG&> zj*_W|RcpZtk{34u8G5kiJe)*=5ea=~zG?jhI$@X4!Ap=9 z$db1;J-g>z*f)^!j-FWPd(mnFdbV8U#~uv>M+mRGk6LNv>1mAj-Qw1oqCue%)5DhV*6OhV{L4yy{3pHD8`8B$v=`my!>#0G(;$p&VcuPlqmYnp^&BMV%-!S5~ zNXr=ooFowquE7sm2KDTFWY?d7lAfjn>qQ4jSJJ;WB#_N|+gth)^&8~1~;?GOOJF*r5zS5JC5mO*=)>Ozl z)~Ac##ojie;DFEBaQQI)hEy`v=Oxiyt#41Q--dIWx->anC8;bm^MG; z8^rxg`s3k_H^3R)10xukg#jkSMP`rv_d08F0N;y8El3oBd$Ybl&1JhPF!XBK9+!vA z?$goAx9|6#xrs!I0tVN}S^df0bgZk!$Z4RxPV6luOQIy9_$wU&Z?367TZnMoTOSb# zw6Zcl?&aR#H1(qddG##w9_(gn#*rp52Uavv`^MlSR+sAvMAx@m^Bq)G!p6AU+(a=bS z^Z`4QCqhmA8={Z#0cz0lL8`Gr%V}qfijQ;VM@B>A3Va+c4u$O~$%uD;eb}!PR_Js9 z9i_K1u>!k|PUzujzJ&bzn;2im0T~Ng$5KvAs5_iz`z-ph6}GthH@%bHWWhUbqYrr~ zA^oDjJqZ;)m$%&>*po_)7R2?zjcm5R^Fjc)!uWc;qk9d4yYrJeTpO@5Ov0+q3lQ<( zDrWi!DryA6&R4*UM_T+E2_^s#{Jwl1f8S1rC(OOlA(JR@5~`y8O8{m)JlyPLg%Y0) zZq>kEWB~^Eb~ifz!ywtSK{S8|CS+ACR=efz@BZq13$2R*95ZneV=BP#gubXS`qk(^ z$Mgg2uZVm)8PfjTWHY)RUr`wHsvp)#3Wyb5d6|!>;k~6<83H7H|J?SA=i5g5+RI>z zu-C*7JV`Q0NQf}UsGivvHi)9PJmw#Vp7@ssU?AuNBw@+r|lCC z`2KZL^FM6SNuk5?gx)#o#;Uo825ModMJxFY2lqQmrDi=Rh0avy?`~Vj{LA$~MY~e& zdGQIR2}s-c!ve~kS3T>hV+!PY|2*#2sGz|;r9C&E-$WQhU7$muW03v1^b=j+?b)C( z{oa)pnR(hd^b@i47q9*Xg3r~)i}f82*bC}wvp$SmBoH=oR$YX@94sT$i;t3H*=$&i zIl78(u>QVwD9kaqXA3|f8I=n4B&JZ+C=H0!#+_7k_Hs6LGxu^s_d_E;0sY3B{L<8* z<@@?`FN5#LX}L_mme$lORi!=pR1ogiV%DD+m1SA> zE@9mCn<(|uVyj@4GN=L@2ste?*zril>EDi%d~T-NT+I%O97*|fuay|EE&LCsknQM) zZNUQn+@~6ofC8HqBp=!lr@mdj=l6R=n=?$d+WWO(zhB7v2X}q&5F*rl{DySj`71{X zm0s2LiwA!AEu96gq#_hbDd~yAj!vH!Nmngi%jYF3@{7)8-izCZdd#HmMoDf!0jgX` z(la|SG(-rA?n5#4*2qtGWfnnH|NpqsBtIK?H!2YRCMwLm~MndOi}4%i|}2> ziU!gw))3rD$5AK+Gr&;4k@&)xwst~qB@tBK6+>j*k?9^kr@phef8pcQe+7hxV39+n z?Ap|un6!+O*JMZNFyu<%BIKXS0xLz`DOZ)I%WCr{VzWOj8WNYPawXr#{&T6=>|z|WH+gC9r z6&mMK)HRRB`duUY1eg8F0&wW={8D|+vS>;qYN->*YKhKv!sJ;czR$$LoEBYLve zDKKm&Q?J4CoAmoojOM4_)dlfv&4vO2-ltrvEq~*TI7B%pE-zO+J(IEMavGK=fqL!= z2v$IuA>l2v{iaUsbEZ)r~GW`u>I-@B~!xfPs}ej@wSiQdh&qI%@yIcgO@MPM%vG2 zGHV54v&~wtRCen}(Vo+fEw`KDlxgv3!J@{T=b!_$WS*X&wMmB9ECm0%*-ZnFQMU{H z_%~ZFTVFUVJ<&4`XNlhMejw-hAnFrDO!;p4RFs)IjpU0t{{}iXMchEmh8fjM7D2!1 z+yydDmF{%GZNvQh*Trwilf0!Ert7CstcJ=vEVU8iLVweP@_!B%z2vTyAUcF#gC9xm zR8f4mKSs5M{c?NANiZWe7RuLMAAJ*_bbLXMrM95TZ{c`hKF-=3xMmhaOo$r4ahZ8K zWnbu4%UMVSF%lhn*X5TZ&3mZl2%CAppa_w04(;kT`i2|6PHxZX; z$7Pk<@un>eoS0@qusW^ax&0D4?a#L0hor0cNGzqYA8Cer`-^OLw*QH;f&j`gH1TZ^ zooNm1>A^}7lZh2fR>{orXiq4)?2z_`NtmTsj|2td#P|g=PLqA84XxeXhB3+%u+dwI z*skwrh#D_>^v6g&Ex9#Quty#(TEypu}gCV-Uug9FNrfIhgJ@oNe;>Hiz4v`vPm6<@(FU1XFO_^(o_E+qj zzw^jua!bf?*YDg@T76|zRllLg#p3dD)o@SyI-uJ8acAQ`odz|tY&p1>pe8ejnTI`2>a^@38c~jKV(hZboJLZz3Q1DJDpORe*pJ=o(|n~(3(dOf4|wEK2`cpn zo{^P-ZrTSh1z%l;!`eneW#^43hMkX6!|69^ZFku$a@&FmB*r2Jt1*skY$_=lgD}|a z6FuV3a?)OQnmHID8BhGAo=9cW>+x({yt}6IFUY@s5y4v}jq`)V{Bu`u=LOsG2*Jve*qGhrj`&-WIy@e4CGervG z_^UeX^;N>NldZ9tp~asZc@w(L`X@uWN;=Vi>nO5Xz#>0P;#I|U`T_q^dx?=b+swe5 z*UJyevVA#L&0IqL8!b9uKZp6vuE(Dl?)vlVFdR__4f|D&wSNbCPMjm0f^>lzX~_vq z+(}teB&BcpH&_F$c9BLI_EK9S4tGdG7#77##u`5s5>7wWp?laE34oTBKAMh{n)9#6 z;Z0G~!@kXqDt7eHoTdRGTNCDgC=|`-Jb&y!ub86~z^XD3f!w+Z?}uViTNO63JZ3~E za^V(z{D3G}vyi_K7J~>$k__s}>q6wU+1VS|70WMDG91h>q#F7vw2cL(g|*wnGTM1wF0HLgO z(U{cf`!_6XlS@YAUk~Rf!_+fEV-L@FfAF|8KZG3YFBE$@HqpC!dwoF*e(Se`)XbzP zjbBiD0@tY@h=x7(n%hr@As-^z^3FjR?RB%TXiF|}1;SgrK1sT}g?!mKm%7B4{^tg7 zvcnMNLW#(Gv`U9Be_7sjTyBfD6>6!=4T3uMQ)hoK#?5fm;evFF#1M=SSD;yzMUR)#(dNW;BCw^mS9^YN~YX`^KdIFw)q)OOIdJtbT zvsQ}kwoQQi$=+qAgKAU>ZV6u%t%{v*wyL|UqCGZPGSuS9CwRzStZOkxhImCUx{Nox zzod(SjwQ*mxbnvypV1*f;tD_bPcQw=D~a?tUcn|kvxR`{@$tGM9Hx!#oLqDaoRE)py;xkOnW74wyCsbIW1D(>ffZeq~iL?^`NM zD+RhKO&}=^NRgzxTF}W&tuYuGG)E2dWu~H@sGMh(Rwezc1A`Vwomm4f8 zNwE61X~A;0njvg=*kBt$Hu*4B`KtXowvFQs&8WbWigoOVhcU1O>m8Va!v%#$IS$PtR)S+<#NNJDl~=w$jR;|_FT1RT-D>qBD*Lte_oEF-xXAtWkWpi&M&yyK~ zJ=@VsCBZip=zOs9be=Y_2LLsCG5>zi%eAq5GrgP3$=*$Cnwi7{q$rs_$rjp_!`hZq zQptIH_s#fK<(qbly9znsQ0giBExD#1&n;v{x#HH_0Umd#QZi ze1@32du&mr(ZFUB9ivYPdEyErQd6AFvdX_)r!rdF_b|YcNlmSuI?hQX?oq=HV_$RW za|cR3l;Zq~^11a%3ZhD+DbamNS9VOf_ke0xT z>_s=YDQUDg1k6ksi^6Ueo=_`mIF+i|u&>dtpwS9=;oX=x1_k1U<9D}T7J!9n(k;5s z%+WGal)iOqQN2H;()k?H@k`X5)zm~4K9z{4`*mIWT5^+LB>Nl$uBDBmWEuAEw(oMH z=r>GVV&B%V=^2ZER=w%ddhua8&^QqGqy>%dN}5UUV`#rAiwaCi)dsi-!G2_ZD}+t0f^b#6LWcD()19qcG>LgtmpA?r)}ryUg04B25E#0#liDIAUm(nD0&@I?Gte*B)p*b~g7 z$keEeb?qHDC>0cpKtV=g#kK|}Imj|Ro&X+?#*Qb(@ z3^Y#H{TgQZv0hoP|N3M6thusH&?%<_Qw^KJ54#tPy=vGbl3i19<=s^Rg*p$q&0~jt z)<&X2#YK`m#ibulPaR{sgl{s{h-uYVq>$;x$zitZqJ)a4qiVUa`!7VB?=Ga^)p_r& z-|L_Z>1|2vrs7ppkT>_J<<1wOC`z~FEF&(qXNHXoi>NBOoT`=TO7>9rM}O;{J?vku zs&sR9Py(8)yz5Glh1LD>%{W`l9_L7M!Ue{KKsPnrYqr$avuVF3hL~

iHLZxw*uT zHWg!&o$a<6n#-+xzd-X1jG{+gcI|7gMKa^f0yVAXm$PS0or4d`V} z*krkFLCjMv?iSYL+qnV57zk0*iT~CDbh-LC`F^N2L9O8{f73j6y@;kT=6iUk3ui15 zEr9u2YKe~9v--N>6;H-Y0)$HFa(#dDaG|w*&CI8htuv-49WpvwedO-j`H3aYDs`4? zbbBt5>GL04+4{*<#!{CUY`>v)ONpj(xikO|G~#gH1AD?aFM1fOQ%ss~&mYR3cK0ce zuqISW?k&Ys#&l(3lHHl$2#&4h>iO<98T^fpWTCBGvhCfv24AIZ)1^C$@}gysZc`dg z_u#yD6sJD1lF}h$(G_=i&eHS_X^s(gG%R2ccskFQE>1WM#^17G@;JL#!3|RB!=ga08QOCic5t4#N5@OW z(Xb=45KBWtLlc+7sSJo1ICiST{-M^$)<;osoCyqhv8eX~pEKX2^SRo|aZ%&H24XY9 z<|n{8Zj+Hn!meOHcV)D;%SL+w^wH>3Ex~gGP*}I#^rYl)7kyGd6C?B6yKqNcxD~JI zL(1f2(!oS90rB8MHiV0M-ETRQ69A(v zZM*$M0R{k^8+e`sLM=wn)TFp9>o+{X83qHb@dTb{r%yS1;(+BTK}@5PJ4qu!~J8F3?!$kwNhX@ z%4)#65eH!|gB2Yx0f7H(qhR zmKN9oITyKl8occ)-Mw%(Y6aU`u?S>3fO$T7o*P`#BhamUA5(6EFG`=+%rQ9I1FL!H0hH>V z;|zH}5t47P)DSO#u(Oi!!tV=ERI;o8im}-{xd5PfovM(FEPv!Ov@V>REFvVI=lb?D zgoVt+zAJWb8Ke<8me{)E{a5^iLbY>w1eO-7&pbS^Kw+$w= zH;2!Sst{XA+A_z_bBE5x^&a)gIH*p>%mSp5E4}b_f|&}9>coGRKDO}iUeWl=v4u;e zDD$9}uKq91;zC!KL{OmT=6Sl_dh2sfA2|87TuX`(GkVNRgl@;J_+O*MG=^2n`$IN= zd16S%O91bWC;m8cs=2ii@7MJCZA;b4GXVav`MuRGD4K9T((e$$4b=ifKMqfOkLPo; zU-(QOXS&6^MbDDfHLjAg-Y6SjzAtQG3>S9K};UnTaNV1W1OwF_)zhZjcIMeAp za4KHydyJ!Jr(Iphb1iZbxxtH`_AnZIx zkl9+#ODVQLXS{t!`DX{W-*Nv0j};3E`r4Ivv>nIAkU;YjQ$q0OJIAIN!*zSvARR*~ zjxYT}2F@Y%Z}}a`0wJ8{xR&1wl0bHaUwECBCfE*Rx>ocgA+11L3Y*5Fdp(|)T|q6x zR^$&ZWG=JcJH=a@K<+@3HII+GAR%dUT$qKbM`O6WUvGW{J8uBcjd~G$UgFAYe3C_8 zO~7Y;&MaQ}u=^w7+=hiweCG$~ysfRO6(~O+{8kD1yQZfY?)DxNzqLJaBiP$wkBY$< z#_P`AsYYmGuxJ$|9omP~VMY@Prw{@hp5{1w4Ji3T~Y3=#P zXKm@Www{yEq-!1W`mTv};(B$hYW~KE^6Hw}sd&T!!QF6G|WEBVE<+y9 zj5=eDcVrQDj~Ou0w^zOEzsC{y%YLr=hje@{Yx3jvupsZ3)K}ej^7WlRW5t$qRS5L0Po7p>ug0^Q~`4JMc|cCoF?AmEEJ z#Fg3m9W~vrw(B^Re0c|MXZTH16BQg{zH=fF@yS@Rqw1k<*k4#U2w>rkl7=Y~$ZPq< zykxCcRl|*A5hKZUtj*fp%rm0kx)Bxp&VYwdPRu%>k(>v?TJ&E#TIpF4A z=F@?aF{jLS(N6WAtrc7&QXOi>XZlq1Gryh6E?}y1lHo*p(nFIv->S>?U60uat!fS) zD+oSqa6R6&ZmoGn8P^qu!pd=w~p=b%$ZPOEJ zb>(%yubH?#>;8hfFWYQ!z%YWZI@w|V1vS+FK+U%4AE;?-c*-B)%rwEB%ZH^WxA~cp zcAF7vd1`gWA)?=Qt3By4ZxCI|1j^4#%}ngz$Kuq;+>Wn|AW5Fhx5+*P+7K9Eia<0> zs!!@6ZdC2nd9|${h%=2=UWXlGaSmjtf3ezq790F~;9l}oVYu)N)Bla2*GSo6^Gh;jZdkr4`zN*&eEP9|o-<$gun_fw{S+Q~Gg}B=GDpq?4 ze31NrwUEx0=RtDycn@b4znQ#x{B5j}CMSi4%yl%Ac4)AqKKBwA1WgWm3BH-xcbxTn zhf2TmTtAQX-w|e)Jm+1aI4caQeCOplG-E}HCyYRqsf3w6ieG}?9LR!pUqWgM6KA_8 zFh1c4eqaSETq|*_W+2L9Qh3gKkyyV*?T>7*$sO6xr?!`Y{3MpG*wc9~tK&z)V8QOQ z+>>e(Tte9Fk}Jq_}C>)W2i8sf=bQ%Zj_Xiu9=7V&Yo1ISJ#}=J?=SEE&T|(3sS$o1hccjXN315mJDvN;gF+OAWaA6tVoCKo zYJWpfn_mS8_FoEjsqbySV3ZVXLYAYpuHK)1aUpp3a(nsS?&6#=l?OwD&DE1b*J$Mt zalBm1=q?=Y`eGULkV4lxs)j0cy4_rsowF6|Pds{5VfDZNr?O;b{=2f6rnUN_1zs0~ z7h=OKlG*dPi;tU%gpMc2DhqZQnzJUDeC%?fH^!J_^0PUYsCDC@**H=fa-FgXz_8VL zEU#n%|i98Qau0Ox0TT@xKdN0Kn znDt(iH+XA)$0_h8M8`D^=St3bA2_D2PY5PO#QynTvXz#{%sN6_w_6@embRu8R6Qn} z$0l0e@K=YC?sG^H<2l~AR4B>54>*)swase|X2ED0Kcrc0*BcL=94B}fLE|VtZy6tJ z1l3Q%`ASzveWO0x4ZZ;Tw|q|v>Mb>@>s*f)Xs7WYjt9aGn8_M*1Fwt}RFW^QuwLe) zgN{Epf?yFuq$^A4jg{8b)U#=+yq{DMPNb*pF}|xoXT3_n2Fg=>-xZs(e)SVHSq3bl zObXqESPO-1sgJ7l8Ha!8v$*u6XzJUdsTZx^SJnP^@z-!pSTU}~1-0p4`AnZ!$$KSG zY#C3IjUD#Jns#Szmyq!cohlbkX5D;L);e3bTBE3jYhAsVX#Slvf z;*12um-!`7<)MpDO0FI3&-#lV=tvsaK!qLppk|C!e$PaUiLG!;yO|HQ=!iJ0?_J5@9lj*|E0O0f^Z7BhfG zfha~%^=Zh6V+&PXj!DQLWsB`oT$2Sj{iXJLkI_1prqoum5OaZ%T{_tUuB0j%C zUJuuTadIf$K3_|k_i95_8tCT_rjLAHFi~Q)KW>{JJh+R+*5HCx@83a?4^^IPq)b5J z3=8r+=>Bs&FESc*=Hgkdl8klY_)5k6qDb&3Es8JAZ`I~3^gx;Qvmw6&X;Rj65ZUUY zAZqqYTY~hpF4k96YJnlz&l*{maCH~Fh2UTfBx^O%&&J>FX#!&^thHNRX*t<*D#b1X zFMlue7lHcKU4s)PLh2dCKK=Sdh_ScC9tr3f@HVD;)5*K3&B=quCrm79@|{%QKiG>O zR{O&Sq8WGT&zaUQBnYYe+}b>_`3$IVPq&cnOE=PZD4?1skc;b0AL9Dx zKYEyHm*&Q?`LrV906vQ*T-mE9>5c*?PF*k2fu^0A@b}yL7oG%=nqJbEu6hvhwmqUf z6bcr$^bbXxR22)~Bc`cS_)aPPg`}(YH4T%&nOLueUB+C)KcB9^Q>RB}Kz$1^Ifc$j6qm3@- zUBiuYxOy#W+}5_i_{9rW`$zB|k!JhTXK-c%zV#?^B=l8VUI|N@!`mUG=-XbCKg9dm zc!6-K_)pz=sZg#*@V)gKj=`0H?D^G7&s?#t>XI^kprFGDx`}1J8itiu1=LEr7A6;M z4RBg56bc2MsBM}MxQ#SE4{p7?)$Pjdeua25p>^mmL_PjX*Exub&f}bC$#5SW_|0yl zfSLDi?me-m#X{Tun2=QwhtacmiJ$kDqhNB-!~H^lG<@!NNVWSMtL@9+)?@$E9eM#T zDAE-tbL2ZtoRXbF{rm!zoprEijwosFSXsHfbkB=SBX!TA$j7sGx zk&6_$h<{Hv^0@GUn;$&DELrw*<-3|SEbDc4a;4QZeqS}SilgXyeATGf&?5MJz<{y! zj6RKXjecubBFJhdywD`qeNmovkT+1_h`6+8@G#fr!|qp6dfWKPvl24P_U}~mzTQ>v zmXBvD>1Rh+;8V7%wbVizRZjUj@t_|1p&w$Ov8}OKpBB!9GIR>s5@{6^TW;=E|MIxy zZ^9YyACCtQm$qvGeS!$aD?<2Zj~lIHLm2TW$QFzX^0n>8kEm?B83it2mdP?-($)2}9N+Oy~;9*?C?D&6~tw z6A6!&+KS&Hu}G-=pbQijFXb6Y)2uKESZvpUJ#ER!5oydL&Q`1%DvfBV`yIKfYBV>! zt{VSQ?3MDZPNeXYhwk*OH|fR)&&?FmJB{J1H5@lZj6436 z+qta|n^qRhnaCg&=xv%GEgcgoblWNu;$EWd$C+2r?WF^GpKUT#TII!I3oV4CPH~&J zV2SgdHwNvUMIYe>Gh}9zy2w*jB}Cno{>7wbfySZU9~O$7s_t8B=y`&xdB}wZ>ts6$ z?bo%g3YB$62gp>OwTkM>MOOfqUH9k36wdiP_ImE2n+9NMO5~p@$1 z>YT-7D+{ytxvr{0P#QL|>WZ2Qt*zSjk3M>XG8etW<6~!H;FS(k=|Z=QyPRZc{m4+; zY%5w$IylmMZ`O*(Xd_3`6wx6C=SS=4;&wWf%44CIDfRioK2R87t1aUl!3TFdPd#`8 zRuu(h;W~WMRe|&OC*kx~cU}TBZC6J;g;Rj%PL408c`f80k6p1(o3=KtDt|rTH<*mj z2a5F@&JN}8xDCQ@?Q;1}zzyY&>52V$6}6kR89eMg_bbfjT5e0R7bDsS9NPWKvOGOe zps)b850$Pk2>Urh=uMdVOX$cwG~L!O^IXaK0}QTi_%#v8!mB-Y7e`&vylP2=6Jg>sE_%bda++{e@W||zuU9N#~biBGQ^;ndg{d!ZEurgcUSBXopcXV?u@pR9p((l?iIy&0cP(O8roUz3H5*SAS0~#XN)#By4 zKk~M(90CQeSI!xqe)zwO)3V$=t2mmX;ae$H#ru|ve9Z>NxI^S`PUO7K`!e?N>Z)B_ z|KlANAU%fS$^3=@%MUHc(k3rZvtvzSTGn^bM+)a=BHDKAxv~a(uHZsBTTvrZ z3c);ixN008=b|HWe4_6vgNPkFKRK;<&lZqo@XW>nW2yMU_Y7TlV7~{pLP9s+t(w2tb`YMt^dq#cPUBsncM5IjR&P_Y8 z5HhnoqUbD{m&BF5YLBp(&Rr}?%TReB>SghYDsFrvNKeqm@r%K%{G9wYYDAc_vzbJ< z!jUuO??aA9&hwVrg1x$tbjXsy+UUQ?8ZeIyipfZKsW)12c?!cuEdYj)w5zHXtN!%o zZ2MQ!^s@1=iu5%bTk#5`iiAip!$b>J_~T5mQ>&kh&S)|Udt`vwWX;m86pwg&qsYAe zNZl8WlaCAb$_FWx>3qc^Z;Vukd7fG1*PSKQH9ab!xilW(AUrgR%F-vxQKH1_zM&duYR-pTx~{9;);$MRriv| zHmTycCHSlRZy`a=;R*|{!pv}Po z{9IdB!;`7z-l^#2aD_*eq=*m^u4n+F)EG0xN+<}yFndc8bIJAVVcMe2NAH^2#k>AB z@Elc=LkP-IIxMoptx)%k-Nug3W?>8@pZfK-1Or_Ew)lqfs6Eno)!;b5mh8K`2r{A0 z!*8Rx#?y_xN@3HYH9t?#hsCqtqNdCzOAM5U+Tvht;444UP^8mc<1`3H44I9&Jg;!P zIeliL;iD}J0vE2pyQMu$zBsLm5rVy;ukP5pnU)Q)SN==)XmmAkCOG8s<3*`^Lok!7 z;DOh7FznMxQnY&j#(i7c%_Ot@lxvpW%3gE(^(06q`S>ixb7ocr(!1;#;jTo_4!v}| zY~Yow=-UGmmNx4WF059C?49_jSSpJ&dZU}2@B1OV53h>-L>NHO>NdIe#PMJk`h)19 zBVo%%klwXkb^z0|)*}fr%lwwq@h%bzE~L;_r2$`KIh*Dd@(qKN67s1AWK6R?SQud3 z$}+hLS~B-3IJJ6KW~9J1g>mpH4B*$k6?LK5W+y{8&QvFb#{F#Q(|pkGcaL3)XB8XYbYoqm_ieoOOUruv1!qbU zy-mr{%bc&;2Ie1|5pkUgV?w9PAiD$eZRk{uD8FjkM64wTa?qG(MKZK8aqmldqU9n? z={rSHsn!fX#LQ(nR$w$=sx{8ae?rCTN*ODrk#<0(MP(d{-m)e~=inj7Z)~n3F+IbI znggEqL@O7s1n#zBZ532)frj++{VG&Qc4;CQyjL+1J?f3Z?L@IBzdj$LG!t2SzZBE8r zgdn6hKDpFwYvdVMIHuNHEKOg>v^F@}CaUR)9@mfu5NM+~rKt`*$t=vT)7_57cQ2Va{H8(j{5!uxu`u)-(1L%SOvXtoF zAxkG8#vRa>?)iw0Uigv}tBfwh5d zA~(Y~5K_!1z5;W@U|bgiG$klx)F#;E*GrW-N*9vyp4|OtLtXjC@J+W3U}aF#gCrF* zhuCKyB@J`B-g3d;UDVkh+SOUC!eo@+#6u;j+jPsMBC>9?JZ*)(1{CjGaqA^H>5-2(h&lP`QT16f;mE3r}H7h=$M zH8yFj>`wB--nZ9wm-^#qX_*y`!FcD>A?l{^ia70b+MWSw?AyzawQgJTPe;gBK|hq< zeATq-ZImHG-3P=crS?p7s*#?)gO)k#s=sw(>I%u&;psT&YH)SL7+XWtJMS;@VIiCN zI2h>pJ+Fp-Y#}Ljv2bI z{q2c~dE-fFtLR2UqA#&#k3m_arjJCnDlCz;KlZf|kCk1xe|j4n&Z zedOr+x=E?l#WK>=%~t9toBF?Wp8F0`@%7WuEq0=C?`ZJHBE0{9p56NH)@-Jq_fsE z{4nIdw6owaAFRT&*h;Z372U0fNTsFg6FA$(z?X8Nf5dJMj8FpzrL*g!gk3YhN2f2QD&Y1bV#RBbeasM67kM#r z+fjN&ZPQuo*XMcPvTpV0l^u5)WV38nPwtk#1kK0|-)~cEBF8E{R8H-?u-u{Zo>T+- zXGm37;5Q9~jBiW$yfRgTsQ}Q35!%+81;nBNNl|l7V+mZTu-)8seirT~NNT$uh<#}x zXkxA`A~m1UP1j*c8E+g8QcHWPdTOcIA@Xo@=beA(&AeQp`ORR3VdwKa&B1Y z9lSxpY($T~UEc(p{jjgvLO(E+)63M6P@*Q#?8jG!C#)t!AV6>-IphX>#VnH~+uQ>8 zLu-6vRY29lH*;tboJBoRYBDs=d{@sZD-Es{y>;wi^l}+jr|9l@>o=-yGDVYOcpZ*} zXXC4Q{yt&9=*tT4C?ZnS5Y=7VLenpVaZ-ZJNy5|*%WR!800r`tj6rGStq3&mJy&gVB5z!TpI0XJh<81Cim4c_V&(n<3BBIopQBxcb4h2 z2eN7P?zV?vLGjxMn()A#2md!%DLof+tl2IP%~{Sp2$*%sx`o@+K@>{YH+Bttj%TavU__50IlHvxwSUi$Ot;O199==(mb?Fq6 z(PB8x$eZqB;W8GgZn>5N8&jE%Z*zy(B3e2nQ?mb4wS1u&Leo#Y{s500EkJG8hY#aR z9V%dLD5iNPspvx{t6?M>$ieerE*dY8NeS~3UT#2yJSE7oi-bC`RRg}XTrxIWOeDd) zi=+erpCXWP_5;=f*9Uk{v1!wnrOka=Bp9}U;4P!`O(Us%2ih13Wlwpr2QWBrZ4^Ekp6UEJ7xAGM>a`LFh_4SyD zJQTXodadDINo0w_gRy}s71qQ1LW>?f!l7JcE`P{!iy*@%>@AFANeddYri)PRS~cq2L8Rb;7~7>lp%g=^qU46axvD&qjt891u~X|yzUN$=jAr9#i* zPsA(@jEX`x(_|dD59s|L_TD=h&bD0}k0cTiHBl0xgotR7MD!XZI#EX*EkZDQ8Er^J zPY_X}2T=#3jy8xcx-f)MqBDAD7`|Jc_j#Uozw2H5+xy%9{MK*nf3OyFU)On^=Xsp% zxNiG1y>)@xI_P~={)V?n8kE08i^QvvSTvpX6+q|h2(W34x=hUpEgz5Iqt*Xa4yr*= zf#$26)tEh>X;^P}hZ~yv4}7d0*6%0_cV0)w7Ef@12d=2bWMTt3)wzkO$4#sY)!1e@ z!5=Bmr+AXs*|2w&a`7Ul2H-XkRisX!j+TVfi1dM-6`70)eG22cUp*>&PJdxb`sT)k zXr$!BI12g;$2av~LmHe8#DLL;a@Y!R$!n!$sqvKcO{yFHzsb~h{z0Y&g@b@N zAvr80p7!E&LIxGj*OtejxdI4EZCzZ&F_cV7zl$wmKPm zJShtNIN-4#OxDLaQrvhQBWo&q{9qnG^!g2{zWgM2S17wr;R+v;S_C=EZ*tGiWo&;-Wc^&5sWAVuu8ArzSu;uL z#YW_frQ8TUA+B`4?3&Y8)8@Lc z$)O-@b>Yb{EO@&q6Rq=r>XD=^@AkeOqNLUgJBp`tu!~G)NQx?hSop?=;fh%A4}`Sz ziUYl`@+%`=qD48<-&%>7#->6a#=864x#1G9j%^N-Ur@h(o}kff48wW8S6QBl-5525 z$t@2Up-I{1m`)HQ8lNp1^GD9qQ_5z)QGS`j0)3pHJF__*lcmFj9GUgNSY0+Sas4#? znW$#18>2ec@&x#VFJR7SCwVfWXa&~H&IJ8sWE z>V)dP+!<1zA}<^2oh+XsB+Xn|_i*-R6YA_J%X%u~f2+(Q1aO=E^MKHB#2 zHOuSnG_#XvRoBYK(S_PB2;0V;`gCP8W0ULZVu&K>6NXS@Z@9f(EH%{cWE4A<{_*kW z&C|zsS0W?N!lx$vvL+qOrs&t@N4VG@IGJvdUV7Fv$VT4nUx>PsHYo_MNJBwc-!=N1 zOhPYNZ zjmc;DWzAlNQ}P{gavz&ma@R=|IeYt^c4#C8(|+mU^TYmAHP?aO>Mygk^@iE+vOZCJ z=&2j*(w9fQIKi#+{j4{CgDpDXugq-!?8m%vU=pK;-ioGdPg(diymvS=y|R+Rbff-s zf8!`0j9X&d8%7|SmgC>I+-_!P^%&7TIT@4L(~+EZ0!pohRwA$4D(8&--N0dZ!XJ$N zR>0qFV(4L|mU!gL3(i}reaO$(R9Naf1w%Eq!JosA#Li!ST^$G;Hro7$4{&K8RX zm7d8Obw--Kib}6kR?;PvfNswFJs<=^c-68`#O6}n_$|Gi7AK=&id-a0PVcUMil2}0 zS-IA5u$!!Ukj)!>|Bd@Iw*_}+AO#M(5fd4<{Yyha%qrc><19KyEE^y?ef63%R=UU? z`x<`7Ee*Iy`pL*$@GI;=4mF!XPKw$hiA@#?Vbmuhisu>EF=7y?0BkIFQS2~ z(Rz=c=8d_;T=p8f958v+THd*{Z@$}8?m+LKX#WGg9s`!wlvZ76Zj?MP5Xwcp#%o-J z^qp4ScR2q2DuwaXYft4aez*tbzsw{4D`h~81Wq287l?FEzoXZ-L@^hKp-YAbxVxpl zLjwJQ+?_k5w<`r9yY*H4+9;O$=d*WW@>etE1fXf4yI0UQ^kR!MZ$-cA

    iaK-Lp zaUDMyM%Z;T4JSR(75zInTdg{+0gA2s3Bvfyv)#q0wCbxLdkpS!-&YG&aE08RrtB;W zdPl-}PbEs~nJCG9_vH`blDv#J(VlUz=v$>a_Pk8#FRdZCw$)HP8Vv$yLhsY? z7JKQPKUlgO_D?>g4l@k9g@P=m@H8KLZf*|3$!9V|(=-=_O8wZKkq)w&l@iS4k2o88 zJg@n2C;9$(d}x3Y*~mVV$x%i;-7Q4fmWkw|x3LF4>v-VQz57!Cv9x)vWZLZ+$YP zY>VygFc{2zaZxmlJh3Sqv9>UJzAHa0*65 zdcMLqNz2Jdvb@$L?VV#q^HfmtJY)ER@6Bd3KQNL&rA1&VvRb$nue_kDex^A3-LTO8 zE@k*TUnMe=+O!~}o$q5yr>Bc6*WEKE0XmU?iw1Kc;F}s<6)HnF^c!yXg$1=D`4?k?N~gZ(iO+}VX_K_|;GR`{#~jecP5JUa^!&wt&wV*?V#+t8Ey~&5k9zp<`|jg> z)$^f#SM1FfacRL>9%SuH2_{2TWW~0GArx8=!O*FvZV@X+ExiM2j1n#XgAd{xb(pD=5eC|ID6G*ZY7PGrJijc zaY_fDgx;cb>Uj)|%&jN)UENSZ2N&x7rtRm}PqhN?4DX{@5N+WbUNjm!H_a+_XFBQ` z4zVRkQg83X#*{s5I;xZi1BNxZ4>(#HA@s%e!!FhylUYl`>oo7)$xADV0%tw@X*pB8 z^rn?10EVfpAy){jnO>e70wv>2hhwH?zUZ4sG5%~A!yuA2K{d^X# zn*Q4|E&)9f7LAu>aoG8z@ZjuU*fXC@+0UJ&=G4+lREQsOcDgQGYg&~Y(|OHupqY%? z(?W3ZA~52XvBte_yVThg2Hwt#y0vu_@X2vr%LQm2i5wS*fA8&Y9!mw?Ts+GMV>yOa zR@m)zFA|1Av}x3;Z@I&%Z!Q$Qov?2{p6%5Iivc8|d{Of@&6XTj{|Xcpz}A3M3RsUG z^5G^9dC`kA^jFh&JMO;sJXV&iw8=!3cNli@+W{|9Ksc%5)Y0bQD-}Y`SAtKb3>DVh zP**9h*@@PJXOJHNuIg{aTIUGHST>*ExX=ppiB_oIP7u|0D{aT=nK)u*-GVM?dIQsP zD(mlE4|gilVhT0$xOsvsXE|35_XwX1`9A-WLBBu+S>{x73Rn%o#FFd z3_K#P&*Hg?G*U$ycoRbbUum6J@7CN?L>V~mSXR9%Vhz3(%0|D?^R1Xx;)O6$jP&|B z;Q>JQQ**6|at#L`1M!0#a=JkGMHAvq_5IUBm!U zd$8suv8E+yp)`dDq??L2Yfd|lvH9_F4+Q*OW% zgoq<;b!AY_dwNHQ5oT4}%hg}R6zOSE5j{-x0VNxS+MY!cbsKbZ>yuv6Ti74+Q#vdO zTDS62it(LRlk(Czj2)O-1`LQ>d)l~&B@7NlJ8df5b6?LAEAP+>?<}UJ$&@>OsD#2z zXL&zM1!)hddHjgz&((|d>*7eQww=onJ-fo4Gk;Irv;Bb3@2c zNa2y~Kz-R*XutjgLHba*Yw~5K?T8HNX&BX*f~eDvJGDWintB|bCIb~Qw0AW2I2vfE zm88{c4WlR4O?F7@{a`h?WU(HAx;h$EKXUe|mCK`n%Bauk@;M6?I$xHzuP6`P<7w6mK)W@!^Ph8Hy+)AF^Q z&if)%zW^7cT=~CynT*qMf@2pJ^J#6AoYkgXW4wdKo?&-U`P&lMr!Pr*YmcTyny-lO zp>VjZq>mNfr&FFDHBZHk3}%y0>MnXrG)6_3@L9#;d;lg$#O5Q8#{dr`#6DrJ!$RAU z>N=uUZ_kaT5583f({B1axRZy3&a2?mW@LJiq}IN#OxRMRLNeNxN^M!zh~ zwd-IH2`eSjTgYx8hEY5B9;WUW;7}R4UxLIj1+g|(&po>G&e^H)R5W^^@1sacE=DD- z9q%W@09Wh)>nJv2q_36)EID}fgLYGfOS|%Rr;TshD6@Lu#FSr#+7t@HP6As}m921` zgR?k@YBfCT9JW?7R-UY1vk_DSx;-2jntFFVuhqmcnE2&x=X``#?_eS~yOJUqJ-#Ow zPfR#m9@$|d02+z_pUoLrw-LV9>K!y503-T+>j|J+G0BW%^Q3WPt>Kp5U8ba*;(q#+ zlh{D#Er=;CtagWhKvcXXKIRMenYHCsx{M}Kv)>voxl4JSud?ZwKB^!$#HBIDjPqeKEP=f(s4cH2FD<*VaUPD2SVTKa`L1J*_Az$=n|2nj5c-`GAw}p`SSQPPSnSDMS-tr1DvTocrJ3W`9$@XoTQfA5_bv2etO5P^h3@3 zbKVqNXLxY*T&qcm2zdBA$1h2?8EIOh(RUaCXbOrJB>v<^m&=?)u_HO5J5aqnI?nw#hqtB2&Mm|hh zBU<@%Bm7K4&vVp`?g?FwN)D^l@dm)LDgY>dYokZj=q;5KKW`64b{)4x-xCkG*Ob1O z$rlZRfVW|f$>`#%=(Ss{pK`W5R)2E;K?a}1E6*D{Vv60l8jpL0ETSp3X`EsQ;S0v7H}cwl=ubPm))=N7I4K~CKjtpqfZxZ%K9rKP+)(+n+U~A&aywbhPTe%ek>ccP5<4Aq5 z#P*TbiyY}rk&VAFmjZ%OSDSgV&E#2&Js-FIHS25f+c z0)B90+*t}5YXauEzHUgHNb68kV;FoLB)*>w)zY-wi}RvElJ)C0ej7A7 zL%}BTAaLF>`tGEqOAwhpIU|bE&J_wpPgbc8(WpSC4_2CI(EIJv^Fz7^obu}u50$T9iraBdJR+a?6=Cq@3hqKxkCXekG3VW^9=@*piKC0KHYZ`NgjA=D^ zs;vAnI~i+$%cObIpy*C#aXkSC5)6s-`R{q$mmJ+I*mNL!TEYA9$65~8 zwOw>l2C^PfDgb9>ze=X3Wx(SZZZZk-Q*{NdVWPw9=cycyr*Ls zY#3*`M&65WB=*hq>XP@nS<#B^ls6eMOn3`__O?;|5lRytt5&@n(NBLlz>q|S3$b5h zdaYhVmDK1_xyG4E-S^GpjWg+B5bxBPpDQy3pvFQeO)H%|WkNB(>Xlb@s=QU4ub?T| z)L%$R>MLT-t=BY_dMu;pP%fK@v#x}X?dbpv4&G26??{qg->(VXS-F=yBiKTUUCG;KKM30XpM+kH` zH8CV<_+b`bQ)pHEvYG6?9fw1{m0yqfRTBm3S^Hq#TSR0P(VerZ16&zHoXC;_m#|LD z)0D`-CU$B8Pt6~$_)Fr((TTB{E7yvOH9zR2`J{H&u6wig)@U|>f;Y}iy-f!23GSUf zzWe%*K5I8Xw*ot}Imk9@6CjFF*< zu&-M40KOL3fFDNVxF5Qy#fu24ihF!r%ph&&^R1_N{_a>%Qs5xr6yN3(+9O6(Nxh(4 zr8yo?bg`{(QHc2lxZ1x6N&ISVktgwmaoxsK9a?Nw*x|L+$(o8BM4bh-A(5mR2@zsO zP+0!;=x;Z?I?V!)Wyh|0P4s-oAgd=^)VvCU>#$)-A?~JAagTHaZh`Y2Ju7^r(HJ1Y z?0q;~GB8HO7*aGDbC`67&vLAx-18nHW^!}%-pTfmd>o~DSGSR>0=>$x{Ca}gDGl(K z4DqwKrqg=xLMT>q+O8^|j>*lmc5HoQd&~Rk`%K~#?O$J!Y3sN&!CiV0a`!yQTeh+Y zTI$dFWrNy}L{BsgmBS_ZK|{kM(`Jb^@)fa$nt6TWn()dtmjnqDBGxv(`~fZt+yVU# zo0o-TYGMNKtVZVhyx+(1jS?&@XHxSj$TF+W@QJq?ogrb0d7>5=a(Dd5{zD(qw@kRj zH^%8M>WNipSje)8eF-u7I9<&2H2U-m%2fDae1=$K@mk)dlrh$a)9tf+sZn+?WvIXz z*`8;=KzjC{TmXx6e|uzmCGlJ-X7#f7zWGVYx5Tar(w`mC1{1KLoWo$vV@z{Yw3E>$ z8VT5P&dwyZez1vPe+B%AtlFWm{$^$m)_llIt3G3Y0p_<5z%xP<;S;D+ELZQ7`ARZ1 zO!drJxqMTqfl!TG-(yzVH+4WS-b*>zz%2`TUEs8jemfWwgmh)=vh=$3(Ryd&MVerC zA8iv+*{d(hQx4s<3(B$=PYzsUjlRRmw$l8ZZZS7;1utO>;Gu)rJt}=YZhALcRc|hB zti~XD45PPK;oCYD;&DI0hA59+@~=})QX#GJ2#rBBU@tF;$cNnX@s&(;{E8VrZx1$y zy!`B^aR)pC7dW@hDv7xW&Zp;~Gh+=k=I!=zN{Jz!{T(ajMs{(s%l!D&$H||6X7=Tognno1LRq-xo1l$BogAePDg1@36?`k{^$gbfK@ES zRrclzY_OSVAx(eh`|gqSEwf>t$LGs6`GhL?pTQzRxJDfWPIq|8F3ky{@{KgS!ytyK zsYCUZm?Du38S&l`1*wXTMiHl7oxa&EDznwC-*Q zE?nPXV<9-5xJ3ynow7SwG`Vx7!wITWUphRDN3QYF*Ne?p>)TE$V+NCUXen2iT%nfw z@k6}%J|bj&G)ec5D8fFtBpW=&%Vh4RL;SX&VmZ%)+izfym*Ut`efKaDC3Wh8D16)e zJJ5gzlX_uxpH@@A)_a?on8)JpO-~c}<5kjl-%-Bmp&e|P`pnxMA9DH7!{VQ`?$HeA z(CLdG89q~Mk&s!`kMo_5kkcA&pRP6^xyK}3QWhbjob6|wyIE!C6g%6{o;eVQD6pk0 zVP9_ODnNxbuU>CZJQx4jp+!Q7Mp;*v90?e}VleXZ zP#)+QdHu?iLri6N{)$IrP%_8@SGD4vc8+g_mAX#R(mXdW-K$&X+Has8=*u-v zV=GD99GgCras1Y?&{ctFVF`Z-sqlXhGHDuJzi#-3kB&xlf+Bb^)c2DQQR}2w5KrQ1 zI%UPCjoexf{xtQfVNuGcFsI;-L{ORIYPL#R&bzA?k!CDYW?r`%Pxx%zeoYIqwauwMN+-7Co;JVJRx|%`G6YWco{Vfw0fDU&9>j^6*;FZkM3blg^lGo-AC!EW zK@I-!y&J{!=|P;}X9@LdqAYKTAcC>0aWakjK4CLW%JtrACi_Uqd)&v$m=i0 z`9ViBH}0Wt-96730Xq70EF^7obDpCwws0Jl#)kdnz8Cal1;-H1wGr9u(pse01pleu zp3Vk{1H`FE93*x>ey{ZFt5b>c@0Dn;VgmF3Sc#-GT<4FKoQjQe!;WSyvgq7m{k>Am z$~3xwet{xV4YoWg&_7rD8?&2j z^Xs3_m2Sl77>xd{kUM0>U9GBTFpK&sv0r3xi+Z8UU#m2JM;PmBMsS{0uAaUihNkw_ z#?Fv{U132SraQ;K)4VuO_opo5zN-y)5vBQU-MZU8ts^D^(N1~?dQKXQNx6%nGafQP))zUp?S>6?t5kl^ zw~sKElIFwcnxNXwP|s&4J-JueUTIfH((mF!(4TjIGJ1ByJ1;sxw*T9RRN39A$Fvj?8=Y`fn~?41JWAB566t zPXU&*7bv|v9jo;$v+E|wE z0WQy3PED=%5??lm&qRgctJD5Uv#KGcwx*`&cv-;guf-|rr`{(-hi(GM^7vH^(N1e} z$rtus!W-riqJG^Wwt(GywrjZf4x~o-;h{Qo?|ZYwL>02kap9w|9mD%6s$u0YL0G@# zR4ub0td0mVpM_5CP2eN%GFvf)f~TN%$TSkj_vuSwY>Vhq#67#AL#r)RksgynyY)a)#VVMb>&k zw!>e|DY%ER*;ei=SVW$EaZ^`0=t!Vu?c*(#`&uZ;j96X$H2G}r)6UB0@2@rE)?yDZ zS6&+C)b_2QJgyE)LER|O^r`jynO}Y-eIgST z$F7tM$|vpqj8*JWFLp-j+R5Peo8ZNt$@a+)O-6wL8)O_la%0p{BkxXJ{F99LVL|J8 z%TcwqV|yDTKqKGtHDmQ@tF9*kV%m-4l@v3q7vTcg82FT~&I|aSA>mwXtv8MqL1S|z zk+tLo!Nh~7_!z~^$(zM;_MUF5B5kW*Lr|_xc`7}sAKfrsbM(CBL0b2U4k*A3$x~%^ z_-s_zHmlGv=l-DfTHr>m!{GzRKEgqZDI742+A5AwrM7I9cicxGo~UqU^*+p>Dp|?& z79@Xirb2g1<~TZJCvcoD^B(?MQ zS6W4uLq=!HE)9on4~9?P_ERn4@x)UJ4{z%sRN+NRTn|hMbnW&gY$1KAWTT}O!&X;U zZ~E03fA(7+V2tKx!}UJ#%l!*5rU>6s(=#uI{^OG@_3yZkBlPYjq!Qd(my;gzWjO4!%X&@b=QV;gPYq zY~drE2X<_NsUKd#IQElG64!mX8fp;tG6(u~f|i2%iSdCC+-Et&PG5lCq*Rle;R$4n zkJ+kdiUaYGJ9b<HN($!H9lF6Ta4-y+nETBlB% zii_MQ>3}$_e-uo_%@385998zI{lG8_t}{lJ!Jo*r!szL|=2v*j!=`8$a1$Qe_k?So zXkB%L;9=FZ*)qDmKdHC0P3LuPy;L4au?)qlz;txzxQd%LouT2dV+BZIAB}B2cf(R7 z89dx`D@(^YZml9UFaLqE1)*;D5rbf^ZvW8m#-={Z`*hpD>{OUWyqrs6_Xt4vR@b-p zf0tC*gO*EcZ?Bz9$#lp6?4=zvR6abq!)n?DfBdEE-{k6Yx4`(9ld?xM)1x)$6meKx zK6aUBf>rl9q`p-82l`a{w!zkJ+J;AGQA3)(8zv`M7nwVi=l|T;53;pPp_Jl4eW!c< znc*uj14(ntuTO?va{*mlY+5{-bnbaILrOf<3WSxUWaJ1K4EIvF(s_xyfR(W@{`+mP zr={8J+(J{G#z$&|4uU`BSrT$a-W0>ow2iXFtHztKCs?mW7m%zzsB3s;`>_ZE|+2XRM4ctSD}Y zrN$*0CxdiT+wZQM)yO#}iJ(d>2 zLM`HNLn(3$(PkJ@QFk|2x4qSEax&QCkGe*W&&(xXlVu%c=nie-X(ohSANZa(+C)Lx zYu)in^n=o?m&x*u?pzXRfnFCgc5Amt0zr*sqdG8}WM7o0r!1i^ zc-op4ps*-4)@`PwSrad zv#-~Vjz)#`qvPW4gE#sGY>E!A9pVYLj(mPhpjiuRF&OU}Zxl>)vsqRsS=@+!!K6yl zVk(|wM~!RZ0v`itYwqTGYIi#UyTW=~n zdYi_1Z}uDnnq zbZy*Jf)Jn%JydIqCNco?$bGhNo?k0iSs!R>>cjZ z&pRzE7nb^LbaGeFbtB~*Qna~xhd9gfgAX8~Bo@Eipw!EUL0SmIgUc7+EYc$qyb=xf zs&)H5G}JVeqD&}C)S~>)f_Bw3I(E=y?kA8(VT2#-z|c_t zMr$j;>b2p0oI{3$2fJ72a@*p#<7+L}@>XCqL=2f|h41eYnKF8u!T22XhZEJIk3wPGW^ z{>^(^i7i5%;=BGAE?yy`0DpK)^hdqDCu3;mhDFz}y=|-{e-2$h2Zbx`hOtvir67wm z43012PVxmGatssFA6s?^$n~4IP=TaWw z=WK0>-h>3O$G_@$i{-Avy~bg$T>FW%LuE(@%T4+ER)XdD#}YuC3NGc6OBt&Y%b_rWn}H0_4A+UYOeTk@y+zs;YMmA2 zUWXvq8r{Ie2<8#yR1el^EMB3PpZ_ixNiXsU28Cw9PJ8_};;5P2mtpWtRbwpV16Jjs zo9|A=#P(+P!vs+D!$loSMJ%k2deMp2 zPROQG9pz?a8tl ze>)u#+b&)rqxMjhJjF9BwW67}-|#wm4~+3o-sx8JTr7P8$-)&^TfB(qP)8G_=c`xC zIs+ZKjMln8j_CZHLRr3mj6J=RAj&^@3~|RN34sOmN-+bOX+-zZfOFEDMu&|h9plBx zkw&6rf26)y*cZ5=h9xSiAE|-cF;{dj z@r-di2gy&Vba}izmnNBfU%fRFUtJ5zXLQSs6=BBbhQ2PvzW<)?Qmie!w}@5si}W35 z7!#;+w~fq&ycM~O@m`3<9pRA#p6)QGB}h1DF1%NHzzj3NzlQTE`jr5}`tVSH%Y$e2 zA;|qfZxRhg_YZ{wMZOV(Ia>@iS0C2R@I;4ia^B!sm;1vN0e@7WBB;Z55}%a)jJ_Xt z*s?#~iiOMG7RQB3ZC|4R3p4CEy#wWce~Mr$J3Y;=Z{E?`t5)MLl&BlCKFpd_5#pHO zBAp2E^8P}ct;VSsFPMbe>pM2VuK8;@YsoUmWl1LwJamH|g|;qd^!j;uS~Y-Eo~Ck% zwnpDne)84h5i>&Yh%8sWZo?*3C0XX0~F z4xEBOYDucY+sDD60e7HOMn2kO{mZdh3`|?{;V--Z`Sl_%t=;vPIt1$IX~PdNcA%1P z4C(Aki)JHF7Ew$~=bs(Gg>*(NzG{MR$>@Vd?2z2#u4~_YnOMguAB#5(rC=5`0+{1yWB&j!?ErU$i>BB~Y7k1yL{tlOa zeEF4nVLH}3Zs^^Giv#CY*fV5Y z^?t&C5&^HkJ5-8?*QdC&7h6 z|B>ntTsFTCe2~wEPWl4D)$?=Pd`@LtTGah%!Sf+2FM_inpEI22oqc0l5o3i;aUBww zgUHMj_;o$`?tw`;t9#+#jeq#&xf~_50A|$*^RW0!%>mE)n8Z;2ZcN%5bL!Xqkh}cJ z_H|9IveI|VU6wCt;aYFdQtFYr6&tsg?Cp^}UB@PN)UK4D^Z!Pl_cY9?CtlE{I~R#o zseuQ#Y3l(~$)B%5-kPZRpRhA+43aNnfW%FNwwN2I`93tc{C+S^U`1Y zl9~b1Hnv{90j}o3G!)H@_+fY)P*-+frXT?(~HxT3bzQF;O}m8iiG_MIlt9Y z#RX`$h`j=UT-VD)h>TgK!hg~|v%miqAZ;5tC4=k*0?NB6bGm=hEO0D}00PB+z0_yH z8f6On;WYtMSK7ZI`Ftae%Kn?3Jc`(77v5L{SS+U%U-P$u&JQF2P$>!HotmHv7w-b# zbN_GQ&i5Jh6?m$m$flUU#cO~zV@)qT_xO*WuLO2I9{rW^GDS%Cg=w9qZh7?oVFQo! zfDQa=Ot%Hl?FKW-{K@|c@ITQ2^#=mO|B|Wd#hL7zIOUb+P z<$m&0B(X^A(;U>8bk)uTT^i*0l}kvi@T|1@-{>+9K#KFor!gR2hyd=RX8b<_{5!b( zvZDKHNdkQ0f+J?CRD!Bs8p^cBkap9nT!UAFPXG3&D!@bM>DZ_NETICt`p%4j^2mN% zcfzN}xM#spS7=5IuiDz+R_CujAzfuO22lA=kpET!Sc8Q$CjeNth=hm{N+1Dk&r5G! z!_-ZSowj1)gAOWJry)CH0h3Qf(YUJV{4m@tF=d~`s%en>(RlUN>m=RGhy9zwhAaoh zt4kY-L!ye8{>en%Yx!^5Bs)^cTzK=DSO_QYd6d5v9|kGZ5GM=qP@0|DHrP80XPnSX z7Ktj=!uw%LdB;nvJ8_>SbhznRQDIr`-53^Bn6NPE-<4p?3jl3{em)JnaB-H#kqWHp zYi--+-;Uv6;fT`{9O5lH$-^qPcGe(%nxCo2UQxy+mQLcvwl2-Cr!Ny;L#Qy_mB+K9 zoPojgpJ#A^p!pmLnV4&U6wctA1~iNFfr);tn(RZWkBjs)ppk5?+vEzI*vjFo+tCr5QuwV+<^HqDB+& z;OU%;^A&D&HvGEZ;#}w&jP`=xMcr~>zlLbzil>+-LfkO_ymar+hxvaK0=Ct5P5vNFV2S4_A7u>Hlb0Q!|c9D21%4OS*g6@x9+(<$@KtmzKaZf;SQl3e^tTiYLVunz^D-b!G>rIyzmzml)eUeDzE4og;}z9jZef6TS`FMx!!lJb zE`@?vO+*noFWYxm`d9S($0Hn>0VBTw(X@4qaHwKJ+$g?3}t=;eEbGd3c#~R(30U z7ubIAz@fHAQvi(BrawJG(4r&w_U!?`$6dyz%AT3ClNZG7j@IF){7 zp?v+8w5^cbj2HQ|j?vNUJAZCain$YTO^qwrUO-|5Y!=J>o|bP(V$|0Qj<_!WyNz%a-8MpG9>Mp0i*FVM= z<1I!=0#ydLHRlp`!dor$omXrj&V$K5z(ineoqP$cM-Ewy{@rg8<54cv68o`!J%l)+ z$h%&r-c=UKo%D%UpCgVeL16866yL~342U*P$v$7+9-5~k@N`$vQZ20DaQvn5fvMEj zY&5$h-Y}SKAP9lV*VCcZ)E5y|Kb`;VgWs|AvyKM2?{{63TXsIPVpmj*16{d@L^XIz z)$g{yTE0WHbgZ4AOx_ijpG-A6d%Z)n2VXkZDu7>X+{Er>!}5AnKPuh^FVhiq6e8B* zOdM7Q=4sWsqBj^#mpuVS{RKwP%sdk{o~|grFJ>QQ)&1iWFURYmt8%&jEm3h!pQJOe66j>OLVX?<(ZDzyiUtYs=wpB0yAKxm+81` z!lZ6oKkvUD0_LFB4@bzBwv>8tWA=Z>Ef&29p*Dl=ee%3HA0BMAqU*7LxR;-wQ|TJ1 zce~=KlUNaSeBIKzbf zY2!awnT)^>6Krmyi;b?f_5c!}SwZgiaYdMboC5OiDyzgZFCRq#ym023d#EUDXYeP1`FI%^En z7)PA_ycPx;;W9aL2?sl6OBdmJEB3YqBFXA~ozI>C5$a^QBlE8okRkx9v6ln7Sy>#3 zh;A`~W5vty0-$7ZM#GD5eEZ-q-I2|yyfpcvI!pbL5P8m);c%Z8VCt3WP!4*E+8Z9} zT%(Tv?oqKE= zVWW0OHu8~#^Yip%-PH7eJgs4Y&hBe@zy!MHl}z}!vt9ii;k!D)kVT0y<%b2m(QNXI zK`^o+lqMJQ;UVPH9r4FH>&b!|@g}=QWh%T1v4)X7@!6C9$u&h9^^bTA?IoSwp1SNM zTj{)d*97NJs-2Hds+nulVpn#G14&|X7;XFSrqsZ;p5>ezrnn#BPj1wW*&W7pI$nLh z$!Wx~{&zb^K^Cwo**pBW_dpx*o;q~i)Q%_8?L&xmuC=WR)H1JCoeRrCg3$J?s9C~g z?#!KVBzUbD**M-tPsMvoY^l|bA5=zY^q~v{MAd_zW$AN7e-e4>+Nh&z5eVAJdAZ!5 zS3SbpF7deB*vSi)KbNhKEA(TJed#mvW(_-;c2Z%y?pZ10Ag7HXzj_G?yO~}yM1K>V z)$~=(M5;*Kc}a!W&5iWu_E4NXLua1)~OB zX~NX@P`CNrJHqbHi9_yWohTE2#&06PEW?HnjV&YI(`^*T69{9su6Qo~c*4=W8)lG# zt^?lSVrpqa9Yb8jC}bH87*rm)7dhC*a>QyxAOeSMG=)~owd(plM;m*C%<~H0Vnna* zVIR7Yk|VS!bnpr(60=H#4v}f%<+v4Zht-c88x!EXUP5}<(M9*ATZqx;n2$)C;Vwp+APjS z9uX|d6}>DVz%TBF-v z^Iu>No%ii+wIu!QC)7rVB$zQp`B7AnS!YIw`y%O5#C#}R)Bz7ez&1JCvX^Lzh_?+c zbFW5Cr_$i#uDL~;(NWF<<#enU>n{gMlLGI;|L>1<{wOV~fR;Lc(DHvws0Qu{deEj4DC1la<3^H-WWiC;8&J(0b$?z53Wg;Y9a*7zzq0by=ffi2|C+|L~JVY zCoELF9G;pQ@_@qfB}VeH(K#S`(uSd=YwTIxS?S&{9LTjBhZNbCw_j;BoO+%q*D*LM z*@xw3=4yxVjb)MXN|p#24;`uVZmf>LKef47=IA+_D+cY7U76dvKSi*YTY22WqcY-f z=v|v-GIXT#(TJTYYsJcXbu01#($OG}3^9ZuhBM{G-zLn%y>PEEfSf4@>T&+k3U1yo zQLpknCq{%F|6Pmy@l!~|c_KgZpH61}s1h6<&*uVs?|?~zsqFls`uFIoYAIj)`V+UT zePu1%UHGQa(4hwbf0sGGfAaS}?>_KcLnIT^8K-V`E={Mq{8L$e{^;|tKkpj#zkl-o zgL83=D$KI=R$l1-z6q@(v(2b_ecr-zA0FBTD9tYuhXrx}Y9Laj=N0Fj*L|uhlAnbj zW8J&sM#20V<4@n5_Nu+clv-N+TE8Cgw9sMq>R%<)A0hqC%kv48$Pd8+{4+JDX`UnM z*gQ3O21@_p+Me4bXnCMYT-M|L?@ciPTLrq&Mu0&-d|zob zio5z`dxu6fx2V@ST_pdJlnD$O}!xybrX4c$TwOcne_&1`Bj4)wtp2ENOXXw%zLIIDaMdzOl8-v7rsAv zJ2X_leYeB;_Gl|s+mye5kkxn1`~Qq>*X-H3r$7A@k7kW!S*C? Date: Fri, 25 Oct 2024 22:44:03 +0800 Subject: [PATCH 6/9] doc: graph: add document for gated mlp fusion --- doc/graph/complex_fusion/gated_mlp.md | 123 ++++++++++++++++++ .../complex_fusion/images/fp-gated-mlp.png | Bin 0 -> 19269 bytes .../complex_fusion/images/gated-mlp-swish.png | Bin 0 -> 16096 bytes 3 files changed, 123 insertions(+) create mode 100644 doc/graph/complex_fusion/gated_mlp.md create mode 100644 doc/graph/complex_fusion/images/fp-gated-mlp.png create mode 100644 doc/graph/complex_fusion/images/gated-mlp-swish.png diff --git a/doc/graph/complex_fusion/gated_mlp.md b/doc/graph/complex_fusion/gated_mlp.md new file mode 100644 index 00000000000..1ee9e158af6 --- /dev/null +++ b/doc/graph/complex_fusion/gated_mlp.md @@ -0,0 +1,123 @@ +Gated Multi-Layer Perceptron (Gated-MLP) {#dev_guide_graph_gated_mlp} +===================================================================== + +## Overview + +Gated Multi-Layer Perceptron (Gated-MLP) is a variant of MLP which is widely +used as the Feed Forward Network (FFN) in many Transformer-based Large Language +Models (LLMs). + +Typically, the FFN in Transformer architecture [1] is defined as a two layer MLP +with a ReLU activation in between which can be replaced with other activations. + +\f[ + + FFN(src,W,V) = ReLU(src \cdot W) \cdot V + +\f] + +Gated Linear Unit (GLU) is adopted to replace the first linear layer to +improve the quality of Transformer-based models [2]: + +\f[ + + GLU(src,W_1,W_2) = (src \cdot W_1) \otimes Sigmoid(src \cdot W_2) \\ + + FFN(src,W_1,W_2,V) = GLU(src,W_1,W_2) \cdot V + +\f] + +Where the \f$ src \cdot W_1 \f$ is usually called "FC (fully-connected) up", +\f$ src \cdot W_2 \f$ is called "FC gate", and the last linear is called +"FC down". + +Swish activation is further adopted to replace Sigmoid in the GLU to form +swiGLU. + +\f[ + + Swish(x) = x \otimes Sigmoid(x) \\ + + swiGLU(src,W_1,W_2) = (src \cdot W_1) \otimes Swish(src \cdot W_2) \\ + + FFN(src,W_1,W_2,V) = swiGLU(src,W_1,W_2) \cdot V + +\f] + +The Gated-MLP based on swiGLU is also adopted in LLMs like LLaMA [3], Qwen [4], +etc. + +## Gated-MLP patterns + +oneDNN supports Gated-MLP and its optimization through Graph API [5] by defining +the graph, getting partition from the graph, and optimizing the kernels +underneath. In general, a Gated-MLP pattern is defined as a directional acyclic +graph (DAG) using oneDNN Graph API. + +### Floating-point Gated-MLP + +oneDNN defines floating-point (f32, bf16, and f16) Gated-MLP as follows. The blue +nodes are required when defining a Gated-MLP pattern while the brown nodes are +optional. + +![Gated-MLP pattern](images/fp-gated-mlp.png) + +1. The first MatMul on the top left calculates "FC up": \f$ src \cdot W_1 \f$. + See [MatMul](@ref dev_guide_op_matmul) operation in Graph API. +2. The second MatMul on the top right calculates "FC gate": \f$ src \cdot W_2 \f$. +3. The Activation node is optional. If required, it can be constructed with the + activation operations in Graph API, for example, [ReLU](@ref dev_guide_op_relu), + [GELU](@ref dev_guide_op_gelu), [Sigmoid](@ref dev_guide_op_sigmoid), and so on. + For Swish activation, the node can be constructed with the [Sigmoid](@ref dev_guide_op_sigmoid) + and [Multiply](@ref dev_guide_op_multiply) as below. You can also refer the + [Gated-MLP example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/gated_mlp.cpp) + for Swish definition. + + ![Swish Activation](images/gated-mlp-swish.png) + +4. The last MatMul on the bottom performs the "FC down" operation between the + GLU output and \f$V\f$. + +## Data Types + +oneDNN supports the floating-point Gated-MLP pattern with data types f32, bf16, +and f16. You can specify the data type via the input and output data type fields +of logical tensors for each operation. oneDNN does not support mixing different +floating data types in a floating-point Gated-MLP pattern. + +The definition of the data types and support status on different CPU and GPU +platforms follow the general description in @ref dev_guide_data_types. + +## Implementation limitations + +1. oneDNN primitive-based Gated-MLP is implemented as the reference + implementation on both Intel Architecture Processors and Intel Graphics + Products. In this case, floating-point Gated-MLP patterns are usually + implemented with three f32, bf16, or f16 matmul (with binary or eltwise + post-ops) primitives. +2. The Gated-MLP patterns functionally supports all input shapes meeting the + shape requirements of each operation in the graph. For example, the `MatMul` + operation requires shape consistency for `k` dimension. The `Multiply` + operation requires the input tensors to have the same shape or the shapes can + be properly broadcasted based on the operation attribute. + +## Examples + +oneDNN provides a [Gated-MLP +example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/gated_mlp.cpp) +demonstrating how to construct a typical floating-point Gated-MLP pattern with +oneDNN Graph API on CPU and GPU with different runtimes. + +For applications where the weights of FC up and FC gate are combined as a single +tensor, oneDNN also provides an +[example](https://github.com/oneapi-src/oneDNN/tree/main/examples/graph/gated_mlp_wei_combined.cpp) +demonstrating how to create the weight tensors for the pattern with the offsets +and strides from the combined weight tensor. + +## References + +1. Attention is all you need, https://arxiv.org/abs/1706.03762v7 +2. GLU Variants Improve Transformer, https://arxiv.org/abs/2002.05202 +3. LLaMA: Open and Efficient Foundation Language Models, https://arxiv.org/abs/2302.13971 +4. Qwen Technical Report, https://arxiv.org/abs/2309.16609 +5. oneDNN Graph API documentation, https://oneapi-src.github.io/oneDNN/graph_extension.html diff --git a/doc/graph/complex_fusion/images/fp-gated-mlp.png b/doc/graph/complex_fusion/images/fp-gated-mlp.png new file mode 100644 index 0000000000000000000000000000000000000000..a52952ce87bc9130a982e09315c25880576b7935 GIT binary patch literal 19269 zcmeFZby$^6zduTc(j`cDE-8_cP?{xDvgnpr)FP!@B&9>?Mq&}V=$4WYaFNm>-Q93* zp7(wCe%`%*=eo}IJJ#2-HTths0I@5DVbpO^_(d#!-~;PC?#6cl_VMOh6L6jTuK zX8;Ea_$A=#wioyd)kQ<$B}(}q%?1hz1B#ODi#HxdI~krPiCwcjtw-CIF7(o9EZ!L3 zi?uj&!xTz&U@ypc$dWK@;1%}u&-LInQzc!eAlxt*9P++1_@!uT^m&)*$<~N>%!DQzfFnzcN+DOJ z@dfxN5o#>LnurLMiAW1WQBOVT$%ACzw|I(=s7%rrxe+Kz+$Bi4b`s$Kkh%QQ7-7GJ5An#%;5lzob8~jo2pOnXO(ld7#v*>pAxJ6-iXL!y{p$7xJxAW={ zPewEo1Jf+8NE&QTTHv*xzzmO9VUq<(+K>xy_eBNDdGpGk?DMua-k-087wQ!9WlTjH zzYd?k)|}6Xl1z(|ndB}C-9MO@)^FsFpOF0UwjA=r;aKCuLl-$8ly0keH-F|*dSaxy z@`sfO|44Wn`BiF}3RCN*t~6@9fEy z4EK2(`PD+78%nw8hlnf}GRRL?ID_6Vm4AfDs$t3NCXAwpzI@aiauF%!KYrK?@s3QN zujX~HKodd-XF&{pNm0Db zYK@B@S4Po9J*fwM7`>B*B*W)0dRFK>C10n)qIlUdIf8fiA)I=&+g@XHY5m()V#CM-vYoz-;) zywk3+g+fM;N-goc{D;3Dvb2tE;&ANjc*A(Utm{5?)yl_tFXm5Mu&Yf7yK{YIwa=<0 zcj3ZBY}}b|!RsQ06MaR1?K`ahr%!UT_x*WDN%~gC$~g+HY5|%8)?J)Lf}(5c<~9{7 z@Fmou)!d2deNdB^ALlJCjm!Y z#1s$BRq41vzaoGk2GN5IP7;cg_BKV=r_xiJLSl|EMQh)NW7RU~w*H69jH8 z%J|B(pB@v)X5~SDB>STAG&V_)e+qhf%CnI|M6wDdNCdKq`Tqyw08>JUVd5Hkor$flA)+8C1&+s zZ9e#V=jRqBLGpiSG|Mu)#=+y57M$!$n$2xoQjE;~3OTqdk-L%Emm&Qg9+Xf1I`z#X z;xHLBVBqfwnru`VuF(j~F72ro#6xIS(;8j@b9xvs^)sU7E&Vky2jp8k@wdsqAqphI zjYYh~knajdA?8a#QR+9i9Tg*HKw~6ffQH~iMUbL0nLI&_Pp-c%b^{iJmn_ICMI0L8KCutmw%utPq)h0hKeCiJ{D}Ag}XEnay%Rf9;w-@yL zC?%i#Os3Ku#W`y#qk@HH0Q+2(Qq4?#bSZ)=_!P@_3}RH`g)L$!*iCUcH5ExXqfy!F z4o8kzl7y*3L8Ttj`p}|ReCtop+6V4gUbDXUMsAr#W8PQfAtMUpK%w*_WRKeM&qgS@L!ak z&BD&SGSHGd!rnDegJGIKtRK@eaFa3mNYk9haZCgR@GCPpvx-rR>L9Mc!1PFTO$tTE zsVz$lnU)lP{x$$Zt6t_xSs?op>$i~1;z1b=x6~j}#=!F?2^Of_Pglq1nxcs=e&Qlo zn3ZO8QyLm>TVa*(D1#v_GK}kO0+erUi6Io=>`yHkuXh)Tvyi#dGW7}tr&W`UAA;Ej zuw@uUls{#Wd;1a9C&exo_vOr#%%K(zpm;b1J?5ALx( zQ3f_!Y(2Y#-*h{yrK>JRXl&cUxZ*rn6Z9c&5txDEvP~vBKlIDGv}#wwE9!Dn+A)?i z^^T8+7fj#fi}rl&8>#=khF(V@9iH{OQ?&GpZavO#tWT#1r+5*sIHKX(=ZaTMm|j#e z43b%N@rWoYLqk2ML`Y_P&KyByxY^eu3R|V86>rilc1PYwUwg)$f1h9a(@+y4zJ_Nj zNMlFyj1uUBqRIS=`?kn!_gf(RSfNxF70s&t-A$ix_=koaVfu;-f z1jNA5i#Kv6#;a7=a^$svYvC>sgfkAvH`Idn+4hiy$wOdM(3LYLU7)1*6w8&kGR12U z6V1%RjewH(lwy?^>BdMd#u!L&fxU||sndP&64;q2;3i3;v4DKjmulY39Y!2|W_mW6 z^sSR#z&qj)YC+WfE4!JB&ktLKo>>yBp@hRvK5xmPP2zC9`PJ!|ZGO423Eus?4#Q0IYKd+hX2b?RW zqjjIZRQ;7*wDgf&iTkxcW^LJ$Mq%Z}f)Agj+BWAv%(VNaj-sa#w2Kw*)wQWtOJ7xG z$&V9Po@1~oy`r8Mhn&;rnU&rCPEl@5#~0#Ogg>GGQbdC()cpPUM$A&M`)zF zY&tTxNqR=HUtJG+Mkdhd_OZ2o>+Y?D$!OXdEbJozDW0@vV zUCK`Na05q+oZ27ikNZx(rX)dc1(;1NcxRI{tyOM_(i^GF&x%XDlB~7W6H`RA>6h@H zWVNo?eQwox!ON}yT}etmx=@>xEP>ByAe>QyxdxjNf!~7hXwq}&&3WQn*a+0;n7y4d`e$RDR(X%e$nC_QL4bXg zR%%(m?UI^xS@TF=CiU;7k-4T$t4Z{m?2@iI-HRa_VM}H1Arx)@?a!b6OYIKmsbgLp zP4Gc2c;T;Azarkae30dH^^q4M+sR zDIh0BIj9XRqZi?g4snP4W>9%i5w52fN1v2`w6@7e|&p zIPzz6X?>g8`a!9*-v-Q+L5jfON4N}9^tJRk^Gv&tYfnBVSpCN5Qv~3%y{4}b;Zc#T z__7pls{hFD4g|oKJl_ z1GcP@Z=-L2*z`AD*w!JQ>g(&<$eQWh^lI!`{O*5F7-XA8lKPlN@$Tj@OpNdLk*fMF zNPzH3go+s&p(88Iab@I{54dL1Xx*m@>q{W(oNn-Ll8_hGYQ-Xwr1yO9GvUtDbN?ZG zCGmr537jvss4M|40sK|i>~lt){=Nn))dM7T1LAomrDf<*OYt!Qd9G(V1pWPYH;7T! z!Nd;lM!YG?ElQIegB8sZ%N1B6?`3q@KxXSK*mkaABTBdY5k2^!&~TQELjt*F-SVcm z2j4XuPWzptVxf+N4Xo1D&f7XV#$rBz-Q|q3kS0P{)}6ZrS#OAa(5sq_eUvvJo3k$@ zOgk&g-z^d<#E)yd!3d}{rJ-qB{kx4PnePKN zz1D#=ieDCCOCj4+dNG}exzV?aC6|138P|$FL}Bel$ZH&Z4Veu39ezv&$rPj7o?E{7Dpy*qV)$S-d&Xz#F9TK;I!VW7uSN9qT!an! z+KdNW>+Onv|0X7UGDmG)Qv4!gdS}^$w9*5)bucJ%c{_5=_f(EOjb0&N$h*V3aG&s! zf^0JWYs2$lzNZ7{wq?~d6ru0-PMwlI=8>m#UKCxJDm%Y^DkWA>Ffd=o++j*D&&{*! z!aS$GDKW=879Za65tgAhDig38N~avh>9T#j9y^|1?apptF*^eLF&y>dDY*Kb!oos% z#tFQ-`k}Q;i?{~cb>?a)?r@C5*RAW}{H|3v><5*n@?-mkv=r*93!UbjNcooaDc7Ui z>||A0pyZ3?+^uVqPqhvWAfcUu^hIWYpM;L3wR7oMeko_ z@mbN<#!xYkfBxgS|B{Mu@`OB?pUws>;~^`n&2(-t@Li$%ER2$T~k^^1%*=rKVz zzQ29L7@w@3FJm2=PmadJ90BZ}vdbFrDx)}*kpe+qcb_Th zHKIB|r=uNJfMV@&Rfxv7C~gyxRCImJ(Q)Yxw~XgE_pEH9q;t zD|vDzD3zSp`*#5TX%oohQ0m9&YnWz$Zpd30yogVBDs!urMIu9bzr17tK`&X1_#nDSy8SAd%5R8(-Q)hpx$6+WFM6;qoO5ckmB9CiT1b*JSxRRCC8Q3nsP-WpRFS z2qqPg&P)7w2un9?B2zxs@7rpq>3sKyuzEZD$aV(J-QG7Zp7SrSuP*Ar&W@%=@uzn? zTs;d`60^}|3f(8|lc?bymfpw8g4zw@m%2kSI8)=z)>t{#w2P7q;Q z)&XEk@9XZvYd#e{eOsTp%YvJ!^-B*Mx~~a5j7nRxwl4f49T6uhKhJv`v11Y-uq}3I zh|`jSix!(wY3;Ib@>(b-atTH6nx-S`%$oc2VH(X;B}Hv?dlJMeD9GZGmH8VfY7!ni z>{N7*>FJ;>-+l7T1`g^Jkpxo zC6RvpwLnv^^vPDt5!SnHmtq<<0@w#~7`#KPoGfZPA^YYv%Xz|Mddnu=Sx%6|2`_Ed z$Nr4o{ihfF4nK|H*+??z`b0Jrgv<7O1HNn<9AV{Nvh%X$0aHGFGxEsQ7Sj2so7c&PmoAWDW(3laQdGrpBX?B%fgxW&EH_%A=xAd&3^ZXv7<1%t76Gr ze{+9fQc?k;H~(|Q;ZfJl{9GZ504aIC-sdOKjbbSd5KI$gt*jVXsO2c6P#gNo?lWJH zLlELPY98eDT;`ZbA}W}h~4R?xJy5+F=_ z@CkMXMyt5t_}-}p%dPY`$CZtjU2zeZ%a}R)<@4kUappK`J6Tr1mffru<}Ly*qrouB&F)* zDK(e+l&C|H6j2o0Un{(HFtjNyyd;NgE%#%Db7B_j_`EQNzg7>eyF$>mV6sBxzFB;y zX;ebyUN|j1f9%C%G!yLab?#VPN89M9$hCWESxE7_fck7P{WjcVJ)OBWbV;X!y!`FX z+0sT%Z7-o!sut-$<(fp@%Bbu0VHuB#-O7od!O`Ju?B*jsujSN0%H-dSGU~7Z8PY!y zylIXTQX3nlpNjllAyxpY@LQ**`=AUqR#ZqK34dN4gMT+OUlP7GnKHTjwMfsY5;VLl z^=%_1pFy}Q;?8G9+>}}(yis^!IQ)EWII^u8OR1DSu4n6-Z~L^8T`9cBR#7kB!q5z= zB;Ob7m~RZFdimn3Jn$RqdjOKWRMgWb$YZ9APkyV!2vtfVicfyC!2nhIL=XU^>fNUX z=d+rM-9I0a&#L=!{Z#Gy`(b5pNq#IMUl|*YV0m@_AMgVu@Ynm_|KFY%!3KKf?yJB1 zMg5s&l_G`Fbw~#~>h^5ISkEKl>4UN9V}T5mpM)ct!&K6aetggT%`FDQ7Ng()iH5lp zrg&mZ>FcRA`-X<5ymC!_gjqe41U&57Epz|qU5Le5 zf>A{y=qNZmY_jPDUzAx@DNkSa=r9lDTkHGl$0Nfu2P3l4!*HD8Yd(Ji*9`hp1fSK; ztdg$o*)g@i(~fJP(4gSPcjL9=<4@Z166nKK(MZlxS3@^?H^ijG@-d}9w>l|&=#Y#$ zt%(|nsnf1F^JskoW+M=8akEh;L`|g!fMVf5)lr_N-lt~Mr<&*3z7HSR0p-U*Gi)TH zrETp&1Uqv(%Usl7zXD^Ow!%Z;zI(q2YQ~FiN;iQc}(FC@PibDc%&oNw}S@bQPNU zCe_YO3?qzBN=K|i#v}xx%EQ-T9$XD+z=oEa5(#Vw1?B368HI#mk+sH!AR%%{8osNu z8Sk>?D?B-|I#EiOM z(j7)=i)XAA#ZOPxj~W15dS012Xc`g&-w^tBz3%qlWd{8$xO9S%N;9x|^e2E`K5fMC zkA6w=U?c(`FTj`K+#AbTHQCJ57jmEwqaL&#Pz z%%Fb5%G`a}%ps~6$3(zRDCe!?XC38;_Vq9IA`C!b!4N$sK9k}|S>StKqzo&_R}4#e z8R2|itaSnTN^snFb~G=XSj4fKK;0`I)-Ds*Q3D#Hu!4&h+;9=f%27E$NsM>I>(tO- z=>#*%i3aMV(j$-&hwX}sq^J@A5$(l6Wx$pkZ^kEP^MnAW6M3DT1fjvx*<=EQM{JDCsxf#Yd{ug48%Mm`p?hbu#JI8hK{;rSaIL-d`JYOez0{j(RC>Afb^J767m;Fd%aVihhl$AcD9;=EQ(riPzyAb< z`k#Q7oApqkgYxJ_6%EQc+aP*r)p`T@p@ItWCL^CWQK3F4c}sN7Tx;;kB?-W44u<`F zAs1xb1bp4kF3bj69{t<}M}j-9`1qFGjei?cVFYzJ?-zlsKj!F`8?z z6|b|k^0l?!L7PgS_&87*WHutfEx4SSBXbAgZ)|GG^evR>ohVPdvMWMM9I3lTXjX7K5RFpVE_e#QU}$Q&Ypt z?QS313eZOP0wdX39>rRfRN_CuIS$%UsLnXhg|7>nd5(I5C#Me~JNo^T*Z&FB^Q~*c zrrkX}dS#0)eyL#uwKbs?ICq9I5;O=s^?x3w-?=mG?n#-}*aC7N<)F@R6_(;$`g1)8 z*Z1`~sLseDwDfy(K--?)US>Z^RW!Ap-*~Ysot{2_+3W9v=Q8ee^@)E~L6+DWqbW@o zJ;rF_X8t`1uMEz>AU3_=s;kUYu6zS~rRPx*1JrG(cbJ3)B;({hO{r9>FZ+Rl0ZJ92 zZ)|=~pGR@Y0f-&Z0zl_W5h}fid<%|U2Wn=hTjY;nVEU7B+OmzxFNjG{FStkS;}Zl* zV0v~La2x`!Yu4VJ1@1shr3b7&91gx8;}FMA0wrYki2V|P&Kt{btvTb9bLtxHqw>c< zOfnB+flTlU{`h3c*YZjCh+WTE{5}Re5D){Ido55xgWOwdVDikVI}wscB71tjXxt zKGXOF<$d6J%J+yJUO{^w!-r%bhKR;#poCc_0R*6os?A(JX1)*n4G_3`{z1MzqVw2@U>A=R@Bqd+?(t6+h=vuH?wGzf1&&Ep0Tl$ z@85}`iq!&Tys+-MRcW2g?V5N#asUM3zWr2!TZ_Olc4XzjefE^TM1E}i%$wX|zMVPF zy<6ld?M%$!nbEG6A$x-e^5l%gL?*xbG~zb(ZD;?Z1rCqFyc>T2TJsi8i!a}HSe=~z zRB)R;U2d=P<+FYJxI`bFvbp5JOEXpD^p6m!9xeab3d2^b52O&RnTuP-Yfmf9A8zlw zUtj>8%!E^mtre=NccV)6k(-`tEO)`6>)Xg+(Bt*nk7AUkNv)j0AR*elCk@!4eYK`D zgnJ!f{YAO(77G8&lO4L_-F5I_^U0%i|IN>nw|!i8-kx%kTYGK+D0nH!IG0Us-ir?V z%Z)M`>dt=iM&K;~YU#Mh`8%QvepQCItUc*{QK-^>L+=;zbF9TdmzMjx18kF4XtSLh;uCD zyhCwEyz17$pOD`~wW00)fWJgCGZmZL{L_J@P|fK1K75Ad`R$IM^{Gpgp=L;V5U1BS zEQ|SG`FHVh%j#ofZ7Q%gdB@_vr<5HAy|E9HYsuTivgXDEsF;z%J%uLTaAR(6h@Ih9 z(`o^_$HgY9p6}6}6HbM2I5pYbv)dw8N{>q&xIsn{jd4po zgml^s{EUg2*O+^HTrDz2Taq9WL|mlWLUl4HF?l;>?08z%*hxaLWoew3v?1mE^hwK_ z`cCSjRn-^XV8Irbp}ckB{jTNN~pqdTURgXZCwF#Z@r=uTaG zAQ|__>AZta-ec{C_=BRzuhspk#v5JQ_BIuZS!F@xNHHevlh@?wf_wzsdnbTo8KHyn z=0F|eJ1hLfnsMMm*j)zXOkqeWLJ;;T57$y|@d6;`ZSEu&{k3zA+c|qI7Md0Iy6S6h zNRr=2+fz8%UR*r1NTms?4c%IM{T*T9KjL?{_tng&#k8I+B?zV}?F}Z)Y~3(0xGgQ> z+ecJoxP@t@dElb1#x!F19)`Lheu~<>^NkK}ymcAfv1(;3GW))F>+_xXaH0_=`!gkr zsqXRf8nrqHQ$VyCe@s94qb8@lXbF*I^k%>@w@@fRHiWTbTo4*A0VkcMqiZZ8PaH4z zl5+5D#3Ia02sy%LgBpY+K5=59uU4fnz_84diTWe3w*9c`?4Hz*D~~MaqP-Jz#K*x1DE2^^4I4!YQD1sjraRklZ|(> z_-Iw3^UnDBT173bwcc*a8@R6REsAWtgxhT-3VLM-$`qV9eI%^sq_B4r+JZmkX%x}d z?~NL)p<;ul&6RbF&p{@6@ERRlt?6{mooFxFbfe>wJ(ezzi~60giup)iZauz zMtx`o{|ZG^i|I489m%qEH)02mn&plB*tZ$AG~pY%l$s*+^+g%MemWpCWBJ; zxHlns!q2Z%Sy+m`U;Xm-SYD)ux*0p~(VIvep`B0b?a$z^xptqoClH!mm6jqTo>!$j zsz#F3nx}}xA6OWQf3RR%2qd(1y804Ds$9Abf84YIZU5LYvOn$Q``W{sa#I;xRuZJ} zWJWN3%6=tRb+cOKs$7)?bK%ZssJyG%*^0CfUuEo#)8+%-$5YJiwj5J8Q%T8vgq~Kb zXIo=U`0a1Q zT}N9{(Y+6@Pyp||C~NR+f*3lkusYmz^5a&`vy}hhhNsn5<(?=dPO`(zg zH9A*g-Qqb{SJn&q_gU{C`AM*CGnHMmiQo+7?!uIs65nZp}$oOQr98?sgtG zc^mF32`_a?>Q(v?ckHi8cr0w^4f%ziez0U)$WTHT$DWi)vpwh;qW0q3mTCNBn??8X zh7%iY?xnV}b+yJuCdoP+(Fyu}mV9SuYc%uNZ@(y1Tn1M;8(u}gZDklnoi(3N`<%R# zqUt3{?T4k4!MnL=V6AA2(xQi|((2pTR&H#}n=S|sitVN+PkwqN@UW$m3p{E8?r%F@ z*|T9Poomr*{Mqk}KNb<5)iu(t6gk1z&yQg?q`Y11HoN|4t*v1k)EAaAIRpr*2Ok$~ z_)4_(mhh4$c^>sVVrxOr8_z9Be@!x-*50XmyDaH*E7ADQn<8->5LK=hr*FivaIj#k z!N|*#-@50woEsUTV9d&Uo}WEoi8b?8c~es>8+;mAn$6rE&4g7r;4-k+=cOC3)SbYv z4{R(RzJ620an$=%{F04p%Eorxg-!(#=jVGP3I`0c;D;F_5w0W+qw~Nw?YT^oE_s2i z_i^c2W%25d;Pk5bJF3_9m-0LA)T>5C_T9!FYkNZcn8xDsbHkokTjEmwc+vc{p^9Cb zQFmt>%T=M^&$%YuS;Dpoj~)|xRIS!vK6m6P)2I9 z54Qc5bGI=oo!{!7Q%|+BwVajqkPJnp2pGgc6l+Y~rs+ABDUZVfZ&q z**X^|v`=ZSB(G}O%FT_FT(XD8 z56b-v{Hw~ma&Kt(SdxxKc*x|{l^*|2wM_>W02BB|8y^3qit6Ol#X(fV^eCene{L&D zCNJ8U52P9j(02&)o7PwltT+MSO?yo*F z@C+aqKYCxnebP%H2~aWB02LGEk6Ic4D3p1=0x`v{P}}U>=b}=}yiX!Z1(Hzu0{~SN zogs7zLy7y1SsVyZp?K!iYylBZc_85MucToh2}1|#US&lQ9c5Dl8Y9xqGamBx;)9;} zeJ-p-_WPvwKoX$ZW&^4%%8#69Ks)_UHld21-R@=0jC!8kQa|m6#S>II9Fx6qxMa)u zl&lML&Ui4!DxRf%-k`{sWtFQ>22Jw-GZD_0$$Qit^>be5v!GcWGo$G5ChIR2AAw!i z!;;#){r&jx*Dsl{13fX7SHg&8XnK2TjqhIb9^oJm2(znGL%QzO)=IXK#ZT#}8TLj` zN59Hv7Z+l=(GBStIsYWP&v_xkW4rB%&Th4KoWz_h5}jLkNd)jg;{pTv!n4V#ryhg* z#Gjo8WFpG7_a=?jY;BtGBg^#}<)q0lED66K1Mlfkz@vY4)bZz_(pSOL(A<34{#9rD zzq|^wOsOg4JL8Vik3RbRs@_9z;J={jH_C<{j!^9*~6CinDtH?VEt$E|nNd)@rEI#_#&l+?JYhzjkx) zeZ^1&3dfP~?hnVw&|h=O2fD=Gx+ipdDK@>Gi`})aK>*1I3v&x)hPClcO(SKvn=Cqw zlK+z|;sg!M@uhNJ2e)8i*5sF!?bj$*XC(etd4Nk4cVMu``adWu#DV}9xVrje(r_0r zgv97;pxXP<#3i3YcRzrs&LclRN%;sdh*OO`MGXH3fA*tR5p?{a&EV0K6^hf`{x~Z@ zT7W5THbTlOXEM3yAMWEu&aS7^D}ceH_Xd=%k}^a=5y6ii->imLn%Xv9Gd3?J2(MLwYI*@-g*h~~< zUnLzIG9p?=C!Kgq;uPI)Q)MtDN?+o0%r6=K%@8^ zl<}AqO*Xp-;Oiw=6()#L|NqDTFSZPWO@W2+ZVNhItL1{!8QBVNO-e(3?T;>+f+QEw zQOt$yM}QeOL;_RhzEK(21HUBwMTaPj{|TnC#;}s)dxl~lffB;XH#nkq!+B%lFbIt$ z6_=9(4i%{4%EA${p-0Nxk5H~(cblxF3-NW2h<>7{X|3RLo*U8R_^SgCNy2RYW^C+a zFzfy`{#bD^M=QB0IGTwWt(lQ%4EdnEhH_LKB^hP^9f1WS{HvEB%eo#DgK*%N6v)IF z1^nwDIy*92VjS%IAJ*n5mfeE{r}`iCe90CI3+TkS7Z_}~bKG=oy^%aq{qY_Mb!%A$ zJdB2v@&BT@@gb#l?Y% zEjI%8#l@9Vb|w^H%Cwl65)THH)*)?1%~wY)U&@%*ek_tWYP{jn=QuRH z@ZZ$(+z^Wbkc6AGe5M4jB#8RJ)m0wG&13cPgsp%_qg!CThr zD#A8mlzg%AE0fLZxvgifkmf+l2POtIXA%>;+wQ(};tpC@TWh}{1i*s-hnEcv7@4Tu ztL3nMTA@C9_Sv7Ru_B_a?pqLgV;}pqwcyY6L1 zFo3h}Jyb=;^DvS{y8;tQd;wr*(0$24vvDsUf^qNd2x=n8dl5331aLRt<~Zgv15zdr zE0B@o<)WejAc*QJ|K)@D9v2O$gI^QxGvb6W(*jf1TTZ|YkxcYJLG1oT8ZZAJ@uD_9 zs4gX6Jp*}r#b?isksulS60Ems4dmum?mo5;khMM%Vfy^Va)~;^S$=7=-1TjwPa{gezXAH{? zQZ4)~9)2@#b)vJco^g8QlJaNadlSpf-LzJ8`_Q|rj{j~VtNJ$++3#38cXmLuoJQI@ z(ET=1TDqZK zDxzlm-0=i_B{V~?`cvjzJSz*d+1vgJmX%IbOB0op=h=c2a?}3is}i?2)XmH6<+LiQ zWRjD?Hm9@KvR#?Hp*uE+Th&IJX9>T%uWi*MA{{Kt4$Pub3k&{e3-ps-3>HUlt7-+q zwO4qPWKbs*kYDuTkT)Xf)rATHeqfy&UO56uN(OL6)OnY`V@*s&hH2WS=P0hXgDdt& z<$l+7e&&%!n?f^a%(z}DtPs<7_S$T9e??J9sM5tA>QlFPVgtVs(mw798^CyekcGu_ zfiQ|lf~Y~FO5L3uG1Hs(GBzH=mOkYa$}pJly1`H!@dH{tBMlyfgOB=Vcm<=m+YYRN zGkM2EC3(52?0AqS2|QQvi_-t5U25Au`W#eF zE)IB#6`L~b7Yp)B-&+x#|5w^?j#2^H;$@(e;3)N(wjEPLvXzVDdCn_zYRo+v!m z@e5b*`Ce58=}qIg@WH0u?c2L|b{SquehJ2W+(+C=7atqo?6#kLv$Pr7F;Qg+EOOHAI-w&wYNXCVm)e zsyV!*eb5MU^Ki7wV&;bLtwL)fm+U16;=6K$URHSKl^D z{D}$|?<8}4DkxtYS~&QL?rEkrgg?&dG>EbfrL(WE{mrvE!T#UNTGOsh|iFDq&-(F-@@! zUpb*Ao~q2ZtmQ;PJum9Ulz7Ryq|I*2qq@>r=_s}4tuWkK@3{0V= zePS9s%Z>Vq{Aa{60dTBW>Z!>4PtZX=fB`Q7y7~!ut@#9@QkUWUz^JrQgA@JWMpjtp zps2GqN3@W^;^yQau_|G+il4|xS!%)(rxpszgSGp60Zcwf?6rf63YSx?ETCrG6=%z- zDQBT?TK7k&C-6Ee2MDMlywGV<8MJ+Vzi+&S?v8XMeB=`|c%0!f1e~=eU%ReIMUb6b zmrw=I#SCVCqOUeZZYpVqtiRqd?~ppHwOJB7Wj(FiE{4$9Jv=@ivb$vE{Gn2{BxQW@ zdQx6>Y4@klY)06g%B(AhI1Cd_1I$f<5;}!_&mw&BY5Hpkbo&3u68OKhsQ-sJ|4k?V z;gdPEr9hNZhuO{L$cmlwO?xk3EA2=FY*=7r162Goaj`Q8_JhX)F@I+T?hgfo2h3nl zrrmu{eHCi%Z1b*WCa(L1tBR1&mm`ny!+6`m&KK^04J%s#gYs`ig#o%T=|>?SrB9-F z+*%5|rz-(_jA>~i7ZV!4d_eeZ<-g1jwPC9k+Koapf3rf!O8Wu3xxYL2WNu^41;78B zbMGgeMn#yk2ac=a!+?LeI(FEw7xI>;10LMG{S#fLxbbtPlm9M`d zxDRzlGbtKoB?+FPMzS~4T~HKbG7;TiL?hD??H2lQ*W61t>+tQ1NH*79k(&v+Y7QAZ__qwZ z&Sw@BAH<)_mG`eGye0+;`bV9cEXYYe2hn>)9A=hW)GcX$r^*ad0woeRf*!uTNokcVFhs^YfEzK+`%WHl%!65HE>1OzC{r@Wmad$zR!%Q~^(# zx1~sCej?CFMAH~%vUN8Wm%j5lg;w3+@nK|KY2uj3=A-REs(% z2skRuUH9D>rALD`0K1-?7o+Y3g4Jqh$^pqKFXejd`luk~zTV74Nk}9_+af88Pm8vf z>RZ4O8;6~g$kxL_O)4wxVYt~gK4g=am8h zpN=T74zR&Sl)7qKg-aik0ypaT6BJ|L7pn+@zv%D#c3tWB%~t+RVNGd?eNW5Y8l*?7 z@XI%#Q>`)blOgvX*@KQ!_Fk>3iIr=rZb5(ddPIs>cQB^kfY#}Yu~!p&johlO?7*Nj zLQ`h}o#I)Qd>46u^3EaD`QkCm^!8+S(Jc`&xVjKyGyl@?Ps5kS!((rtO@P~TK*&RY zgu{Iiww8r9+h=2U0Yf(RLB%?FZJ62Q zZ4ZHI=&7gX%Q#?t+jqmr)wpk8MlPA2HA{ZUq|$*t%1o&l^)u5Ks8JHVqVs*(O8s8v zeswdNf4H(uUjQBEN)o0F#T`A>aBn0cMo;T#AHrsUsNPvuAqC0GhrqJ{w_C{Anc5LMmgF*H#c?qRREGgnJ z+WP^#17#SkeG28={3IE!k~LOSD1{RJ8wDvg(lKca%w@PN@JXu5QBo!%H>7iNy{e|D z5qj*yeRKf5qMokohPK6GYk3O95`C$~n2bTaR?MGBNl(n2L_PyQmv~FF=VjT@q5=h) zoxuxNHWGocxZufSFJsvBCc6J@(6xD$!>zS1upea<)gu~JhVX9tltK(!H1ptkVSmS* zTjX{~e8kP_xG;)|9qmAcAvK?TATi77S`g(q%X`*a{YrH7J>u_?hqO$L4v!x1t+-7 z!~fWO=B}AD_rt7pKhUgI&#qm&Y+LPLxU!-&CK?GE0s;c2tjt>#1O!Ae;7<+}1!xKU zb#My&Lv&J+7Dp)mO}2}GK!qUtR!q&^;2_IG5C89DpJ9$ZE3O*8a(NG@!i3xq*TBTY zM77EQZPH}xDRZn~4Rh?6;Az=AiHSk1y6UkpK~9!fxi0%n@E;f~0u8nQ@b}T>3q7Q_ z+Ns7CuG>Pf4>GR8N5k7^-ot{FIm6!OA_Jn&ajB5rigtlgLv9uI8Ia!kqxwT-h!BC+ zbb3+9g!9d}GobA=XqV)PZX6^7OIH+e$wtLNjHl9n27IziN03z>-!1x$he#s|`D)`2 zd{V%p0$cOnkMSG23|*gQPh~#$z*oIK_Sxwu8>3ONqFM1qf{JZ(x@C}bO{E_O4qZGy z`nn|Ln1BWdBx$~-lJGRQApSrsUIk9n5)o?&ve4H)<^WB&C~nhFFXR)LI8R13o> zcGl8Te~Y{TZ2^#BO|xc-1A1Oa6#8gaXQCbV!TEY{to(-aR2 z9m4>LJp<8{dW8b7r98L~%7cI2$uJngRQM30(fV#@F9`8nD90eZZQbrH zqInk$0t&(`LK_DurqDor5&mKp_%mqnO^ew=LnmZnbK@MheTk`AM_;#w!&_6!7N)$h zjS)IH8e&dugGVtrTS?^KW6R}!>eOX|w}iy5U^Ih*s7w47?0Gy)>@f9#Yl3;bT*aLi zSaIj|jbxYULwd0vR9rai)pi)t2x>w8R$C7tY1hyz%x9z@O;g)J#?Ivzw$|oNo z?;?1dzwA0_TWIK>$in9O8_ItCNB|*IvHr5O{45Y5zkrAui*Do;ABCH63N0RAv_B+z zU~mOf7=y#Ly+1G@`~Z>1%fPA&!(jvxz}~0Gd;>4Y!rrwD-=4J|&je9|awUwxuSsn_ zkl;ApO*yB?kA!ke_4NmJcRuZeyCy>K*)DrDNIr{|(fv8FugzR?3ZfC9j>-`-J7Q9b zjG1@-$E7o<8Rfw%9b>6OH*O#Eode8GuL>$>f&vjJ7@0fns##nZigUMsCFS-5CsO!B z!CS1>WVrtH4LYA~g-LN^Zj6kz!;^LW1_=GfF~Zg!eJEvM>xwc?j;Tx|VjlY^M7zdDi z%+PCim+89U?}q-t;6}=J(?`DtjhfMsH?~=?$Ta2+*I?+M?RZG~Wp_{rCixKKb9>nk zEZmk^#YoTNvMG|COC>2k#9#@4d(m3?$2YpVvR^HAgrK$UhC2#<`M5Oodly#jxzV(e zv9v^J;iK9!TUIQ*jY9?I59OTv6e;QprYG%!pkm-EKV70-49l{I${Em!3o(ku;)f`9 zo3h&peF6v-xP<-H1e8V;lqu~G-4UEp!vn4-BUBWkW~@>y3*14zSm6A7lzpQIXw+o< zp~anFl@r1ca2+TBiO;lD6#$yz&%hADvU&8lh%_sLPxr5?Fg=0_hXhYV$Vzm>872Wr z1TL^zgy19-WM!EOIO)#hV}Emr5u2~@C6ovf02QG84B$snox932BBR{(OH+1Q7RD$W zq>&*8nXulP9Ji0p_576gD;e0wG*@+(bZBvjvtHd`Yx@72%>noQgETA*1+2N#7 ze6H8#IDwZ0#JQ)e>3s&!Iq@CQfn1uL!&@lHJ~0LJw(0}*);If?8(N*8nBNe5Cyb1x zLggnilNw5nYcFhoj2yTGM&Z6nEQJ_F&7Ma%QYL|9@f`t;FFvh=~DzQQtSGcWhWcS=`k|APW z9Ay2lh-$G%?vzKUsyYziKV&X>@pu1ct1E#Ok!qYZ5B0-_NSeiCc#XN8osf0MKjaItgg?oV)DX*lcQDvrGf!0Uh0PTdxvIMdg z=)dcDjG^d_6B5yvjS$&K6w`9#WU$yBbI(lD$%2BhDuPi(hEMI627et+pN6xFM_!N% z1&VhS)Q9)mgaOA*;#cyw(r>c#WHW6n7LVAvsVPOnb5WUtyvaFICRSZ3IxyN&(|9af zjf|p}SsjtH%+QJ)KdX7sO|EyqAzs~(|8w&<%K6X3jr>5lJFpXZ?~W1EZzo z_>9QZf=Ln+D&U+v=gO!A0>~ppS&`s(S3*4B^{^0$Yq=;&$)fiwL5AhKC$2sHh+VhQ zX(}DcEiw5gyMij3!L?=^%F3Ov9^!s)HP#z`MCD1 z!A7^==6(fi9UYz2wt(EqeM6z10>6)^sv}KnDd-xX?~D2O6se(f!(XEjfHcYwBt4jO zBU^y^=Icu8(iz1J=aN!^@9B8~GCS24gUQ6&Y!2&2+!8pX9010JuoxndrC&{E3v zBtBTG*V-=;BR=@jg(SPI(_`Ivh%-;u&))YWE6ET0?{-S6ou38gk^Gp@XRdtBA0P!Y z18LA1C;ev2Az{npyP4aZn5c5oVSc3oTXXXbpv|{nd#lX?o3TZb`mG(IZg{UurT@^j zS!v!|x^pK}11UJs8$+Hr;8$jLTA#L$WNF!&CnpwM7jpKy!!64Z`Zja68&pl5d#<3u z^W#ggE0Xn3HbZ7@K{E_n(S$DsIlwRpU+x|~zi7gXtGzMJ&&uz2xOeU1CI?rgf6$g;oX(wUz1L9^p2Nu4`oUVizm{aPBXn)2 zRi?pA%!_|Eld=L1M+sHR>^?i7?C~!WSSi(q8}$@^x72}SNuBS& z5XXr`i%hRvM|DkF*Nc9yj@z|Vu7(Psf-ywrpC9$f4-17?u87Xi%@n9RO`w*BM2p-< zt@uX}k2jEdy1Kv1PboWV`JSB7eJwOC6>Z&rBwb?wzqzc+Kf-@7mqi4h9wB*Y%{GD2 zm#35d@``EL&&K3&5`}9b<-~t_QP&Fh4cKGL|#v@KLACB78#)&7CA+7Vzm>W4A-im1x_^d*M(MR>K8 z=n3LYZ&-#QvUNO_^!+^pNAfd~kGdI;b!md&(}PMu!SlzqDwC{n9*LFlB}(5nFiI@4 z-ro{_5#lwJ-d{E0=1^-9PSHdP!`7YCe$ut`>lPn)K#M&{pjmalJ8~R zB0C}0vx`>p>+b7~mrAWC{zKL(rEm#vNHYF0p1 zxh)eLBqn&RcpjDa|(FKSilon9;2kx#WY^M&Z5&dkXwnVQwTodzZmLRo~HpS8{a)O-L-yr2nsYJ=77%4rZP2Wb+CGFm#F-R~Rsnq80m3@C? zX7$kOcp6XK?S21jtiocm$Iq|-W$V=-p4R2PB8ImUVz$lYOW`!`_jlgy=htgo9uG6r z#It8s4NW+c3%GSlP;2uoIXhJw-)pZxT@l^e$p!dWO9X^Ww~xuE(~I)z)ky}Gp2n|l zDUJ^+BrqgKnpW%F#(1XbIG;26HO0RC%m!Wz>W*WkZCXvs2-;_e0LRzP(c;DKe{&wq ze=TSlK#IDf{3Ofelt;?H4VW-0Plfr{0Dg6r!a~#EZ*Sl=QW%Y{uej?lN!7_ENE>X0 z9)m~M2Ly<)C#66;LSk-klnLgJzmeN!6pL^#vUN5Yw?C6^q63QO2AM5}pRNDoq}xNd zPe0Z10K9G+K6cb&?wibXhY(=>CBL^G_aYHmvG)G<3I6^O4{MQQpD5`T!_*INUQr0)IbYg{4nZ~y z^So><&s}($`D=b(pEG4CGThfC-WlqzAKmu&!BIT3ey<9iRrK6@-q{N-w-HeI* zrtotcZnKH%3OIByS-pDVX??Hu02r}0*_Vc6;NmxMo(uv!FA*Eld>ZP(+t#bBRPv6F z%g2mnTbHG8^UHpu-${6#zZSb=E$c-3jhp`J#w8y6p3rz8ygcG>@7oTiW&Zn;4E~k0 zO7443e*&n^6|-0yCP=jIC5Ws|XSw{x-(IC3o-rWV8=U;OBGAsfbX?c?3W@SSX!Hx% zQ)rZA06qgl-scHlrm)g#;SX{4gVc3Nu-hLu6DEuGo;>%4J%1R8qAhTEE09ynvU;dZoBs^PM! zwO6KOf)5SEaP2^^eNrSifngYErpcA0kJ75S8i?J#4b-bKejMm zO;0p>FFn$@z3m^sJe<-_G=A7U#-lAXxm*?Da51uP75<9-;KqY^c}c>Wyn2hyI(y8$ zq;nE}WiL#=-7;~!UW#lVwoJfEmSbP)QWUL)&U-pB%@I~inAi|!vpVdNu?{z6`Un#1 z%L13Rk_91uQ|F+*61;h?H#o24;G%=GRjAQ3(~eH8_9h9Z8H}hP`&lXQ+W9J2(e8M( zR#W-AKHG^v3+c8(mujpqw6NCWd|(X)obM$P*$^UWdQqa(6Gv)z&ev!@YB#>xISev| zF9uL^E)59yU2p5Mn?Ho^jJ6R>{m%6NxV{W&Sl2n38(8CNJu9|avrl#~{^UrQjYZ^3 z&u2U;$E@?1MEn@pR1t3H>)ZLRYRKD9tZg{ghu6mU_iqBHlWn-gi05(XT2rWqHe*Ca zi~!!x%H5HT4?b6Ww#O!-B|UiR=Wl)@el{Clm&w9A9@hrvW#EiNfvzw8N=2;jr434# z$zsdxD0`1*<$LZQ|B&JzJN1iXwn$ShgKr+1>m1h2u~_vK1B*hRiz6`+F?9?~kKvBB zH4t~z+GX)2E&3i8Uy;&z*G%ru@*elZ_^er;T>Yx?5Ey#*Vg}tFV^_BfG>?%+^%36F zuunN%(V{hcS4YZWW);;aDYE=?l2wfLB>dhhY#UVXVSq2ZoS9_5DA~6N9#jB#^w)}N zwJRGiP>uJlboh%Usc=y9r5!inA`Zzo49A)7uzUBDtGXHYSr}oU`nDFi{|=9)%$xci zt#(GlD5%0@?UHKWeDnA4*-A?XM#hh~nfw`kTG;k%N`jYr?~bF@u9vSOKCNA=-Q7$t zJE*}fSepL`3?0{>0Es?tLq zltZLbT`5Ox$-F8x=!GxAALp73?yjoH3_g8j*>2jTDlJTKxLD<4KzN|y(K`F=%J&yWl;Kreqk9MPg>6P?PtM>5jqdrL zy!&obk%8cS-LQ%eMYOL+_ZK8R@7#w}LIEUP|xltQo>v z*2&;?5#|3*) z+vwhoEqw5O<8j%X%X@sHtzrmQWL-eg%hS`So%F2Ag z$YKHt9no~bsUb4-a`O-E(|tv16&c5IG&v;G7^Nt^ly3?7$-;++-`L5)s1WLC#aB1A zx8R~WBvGt}A+ab7&`{HlVc{A<`obBKZa_p>7|v3z`wEDS7hBLBL$TM!uFo1h&ZHbA zcbsUHp@?j3_G;fWFyAX=^5JNn^WW!$m zBs?;3<%Gz%L&Mz>4l2`NDz`AB6%xWDQXL;DvlDP8PrXG()Z~+am4Bv7n{#J5crBdH zjVZ3kQ=r%q>xs92 zRZZiK9MhZwhp%aVc4|t~GM4Zt1T0eJl#d|`bH_vq1alm*YWy_;%Z{j*zXTsF)El#` zVI4}nR0yO~{`#eW`g39?ta$7DXq6^>4Bt)eMSQL*JCiQhk8sCO5!+AXrboq$Gw;XT zIOprQY%}%0HDd9(DCGmMd|L8-?{&BcrbNpprzC~xgdNwo%q!-L>kBvQLZ6aLKd0H- zIPDVXV;+P2y}4DPGzesGK9PO87nIUIhJurk)suRFtP?Xei4B&lvQDSjqJg1~s$K(^ z?1+ogHWmv5SZ;tuR+&Vkic3}*B@F+|1Q4|sNTs+v_TShzjis-r7s$fO(|FyixPk8U zR?wHxl+hrw6=~|PxKv=db!kXqZ!GlXGZ8fEXdqKUA`+j=2j*wVM961@CbQeDs-WM zL^<tW3D3&(SaH_Rgu1+orNYaYhM9V0O@F10ZDuCHj6MgITBw{oC$xF;@qC_XUA&<`oa@NY7C-ndtL zybpQKxm8ShtZVNNrHdYdajQRu{0b=X1?Zy>f3a6KbrIn`gJOFke(@mr(5gYX(ZUk5 z5DlS#MMt7E;T*D{&{#Hz?|itvlL4;r^!@~C^lcR3cLC(ZdiNApY9%nCGegPKs=%G9Q-+=MY`z~B=+;70$P}C2jt>5N&wG<-Q!UT+wv(IA3mGZ`3KfSY;U;meN0BYR}O zWE3$`oz7R?lnVizUiBN{;b$y@XD-_ zAJMntOk zgcVKxxbrb)ZKqjshusJHJbP(HWuC(cL;`0=6LUGjD_XVj;t^ZM4DBASpkpwauSG;{ zc{FyIJzcM;&|LqSj*JY|YBCpc?C#&Z@3S~szcYmAtw#}8Q9{LvmIytP;y_8OhZ8!B z?_p$AhGbSVb~ueV1DfA1a&V9lbqx-}z2-zx8rV<&rk3wiS*QHWV#_@KR-qJ(a2_?Q z`V~mHWhm98CAB&fqr(eekQ8 z^jbYIl=Xh-D_XslhSklxNIERu#w8^)D$ZUT`?bkJocHX+AZjQLubH44;8ILjEwNbSLSyt+YS>uyVKHa=++7g8boPBJB7}firrxYx~d>4!F+bFTkI` z^*I-a;zxd3QNNa@&rd6;=J{GVU#kovL*2Qr$@qt&hHRc0DolBKhCdd@<#HaO z#pk}RB`BFBr~6q1CtuPED0#1du3We9v-82ser4gou7M@W{rV=aE2C7oKuV};aP!LV z{hLz?$BABPM{h0*%GueXhKp^Ni(PiZ33tnp5AVO|!Mr!79eexM1R^g2{alap)2`fZ zrB0MCEMRx~J*q8jggLKHEr4$oT$-rS-9u3}R})uZ;+nFv>hf&z@}^He>kP@R(%`u& zhpLl`p%b^mE6zqHa8Y-h(fdB1VL0eV1Mi^p1@-2U2#(rOTkHE5DcC8>R@lOS%SrxH zl^eEEv4)}6f0_S0{Sl6~h9tWO@9QS8a}x&CeziPcK!y+Kv><}sE~x`@nZkxYqCO=d z^{ln0~3diP{?Qse(+NVeAdG|eb|Htp0fDyZA$`-;A&rTmU8ZM(4d`{|z{ zjt{&OCA!M~o6%Vc$hk+hd~vK2X7Xf)^lAS8u}l94m`0P`fA=azh)#4HUI!K7yj_WT z^0pK+`J!j{Ke^3ntc$@pzVw#;TqwL=mu`n=Qat!>n#H_Z4QpQnf3^5PBQy}zUS7R~s8uiO&vAK%pU}ged=F-e8=e$`=AH6&E$LP}kq{ zeDZ*{n~|?4t{SogaES2{{2ot_S+*s_ofn-Mmp-w!?Q743sJMEe3I3dloZ|)wbGE{!lBVdby9UU7`|i} zw0O`(qOKVSv2Gd48l;k>pzh_nOG;bPYFj(`5@?(254rmRw8_Fww=-#yKl{(w$3t90 z71`d39$eFb$LLD=p1MXn`*ea*ECh#&5y9IR@~*AnvI_xK@jl8=J>?9@)f!|(4|u@^ zvLuLg+bPhOEO0d)6eNJ^!_JpNiGOueFwEZDX`Qo&xF~;G)c9A^^pE<5!p$dMD7E6` z@1J;~=@Y9R_<<&T6O_xp;{J>M*r|Mh8CC8@#@i59YyP0QGkM6w`R2En-=+8T@1N66 z+XX*0IWO7_*y)sCWOUn!GELwnD-q7C9JSLIBoQ zKqW`X+S6Dsv3~ZD=3d;zgY@r^oIVofJ*JNlDP7lvJ7|`X6z4 zz$W)-2AJ<{zGz)x*~9@2IGKc;z}XGSNws7%1_ha1FcmWpi8W#vyz9UQxX73N41cAh zkglZzxzG}E%lgHEE0T<39n*|@Jh~E5ON9Ym8Ah_gFE0P{bDShTSCH&gLxl3xP535QprD0-mm3gG&56zGgR~H7v^Rg0pSMv#sd0h!NmzPX@`rV5wN;JIR-H z9_e)rI~+2SLzS;GFxDhi50Vy-$xnCnPS247kyefjDcUx@bEo$=!HVy>q}#3Q);u+M z#>P1vIk+*0=ISM;1G@A>p2?1wsnGq%n;v6b&(|KO+W3&KGsy@Q-j#VJPK~4&w#}x8 zbavjc@{LoOI=6c71u&r;nrjk1zdKZR;L?l3Hqqtc!bEXF@d>I_?pH<&t!5?L5)d_? zhz%>M9Hbm!SCDZYRMaC(C2cT6)UsdcVv!%P!&h>jEba&s+G>=oF?*tnAXGF*PzY)=4ZlcQolLDhY1$1Y^kM$tB zOfN>+k)~FHZ_&_VRM(PoLLh>wl@F+&PU%%*R)8cT(geTcMvOQ5HoCb8w^f6JxUq{0 zT&n)aCx8aBIVK$}o&qS%=h1+5%#X|c-wdh#4=`OHSvr7l0mrDA<>KUKQw@55El=_N z=hnfBHme)nEa2?L#hI`sfA!;jR8fhVpZ%^W`@>u>9?!Le>r~5k3m<{Td|HG7QK4nZFw9ZhX;2^ z)#ZAf+t23x<*N=hBrBI11MC^jAp_EW4+5zMv4+V%0ulsxQ~1oQkQa_Y($G}napSHe zMj*jDxeKxs5i6o~^9??1j(q0}{#23?;wE*2jAvcvOA;HEp7y-C7EK=1B|g(XmD36ds9MUz;PH4!jw`n}2Y#=ka5T(QvF=WPT$#XXyIcuXR+b39{H| zFQVi>%-(NmpPi!b7tPX+e%yO~r8anfay%HhU1i$KE-38sLeOVmeOpr1h`$;8BK7h1 zqtKPw>^z*N9Gck~lh)OZYxa=3taKPyv~>S2s=@Djw!W8$1 zeDY&VRB=VuLO$+#`H{wxRMbZxWOy4e>0Qf7;j>AO3d6L?Zh`w(Ua+Jy$J{i2y`t9o ze&uT|;+$Pq-|jp^QRAR-pt{#d+abK~7aZ#L26}&jUO((Eo5-EQBf@iTA56NdnfWi_ zuIaS2e0gm%EBq0D5J>N3zFbenC(nMmXxkclhmcnHJvPu(MCZ`{iy0$t{87WZn6lE8 zyvuW#rpKd->}L@?W@~=EtF&Bw7!Z>fUuJSx@>sqWC*i@v8>Ayz`l=qV9d+^ET2X_3 z>e4QN2X8qrPg~cKWVq<;(A@#7xLm28c`+yL+jKf0^X2DhC5UrhX}NOF zf&q--5oVuMa{QevD9b)peEyewcbtY%R)}H(Rn9}Y-!fZ7!%wPKcmWcm zOr=yfJH170a0G~gM?5Zc+;25Ik8)Uhs@OCeHz$A6S!%nJ>2lu1(cUq@7kfDxAL=(VlGu2$t*+IXZt2n()E>dFj}94Z_5|Z9Pik z1uP)t(&P_1P@P?#X6vYmrrcDJjP#_}QUTLb3WQNElc}t{SkAs)$Mv>fpNC%k+|e%n ztCi9koKd<}b7g(b`TB!S7WLuzyU!=MxzUTB^p;(5{EmBn{5!utjfMb>2Hb%WPV-oq zfpn*5`<_Cnylee%ep|zXI<~^&vmKnGNZK^T)r? zu~$0~@jfOP^;D9$yPiyYqzSc}Bd!54uteeHi?#C7t`wvWXT2bvDD1DJD_Q2B&TX6^ z6?Se(RDM8u#^88bzb#|pxzE%i3bsAv*PfhiUegvJKQVlKTKc(xUFCii%aYsm<5EP0 z&t4`&2NA>C62tAr59Xn&ORYM2J&&#Dh916aL+W2Zmuy)fK%f@lV+`HzmrN|hPrXeX zu6yompd`hr9LM?^OodSmW7h7-+HSQRP+&k|>7*byUtYv zC!>^<$fs5gr-q@&>*r!U$&P>VfXMH)3=kszn=SkjMV+eXHeajLdt8Dp2}w+b{7<<_ zrNot;y+Qu%@d{^t@@3o5n0QF;u7hm&==CR{dNDb>aLBx;due!XCUq`5t-`OdcY1ml zem=^j7yne{RhqjG?HFTN^_1dYA|dg~QYhZe;JP_LrWVUvEr=~#jfaY1G8iK3Gk}wY z!9X<`bG9H)nu4v?s`Bz^1<#M7_lFvL&WTY7vXviNcWgZXH&A}~Y}piUy13?sT?78x z@3&B;xs1pn;PT3c3o``@w&uHL+~Jq4JeQq;yP((GX%TfvsrQZavgi)`8J}M*hnlBiJf`hSTB_lqut1mj{1^!5Tl^x? z=bLqlaXbRTeZk*k*QL`CYYJ7pTYI z9P}1mMR~Y!z;?YJv2Ztsk+x%AnAhwC?+qx?R>^=*MPkZgWdEZujDS4i9fR>%$tIuA zAJk59D=wLoK&St3jGf8(7Typvi?L7+dEW0W_rcdy>mbC`^&by(Yzlz>DZMhF|zre5g;+9XzL&1^Q}(EGH1U= z;mQ34RGYfUz}9N{AkUbdBeM{5S~w!>u(;mEuc)psyB>-fDI?t74|FwTVuME=y`9a0 z0K$8^raCS=J3R86mE}}Dxk7-vL(=i&N-&sv#lbm+NpHEdIs0rI#C%5Fz0(dUgDi|41O zQc2d6IFq$+NE)mqBB|)!{)E4Xci%dH{!~`E{B1{{?kD=mw7Iy=@btP#!cVYrazxswBNBiNig1@}y)kl!bEqt zJf}yYvng4gn60L6Hw>Ju{qzRpe}tvFK_m06sOuX2blT$X{%$&Gx&-gxk3TiRZU&uM zf-=ineaAPbDJ&GJ`0pSS;%x&Zu|RpEVNY43HNVeTVs;#b{)7#?TWQN%+K#(`D_(hL zqnbIv#i6O2v~A1r;OhQL(ms8USE#Reu;1my<&s)7{b}yaxj*P=ecv z^q_D0zm)M>eb{;F$$zvy5Fk_SzFM~eg^7X4=-Z`zpV!+D%szGzZf)+?@X#`xMc*wW zyYh{E$D@-SF?g{YGB7ai1SKYgkZ5;}ZB+F!EzUQ-+ToR#OUJUyqRb!+bK?DRETtZw zTmRbdbm?>%biF*<#aF#u#3XxSAw-4B^l@v0xz0XR{WR<`>9ZsZx$xKj$J%I1Je61B zPNCH{V@Qr$g+zr3phfDe_*}%*w(bH~2sFr8l3363n8T#j`JtiAd@DF z0M4N=jx1Lc3y%jj<oy&EJuL;FbU? zR;qw%m@(65UgsjWM>(rmic!4W5119koM%JzRL(Bq_T&;o38Tj_b86*20-(-XO%Qp` zPSny?TnWXQn-ud05KnuSa{+T1dokj0Z_IHrEK%gOWE-`UXN)o768w~%QS?tP#WwUQ zZ^<3MpG5$^17n2Yd{St+qI)_A z1{W#juw@45uYoKeFx{?A>tUtVxT?QP@htJI^VNeGB+3D>>KRe(p)SSa-+9rS78Sw}hJvdc z0nooXRbvx3U7zQu3^&iu6I!K0k^^3D86~s#Bxm}z=Q<@sCdtT8`2N{`mEPS45_2E7 z<*KD8dn@`1U|#T3^#`hU#&Pjdg!U7%C@2v1r6f??j_*-Ilvld0`?_~4H9ABZ_4$7d z+G(P|f?(V>OMJrq59XW0AW8_xI1a!(De{H+s{q?97gywoi<@wqs}O6pWhuegDfXWb zj>ylGCkp3LK9;XZ(jizhx%j}osBC;Q2=Xy$rhGF+?DY8a`F|VJ6ps{)GnF14a{JF_ znkgt(CHQK>mWvDej|mmmv$nN`2KyNA*@w~mqa8K+jc)9&AWud#?&Uv6eW*Sc@~`Z% zjVhM^r|4}wRrOS;x;y-o9@w1Fbhe14IDOU5AEU}!Ma2#H=yC8~(#Z>LXsN4gpn4N0#Ugyyu{RS{U&h&OA@=zQ?(nrL0 z6vUBdKM_`*i^)7U9UqeFLh5`A%9I6)OWw_?rN~g3sEZd5C8SZ!5Vuny7{MGS7euNd z)Bu|&oEeW}i4qj2jo!CP!Kh37(2{)sW%c_9(+z>3P*lDQK&093Z^Ch+b+HM7&NrDK z=utiW%`{qYaUDLq9H16_+dc~%z~A(We>ZVaBNbAIK~YF}=4nxJ_7D=9To7>DAQRsd z<3#_a)+Bv8r(gAoP1Ja^aK+)0B|Xh3eYJq{z+L- zn7{Rt;!8lYMg@RQasUmZSk@jCz#noyDK}YU?jWEq-P1v(`n>P6sMj{UM zg*@F1WbgS~^58ifmQlG^{4^mfG+=-kXm zMtj_pi#?D&z%Oy?hnTJ`!2Vws2cto;S>p$PDR)yz-%)O$(&Fb`~-5S#U zBX8aU;JImk2~el$`Zk(T#S;HDCrbTtuPyeb?E7|aC%-OtX;^qQtdh8$0sxrJ#l4Qn z#H*)2(M9b{zzKS_^$=$-pc3+%`bij+2s)e_<9N2Sst>1lFDfTKa79%U^|MKrPIAr! zYL|dLSMkid2vq$fn0yWXUx12T|BrCTSyRz=*P2oGm_E!%1uc{iX=+q^4n$1k-rk57 zwP(L@g6N=~CaUrb8wTUfH~}qqVOm)u{k@gaiMi2r9GGDW4A2ykXsj;cRqY)futqtbE`^EOU^LgJ9Rs#;{(U(JntE8cYw2K#okhs+bxvx^aDG0C~+ zv}`0mq{tudwQMUKMzleT8YsPAP+XW5J(f`vA35lU{>RFj)R> z8HrtaWc4c)yytwGaU1*?t`C#e5;sYHshLY;Y6;+CXM(rC9>#;RKyyi@;aXYB1HBG% z_~fR6@n2{)W^eJD`SW;9wQ>YVrxxu4zl$4#L;C1jnGn{mof{kDsA{UD3K4Pp>=p8x zKt9B#J|NV~Eji^GGajk>AKc`^D2SNX&Z$zca_`_*^RDr^)pz>&7YncK(fk<^ZUy9X z?r{vuP;Uf#hMD$z4Mg4jADzwX{v`JQYNO^(dxBT?{dah!7`;)B&k59?fAGH{GKqT& zZK!vG3Nbn%2;i+6TyEH2(T%5>SyA{fA-X(Hsx=RZc?*i3b-~!veX?km+{4x-f8-;c zcPwopWWU2k+B=RTc_cM0Yk9?Qf+tLL4j?oyE|F`HSGb;i`6M0JAO*95JV%FBitCXR zRz|^f68wyV5T!x6zrHue4_VN5(v~m?n;2#y8X{JL>=Dn=Ns(fiHs_*$kGrCK*1U8x=)6ymF^AE~ zL~pwBWrX`gKee4_GX`tcZEGj2NH3bBZu%OS{BMiHIc`NFXWU6Vn;GYN8B_{-r>4qr z0L&aLF!{e?K6n2M^O^jWNStks&N5JVY`iP}W8F^R;HWV}hgN{51(IgLjn~arO-0{4 zDe=`|ck()-ob$H#OD%GQ{RM_(#EFM#lpcT%N>1VGjiWF-~fmWvy~{x3AUm+1fi literal 0 HcmV?d00001 From 115ee42390a58fd981490210d01f81638ba362f4 Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Thu, 21 Nov 2024 23:27:53 +0800 Subject: [PATCH 7/9] doc: graph: add complex fusion list to website --- doc/graph/rst/graph_complex_fusions.rst | 9 +++++++++ doc/rst/graph_extension.rst | 1 + 2 files changed, 10 insertions(+) create mode 100644 doc/graph/rst/graph_complex_fusions.rst diff --git a/doc/graph/rst/graph_complex_fusions.rst b/doc/graph/rst/graph_complex_fusions.rst new file mode 100644 index 00000000000..d3b15384359 --- /dev/null +++ b/doc/graph/rst/graph_complex_fusions.rst @@ -0,0 +1,9 @@ +Complex Fusions +############### + +.. toctree:: + :maxdepth: 1 + + dev_guide_graph_sdpa + dev_guide_graph_gqa + dev_guide_graph_gated_mlp diff --git a/doc/rst/graph_extension.rst b/doc/rst/graph_extension.rst index 5d835236442..be81c4837b9 100644 --- a/doc/rst/graph_extension.rst +++ b/doc/rst/graph_extension.rst @@ -6,6 +6,7 @@ Graph Extension graph_programming_model graph_supported_operations + graph_complex_fusions dev_guide_graph_fusion_patterns dev_guide_graph_dump dev_guide_constant_tensor_cache From f4b3a24594c98be338e31803507dbc4b6db68008 Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Wed, 4 Dec 2024 17:49:22 +0800 Subject: [PATCH 8/9] doc: graph: re-structure the content --- .../fusion_patterns.md} | 11 ++++++----- .../gated_mlp.md | 0 .../{complex_fusion => fusion_patterns}/gqa.md | 0 .../images/fp-gated-mlp.png | Bin .../images/gated-mlp-swish.png | Bin .../images/gqa.png | Bin .../images/sdpa-mask-1.png | Bin .../images/sdpa-mask-2.png | Bin .../images/sdpa-reorder.png | Bin .../images/sdpa.png | Bin .../{complex_fusion => fusion_patterns}/sdpa.md | 0 doc/graph/rst/graph_complex_fusions.rst | 9 --------- doc/rst/graph_extension.rst | 1 - 13 files changed, 6 insertions(+), 15 deletions(-) rename doc/graph/{supported_patterns.md => fusion_patterns/fusion_patterns.md} (97%) rename doc/graph/{complex_fusion => fusion_patterns}/gated_mlp.md (100%) rename doc/graph/{complex_fusion => fusion_patterns}/gqa.md (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/fp-gated-mlp.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/gated-mlp-swish.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/gqa.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/sdpa-mask-1.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/sdpa-mask-2.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/sdpa-reorder.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/images/sdpa.png (100%) rename doc/graph/{complex_fusion => fusion_patterns}/sdpa.md (100%) delete mode 100644 doc/graph/rst/graph_complex_fusions.rst diff --git a/doc/graph/supported_patterns.md b/doc/graph/fusion_patterns/fusion_patterns.md similarity index 97% rename from doc/graph/supported_patterns.md rename to doc/graph/fusion_patterns/fusion_patterns.md index 6118a088929..7ae0ae38e34 100644 --- a/doc/graph/supported_patterns.md +++ b/doc/graph/fusion_patterns/fusion_patterns.md @@ -1,8 +1,7 @@ -Supported Fusion Patterns {#dev_guide_graph_fusion_patterns} -============================================================ +Fusion Patterns {#dev_guide_graph_fusion_patterns} +================================================== -@anchor fusion_patterns -## Fusion Patterns +## Overview The following fusion patterns are subgraphs that the oneDNN Graph API recognizes as candidate for fusion. The patterns are described using oneDNN Graph @@ -72,6 +71,9 @@ ReduceProd | ReduceSum] | Pattern | Description | |:--------|:-----------------------------| +| Scaled Dot-Product Attention | Refer to @ref dev_guide_graph_sdpa for more details. | +| Grouped Query Attention | Refer to @ref dev_guide_graph_gqa for more details. | +| Gated Multi-Layer Perceptron (Gated-MLP) | Refer to @ref dev_guide_graph_gated_mlp for more details. | | Convolution + BiasAdd\f$^?\f$ + BatchNormInference\f$^?\f$ + [Unary \| Binary]\f$^{0-3}\f$\f$_{>out}\f$ | This pattern is widely used in Convolution Neural Networks, for example ResNet, ResNext, SSD, etc. | | ConvTranspose + BiasAdd\f$^?\f$ + [Unary \| Binary]\f$^{0-3}\f$\f$_{>out}\f$ | This pattern is widely used in Generative Adversarial Networks. | | Interpolate + [Unary \| Binary]\f$^{0-3}\f$\f$_{>out}\f$ | This pattern is widely used for image processing. | @@ -83,7 +85,6 @@ ReduceProd | ReduceSum] | BatchNormInference + ReLU\f$_{>out}\f$ | This pattern is widely used in Convolution Neural Networks, for example DenseNet. | | Reciprocal + Multiply\f$_{>out}\f$ | N/A | | Reorder + Add\f$_{>out}\f$ | N/A | -| Scaled Dot-Product Attention | Refer to @ref dev_guide_graph_sdpa for more details. | #### Quantized Patterns diff --git a/doc/graph/complex_fusion/gated_mlp.md b/doc/graph/fusion_patterns/gated_mlp.md similarity index 100% rename from doc/graph/complex_fusion/gated_mlp.md rename to doc/graph/fusion_patterns/gated_mlp.md diff --git a/doc/graph/complex_fusion/gqa.md b/doc/graph/fusion_patterns/gqa.md similarity index 100% rename from doc/graph/complex_fusion/gqa.md rename to doc/graph/fusion_patterns/gqa.md diff --git a/doc/graph/complex_fusion/images/fp-gated-mlp.png b/doc/graph/fusion_patterns/images/fp-gated-mlp.png similarity index 100% rename from doc/graph/complex_fusion/images/fp-gated-mlp.png rename to doc/graph/fusion_patterns/images/fp-gated-mlp.png diff --git a/doc/graph/complex_fusion/images/gated-mlp-swish.png b/doc/graph/fusion_patterns/images/gated-mlp-swish.png similarity index 100% rename from doc/graph/complex_fusion/images/gated-mlp-swish.png rename to doc/graph/fusion_patterns/images/gated-mlp-swish.png diff --git a/doc/graph/complex_fusion/images/gqa.png b/doc/graph/fusion_patterns/images/gqa.png similarity index 100% rename from doc/graph/complex_fusion/images/gqa.png rename to doc/graph/fusion_patterns/images/gqa.png diff --git a/doc/graph/complex_fusion/images/sdpa-mask-1.png b/doc/graph/fusion_patterns/images/sdpa-mask-1.png similarity index 100% rename from doc/graph/complex_fusion/images/sdpa-mask-1.png rename to doc/graph/fusion_patterns/images/sdpa-mask-1.png diff --git a/doc/graph/complex_fusion/images/sdpa-mask-2.png b/doc/graph/fusion_patterns/images/sdpa-mask-2.png similarity index 100% rename from doc/graph/complex_fusion/images/sdpa-mask-2.png rename to doc/graph/fusion_patterns/images/sdpa-mask-2.png diff --git a/doc/graph/complex_fusion/images/sdpa-reorder.png b/doc/graph/fusion_patterns/images/sdpa-reorder.png similarity index 100% rename from doc/graph/complex_fusion/images/sdpa-reorder.png rename to doc/graph/fusion_patterns/images/sdpa-reorder.png diff --git a/doc/graph/complex_fusion/images/sdpa.png b/doc/graph/fusion_patterns/images/sdpa.png similarity index 100% rename from doc/graph/complex_fusion/images/sdpa.png rename to doc/graph/fusion_patterns/images/sdpa.png diff --git a/doc/graph/complex_fusion/sdpa.md b/doc/graph/fusion_patterns/sdpa.md similarity index 100% rename from doc/graph/complex_fusion/sdpa.md rename to doc/graph/fusion_patterns/sdpa.md diff --git a/doc/graph/rst/graph_complex_fusions.rst b/doc/graph/rst/graph_complex_fusions.rst deleted file mode 100644 index d3b15384359..00000000000 --- a/doc/graph/rst/graph_complex_fusions.rst +++ /dev/null @@ -1,9 +0,0 @@ -Complex Fusions -############### - -.. toctree:: - :maxdepth: 1 - - dev_guide_graph_sdpa - dev_guide_graph_gqa - dev_guide_graph_gated_mlp diff --git a/doc/rst/graph_extension.rst b/doc/rst/graph_extension.rst index be81c4837b9..5d835236442 100644 --- a/doc/rst/graph_extension.rst +++ b/doc/rst/graph_extension.rst @@ -6,7 +6,6 @@ Graph Extension graph_programming_model graph_supported_operations - graph_complex_fusions dev_guide_graph_fusion_patterns dev_guide_graph_dump dev_guide_constant_tensor_cache From 7de77da6832c5f2bdacc896d184593ee640bc077 Mon Sep 17 00:00:00 2001 From: "Lv, Tao A" Date: Mon, 9 Dec 2024 11:09:58 +0800 Subject: [PATCH 9/9] doc: graph: patterns: fix words and remove legacy gc patterns --- doc/graph/fusion_patterns/fusion_patterns.md | 63 ++------------------ 1 file changed, 5 insertions(+), 58 deletions(-) diff --git a/doc/graph/fusion_patterns/fusion_patterns.md b/doc/graph/fusion_patterns/fusion_patterns.md index 7ae0ae38e34..b39374e1571 100644 --- a/doc/graph/fusion_patterns/fusion_patterns.md +++ b/doc/graph/fusion_patterns/fusion_patterns.md @@ -4,13 +4,13 @@ Fusion Patterns {#dev_guide_graph_fusion_patterns} ## Overview The following fusion patterns are subgraphs that the oneDNN Graph API recognizes -as candidate for fusion. The patterns are described using oneDNN Graph +as candidates for fusion. The patterns are described using oneDNN Graph operation (op) names with the following convention. @note oneDNN Graph performs limited input validation to minimize the performance overheads. The application is responsible for sanitizing inputs passed to the -library. For large u8 or s8 inputs may lead to accumulator overflow, you can use -floating point patterns instead of quantized patterns. +library. Because large `u8` or `s8` inputs may lead to accumulator overflow, you +can use floating-point patterns instead of quantized patterns. `"+"` describes a chain of two ops. The preceding op produces an output tensor, which is consumed by the following op as its first operand. @@ -35,7 +35,7 @@ the producer and consumer relation within one graph partition. For example, A\f$_{>t1}\f$+B+C\f$_{"` refers to the output -tensor, and `"<"` for input tensor. Input and output tensor between neighbor +tensor, and `"<"` for input tensor. Input and output tensors between neighbor ops are not explicitly marked, for example, B consumes t1 implicitly in the example above. @@ -55,7 +55,7 @@ inputs. In the example A\f$_{>t1}\f$+B+C\f$_{out}\f$ | N/A | | ReLUBackward + BatchNormTrainingBackward\f$_{>out}\f$ |N/A | - -All the above fusion patterns are supported by default. - -## Aggressive Fusion Patterns -Aggressive fusion patterns also follow the pattern description convention -defined in the [Fusion Patterns](@ref fusion_patterns) section. - -@note Aggressive fusion patterns are only supported when -[Graph Compiler](@ref dev_guide_graph_compiler) is enabled. - -The following categories will also be used to describe aggressive fusion -patterns. - -- ReshapeTranspose = [StaticReshape + StaticTranspose\f$^{1-2}\f$] - -- Activation = [ReLU \| Sigmoid \| GELU] - -- ActivationBackward = [ReLUBackward \| SigmoidBackward \| GELUBackward] - -### Inference - -#### Floating Point Patterns - -| Pattern | Description | -|:--------|:-----------------------------| -| MatMul + [Multiply \| Divide] + Add + Softmax + MatMul + StaticTranspose + Reorder\f$_{>out}\f$ | Multi-head Attention. This pattern is widely used in models containing encoder-decoder structures, for example BERT. | -| ReshapeTranspose\f$_{>t1}\f$, ReshapeTranspose\f$_{>t2}\f$, ReshapeTranspose + MatMul\f$_{out}\f$ | Multi-head Attention. | -| MatMul + Activation\f$_{>t1}\f$, [MatMul\f$_{t1}\f$]\f$^{0-4}\f$, MatMul\f$_{out}\f$ | Multi-layer Perceptron. This pattern is widely used in recommendation models, for example DLRM. | -| [Convolution + BiasAdd\f$^{?}\f$ + ReLU]\f$^{1-3}\f$ + Convolution + BiasAdd\f$^{?}\f$ + Add + ReLU\f$_{>out}\f$ | Identical Bottleneck. Enabled only in single thread runtime scenario. This pattern is widely used in Convolution Neural Networks, for example ResNet. | -| Convolution + BiasAdd\f$^{?}\f$\f$_{>t1}\f$, [Convolution + BiasAdd\f$^{?}\f$ + ReLU]\f$^{1-3}\f$ + Convolution + BiasAdd\f$^{?}\f$ + Add\f$_{out}\f$ | Convolutional Bottleneck. Enabled only in single thread runtime scenario. This pattern is widely used in Convolution Neural Networks, for example ResNet. | - -#### Quantized Patterns - -| Pattern | Description | -|:--------|:-----------------------------| -| Dequantize\f$_{>t1}\f$, Dequantize\f$_{>t2}\f$, Dequantize + MatMul\f$_{out}\f$ | Quantized Multi-head Attention. | -| Dequantize + ReshapeTranspose\f$_{>t1}\f$, Dequantize + ReshapeTranspose\f$_{>t2}\f$, Dequantize + MatMul\f$_{out}\f$ | Quantized Multi-head Attention. | -| Dequantize\f$_{>t1}\f$, Dequantize + MatMul\f$_{t2}\f$, [Dequantize\f$_{>t3}\f$, Dequantize\f$_{t2}\f$]\f$^{0-4}\f$, Dequantize\f$_{>t4}\f$, Dequantize\f$_{out}\f$ | Quantized Multi-layer Perceptron. | -| Dequantize\f$_{>t2}\f$, Dequantize\f$_{>t3}\f$, [Dequantize\f$_{>t1}\f$, Dequantize + Convolution\f$_{out}\f$ | Quantized Identical Bottleneck. Enabled only in single thread runtime scenario. | -| [Dequantize\f$_{>t1}\f$, Dequantize + Convolution\f$_{t2}\f$, Dequantize\f$_{>t4}\f$, [Dequantize\f$_{>t3}\f$, Dequantize + Convolution\f$_{out}\f$ | Quantized Convolutional Bottleneck. Enabled only in single thread runtime scenario. | - -### Training - -| Pattern | Description | -|:--------|:-----------------------------| -| Dequantize\f$_{>t1}\f$, Dequantize\f$_{>t2}\f$, Dequantize + MatMul\f$_{out}\f$ | Multi-head Attention Training Forward Pattern. | -| StaticReshape + StaticTranspose\f$_{>t1}\f$ + MatMul + Multiply\f$_{>t2}\f$ + Subtract\f$_{t4}\f$ + MatMul\f$_{>out1}\f$, Multiply\f$_{t3}\f$, MatMul\f$_{out2}\f$, MatMul\f$_{out3}\f$ | Multi-head Attention Training Backward Pattern. | -| MatMul\f$_{>out1}\f$ + Activation\f$_{>t1,>out2}\f$, [MatMul\f$_{out3}\f$ + Activation\f$_{>t1,>out4}\f$]\f$^{0-4}\f$, MatMul\f$_{out5}\f$ + Activation\f$_{>out6}\f$ | Multi-layer Perceptron Training Forward Pattern. | -| StaticTranspose\f$^{?}\f$\f$_{>t0}\f$, ActivationBackward\f$_{>t2}\f$ + MatMul\f$_{t1}\f$, ReduceSum\f$^{?}\f$\f$_{out1}\f$, StaticTranspose\f$^{?}\f$ + MatMul\f$_{out2}\f$, [StaticTranspose\f$^{?}\f$\f$_{>t3}\f$, ActivationBackward\f$_{>t4,t1}\f$, ReduceSum\f$^{?}\f$\f$_{out3}\f$, StaticTranspose\f$^{?}\f$ + MatMul\f$_{out4}\f$]\f$^{0-4}\f$, StaticTranspose\f$^{?}\f$\f$_{>t5}\f$, ActivationBackward\f$_{>t6,out5}\f$, ReduceSum\f$^{?}\f$\f$_{out6}\f$, StaticTranspose\f$^{?}\f$ + MatMul\f$_{out7}\f$ | Multi-layer Perceptron Training Backward Pattern. | -| Convolution\f$_{>out1}\f$ + BatchNormForwardTraining\f$_{>out2}\f$ + ReLU\f$_{>out3}\f$ + Convolution\f$_{>out4}\f$ + BatchNormForwardTraining\f$_{>out5}\f$ + ReLU\f$_{>out6}\f$ + Convolution\f$_{>out7}\f$ + BatchNormForwardTraining\f$_{>out8}\f$ + Add + ReLU\f$_{>out9}\f$ | Identical Bottleneck Training Forward Pattern. | -| Convolution\f$_{>out1}\f$ + BatchNormForwardTraining\f$_{>t1,>out2}\f$, Convolution\f$_{>out3}\f$ + BatchNormForwardTraining\f$_{>out4}\f$ + ReLU\f$_{>out5}\f$ + Convolution\f$_{>out6}\f$ + BatchNormForwardTraining\f$_{>out7}\f$ + ReLU\f$_{>out8}\f$ + Convolution\f$_{>out9}\f$ + BatchNormForwardTraining\f$_{>out10}\f$ + Add\f$_{out11}\f$ | Convolutional Bottleneck Training Forward Pattern. | -| ReLUBackward\f$_{>t1}\f$ + BatchNormTrainingBackward\f$_{>t2,>out1}\f$ + ConvolutionBackwardData + ReLUBackward + BatchNormTrainingBackward\f$_{>t3,>out2}\f$ + ConvolutionBackwardData + ReLUBackward + BatchNormTrainingBackward\f$_{>t4,>out3}\f$ + ConvolutionBackwardData + Add\f$_{out4}\f$, ConvolutionBackwardWeights\f$_{out5}\f$, ConvolutionBackwardWeights\f$_{out6}\f$, ConvolutionBackwardWeights\f$_{out7}\f$ | Identical Bottleneck Training Backward Pattern. | -| ReLUBackward\f$_{>t1}\f$ + BatchNormTrainingBackward\f$_{>t2,>out1}\f$ + ConvolutionBackwardData + ReLUBackward + BatchNormTrainingBackward\f$_{>t3,>out2}\f$ + ConvolutionBackwardData + ReLUBackward + BatchNormTrainingBackward\f$_{>t4,>out3}\f$ + ConvolutionBackwardData + Add\f$_{out4}\f$, BatchNormTrainingBackward\f$_{t5,>out5}\f$ + ConvolutionBackwardData\f$_{>t6}\f$, ConvolutionBackwardWeights\f$_{out6}\f$, ConvolutionBackwardWeights\f$_{out7}\f$, ConvolutionBackwardWeights\f$_{out8}\f$, ConvolutionBackwardWeights\f$_{out9}\f$ | Convolutional Bottleneck Training Backward Pattern. |