From dcbf84c74a6f373c5f895ea2a307c51b251b784b Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Wed, 12 Aug 2026 08:25:43 +0000 Subject: [PATCH 01/15] Add Wan FP8 quantization with Sol-Attn Add tf-kernel and TorchAO online FP8 choices for Wan2.1 transformer blocks, with dense and Sol-Attn runnable examples. Keep active Sol-Attn Q/K/V in BF16 while preserving Wan residual dtypes, and cover the quantization and attention interaction with focused tests. Document the H100 ablation setup and results, including a peak-memory bar and throughput line benchmark chart. Verified with 18 focused pytest cases (including the H100 Sol-Attn kernel), both example --help entry points, pre-commit hooks, and git diff --check. --- examples/wan_video/README.md | 92 +++++++ .../assets/wan21_fp8_sol_h100_benchmark.png | Bin 0 -> 85923 bytes .../wan21_1_3b_text_to_video_fp8_h100.py | 112 ++++++++ .../wan21_1_3b_text_to_video_sol_fp8_h100.py | 257 ++++++++++++++++++ telefuser/models/wan_video_dit.py | 73 ++++- .../models/test_wan_video_sol_attention.py | 29 ++ .../wan_video/test_sol_quant_example.py | 139 ++++++++++ 7 files changed, 701 insertions(+), 1 deletion(-) create mode 100644 examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png create mode 100644 examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py create mode 100644 examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py create mode 100644 tests/unit/pipelines/wan_video/test_sol_quant_example.py diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 9a7e03f..a4f719c 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -90,6 +90,35 @@ python examples/wan_video/wan21_1_3b_text_to_video_h100.py --resolution 480p --a - Video Frame Interpolation (VFI) with RIFE model for 30fps output - CFG parallel when cfg_scale > 1 +#### wan21_1_3b_text_to_video_fp8_h100.py + +Wan2.1 1.3B T2V with dense attention and online FP8 Linear layers. +The VAE and text encoder remain BF16; only DiT transformer-block Linear layers are +quantized. `tf-kernel-fp8` selects TeleFuser's dynamic W8A8 FP8 GEMM path; +`torchao-fp8` selects TorchAO's dynamic-activation/FP8-weight implementation. +Both paths keep BF16 outputs compatible with the rest of the Wan pipeline. + +```bash +python examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --quantization tf-kernel-fp8 \ + --resolution 480p \ + --output wan21_fp8.mp4 +``` + +To use TorchAO instead: + +```bash +python examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --quantization torchao-fp8 \ + --output wan21_torchao_fp8.mp4 +``` + +Use `--quantization none` with the same example to run the BF16 dense baseline. +The final log reports generation time, generated frames per second, and peak CUDA +allocated/reserved memory. + #### wan21_1_3b_text_to_video_hf.py T2V with HuggingFace format loading. @@ -174,6 +203,69 @@ pipe_config.dit_config.attention_config = AttentionConfig.sol_attention() Sol-Attn is built into TeleFuser. Eligible BF16 self-attention calls use the sparse kernel; unsupported calls automatically use the existing dense fallback. The defaults follow the official Wan2.1 profile: Morton3D token ordering, dense layer 0, and 10 dense warm-up steps for the standard 50-step schedule. +#### wan21_1_3b_text_to_video_sol_fp8_h100.py + +Combines Sol-Attn with online DiT quantization. The default `tf-kernel-fp8` mode +quantizes transformer Linear layers while preserving BF16 Q/K/V tensors required +by Sol-Attn. Dense warm-up calls and unsupported cross-attention calls use the +normal dense attention fallback. + +```bash +python examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --quantization tf-kernel-fp8 \ + --dense-timesteps 10 \ + --dense-layers 1 \ + --tau 1.0 \ + --threshold-type diag \ + --kv-splits auto \ + --output wan21_sol_fp8.mp4 +``` + +Replace `--quantization tf-kernel-fp8` with `--quantization torchao-fp8` to use +TorchAO FP8 with the same Sol-Attn configuration. + +For ablations, use `--quantization none` for BF16 + Sol-Attn and use the standalone +FP8 example for FP8 + dense attention. Compare only runs with the same prompt, +seed, resolution, frame count, inference steps, and kernel warm-up policy. + +##### H100 benchmark + +![Wan2.1 FP8 and Sol-Attn benchmark](assets/wan21_fp8_sol_h100_benchmark.png) + +This generation cold-start benchmark runs each configuration in a separate process +on one H100 80GB. It uses the official Wan2.1 T2V-1.3B example prompt, `832x480`, +81 frames, 50 UniPC steps, CFG 5.0, sigma shift 5.0, and seed 42. Generation timing +starts after pipeline loading, so it includes first-execution kernel/JIT costs but +excludes model loading. Peak memory is `torch.cuda.max_memory_allocated()` over the +same generation interval. + +| Quantization | Attention | Throughput (frames/s) | Peak allocated (GiB) | +| --- | --- | ---: | ---: | +| BF16 | Dense | 0.854 | 16.147 | +| BF16 | Sol-Attn | 1.083 | 17.023 | +| TorchAO FP8 | Dense | 0.674 | 17.601 | +| TorchAO FP8 | Sol-Attn | 0.836 | 18.193 | +| tf-kernel FP8 | Dense | 0.865 | 14.855 | +| tf-kernel FP8 | Sol-Attn | 1.167 | 15.730 | + +The benchmark prompt is: + +> Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely +> on a spotlighted stage. + +Example command (replace the quantization and example script for each ablation): + +```bash +python examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ + --quantization tf-kernel-fp8 \ + --width 832 --height 480 \ + --num-frames 81 --num-inference-steps 50 \ + --sample-solver unipc --cfg-scale 5.0 --sigma-shift 5.0 --seed 42 +``` + #### wan21_1_3b_text_to_video_cache_calibrate.py Calibration tool for AdaTaylorCache. diff --git a/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png b/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png new file mode 100644 index 0000000000000000000000000000000000000000..a8bf389ebc2fa0f566b782513260fe6a7a2ef9ff GIT binary patch literal 85923 zcmeFZ`8(9>A3yG#ib~Q-2qzUqNVe=n2-){NWH)4A#!?BDkYwMNv5hhIZ7@a1zKw0h z*w?Wy!x)V3?VRuX^B;VF`Cgyb^}4#IF6On|_x*f6w!6<7YKn9hnJ-dNQPC;Ae4#}} zb^a|C)mi8Be}jLK9%#v=`iqK6>BVy$AM7gL+W&?k`_HY~aOkJ&^xvs|;qiDQqX!i_ z8JGv^_yQCUjJgGnH~stlvuod}uFk9n)-Ja6&;G&B^R>(WgWllc7g>=IL{+0sQkg!R z@G*i5{CD)JDPQ~FXDXeDcc(Y=@A2;PS)KoVrtf=LIUN%B$($-cHZJ`bsuO zM{5#BYl@1BJ{*1cKA^?JqhicrT)j}t`u7=>$NE$MQ4LyJ_wV6n(RtR^w&`L}>8GRt zB8zCYm8E-``}Xz@OsfDE5)uLnfx1^S1O`H|BziWXjF5Ep_K;AwWw$@?gTwf#&A&u` z?(XS%$iq`!Qu5Z*b!Rx6}8vX*$EdalRNwgNgiSHHx_kPU>En8Q?@gB z@#2k_mzr6inZ12tZEbBxNNskumAk0^{zA8Lwe#|1ohMF&(w!vim@g#eL0Y=Llcvdj zGS;@y>+@FJnwn16Z9VFH`@uP?eQ{6`lD@UD2bMV`KY2xK4Q<=?SQTOw5>{7Ty zMft%gedo@DrJ2djkCl}~O#5{+6OxG_^iUixn)u2~SZI~>S({5Ot?`zBK~d4GfV3cr z$rHQRL2?k+j@>YR)5`*+_a5sQ*PYNI&-f{q#®5|J<)c8QHNGGGp`g!4c0vp$S8 zCCqYtWj}1LHlGOhHBz#%S%^%9RypBZm)$;@@QwH=yjh&PcmKY5jgQj#t?9P(!)j;Z zs^Csr6f18mA)oH{Xerb}Q*&tN<-Kd}?t-+mDvmrV^z`2B>>E`O>?P*W$u?X#Ei}Sj z<={U0=&{@HlEV1o<@m`)%3fQcI*$5buVPdxX$J1n+=SF+PXG4^nzh+oZ#a}8k;>a+ zTG2=75a;6J2-|sLVhj0j58lhEo}`g1Y;0sO3XU<{Ia2j{pL9tOS{i~oyM#U?CRU3+ zrnI4XMsD1=k*}_sdl>%{%l-Klnl1eI-E0B?_ohf0L>+Metw@wbMzThfPei%og5L+ z<37B*_>MEjcHf!#4EJKMc|J$R_t;oZL2-rI9QWsph3E6<=H?XSP4(F)iy_HS@g$Cq z2Bi+&VeCO>2>I=uop_`{gVo3czs#ZC(zoZ-lQ6K@u7hd*)RllqEU&Hg-8bw>^5q*0F(ll~5ary= zFB52MYir!e`mJ+bUcLpKiHV7>)idhpVtKbFb0bdNX&0hc1;51SMVOn$FK%yddrzLc zoK-UY{E8D-W4{8vuaPn{INcm7MC#aZ36YlvCwLRl(VAa=y0=BHrTZO~Zq2qXnTU`E z=6z|?)YW_QTVuJN5E2&|>DWc)4qea{6(#>1vs_4gRYeL`*DE%I3*0jgN$&3#&CZ7I zuTEh8=1#75Le3mFUa_2AK%79ge2?`>!x9@vqwLJa{#tECVu6l+XG^k?gXqcPV{y2n$JM;mN|G&M`fK8&`Dlw-2c zmZX#?d9_F^%CYFf>nkJ0^3v6tu!G+E3<26cAB?AOT>G0H21%s)Pq~Mpq8rP$jLu_K z*%tZ}XG1=k1W;Py(^F2IY;0_9mdh(Bbf+BqRAxO?rn$_Pw!iB>6yO~t&Jo{f?t8p; z6F&UyE*V$h3y)-~a=I&}1fim0z6kJ$>ZkJZm{#T$%@G%{677t+mzv}uG0x*(dpbG> zM2IPIZ&jk$@JVibf8LbI^EWW0e|z@cX%oy9dH3T&P{>x$fe=CUG@)OA{(Rb{r=vYt zj=%3&_xYISGVjqO&pl7=y=ks=c4@aFYQ`{qGpLGfc!+1+Hs{Jz=G`bs@*6d^;wmOH zKVtXHOcIUqX_>NFqaLUgFlg{PUf{*Oq&Wu98BrIQP4K;T?OJadEeGP|>8+nVy>%6J z)mWW5H(!l!!};yCJK_YSU3$*_jDe4i`44U|G6rog$Ny-hR^Kd$;j(w)=Ds)QP!VUd z&aI@XA`!4q)T%U?tHv>ZbGQhv;c3NOS$*;|GvJW^J4SV|WlH6w*ub>hy0f#>JZMEK z@zwA@JT5a$NO{TX&HE8Cva-cybae1}vOdyOokO%HD{IE3@i5vWKQ1D@r>?O)=>^7lCMCym>P$G(^9&DT9_#*vs6+#KeEzA)oAHZqCld z)pqzu7c@MfjG#&B2Bu`x@>n=_m> zDCo{agWm&*IgLu+6|Vqn=R_SOuT|rD*xq{z!WeJc&KLbs+{;eJr|;e6wKJgm40TQP z3=EXz=Rc5^-s-A#@6<0EW(ZMMRSEy}wNOtTBV1xJ$dGDUFiX+a)ZA*tg$t*JUcP)8 z*;w)R&$s7QA|kG?)p|unha1XtnbDs<{gF8_6WM5QTk)O0BAfEm*Wu*7+}%)mgb-zS ztQ^b0B>txFx;r}rUkF1WRNa&bTi3aQD8&7_xf?_=qzN7Gg9jyLWgMw2eW3O{c_LJ? zX5-{^^mF&&-7xwc7ktOB%*4;1l`JeQ%*-eV5@|)$IW$Z$MS|VqxV4*ev`5R(aBPni zMND)bNekHE4@_?S@Nx~>-*+cCD_d5HhN{NWY^6}y<>h%a#t4}Qh-IYQ+SZ`D>l z++k*{PcO^^WnNOqiY!O3<~Y5WCq99}R5-@>n4&z=BwdeI+tNCEd-ct$>g$hww`*Iq zeSJbQQ-Tfr{8@E;xVJn~-J990ctW=IHO*^YpwkU#WKI9}`1R(?h?vORqli+vI`XSc zdu>j~OZiUtD!KG`9Spo{0YhaHWa{2QWfM~q244U5Or}B8lvEzovZ^ZQ!B(^?%Sj@= z6zM3O-x1FF{YQ!k{$0m+^FR%{Z(xc3ZkaHuw83MEEW$lOGjk7%H}}sr$1TR>^}J9} zD2S@As2H#D3Mos!7j!br{wNTKZ#Y;q56qU{ne5b)JRn(V6;w#;PrTz!HSw%o028;S zrbg=U@=@b1vMGxrBK&A;WoNDXD~ z2PTg9=;<=^>3|8FosBytLcw(9KK_jZtpz2zF;pIA`3m%UN^qF>#+a?8<$#CQ>eO_$ z;n*0t%h14}#(RVEFosL3>vS;_m`)dScvMt%c)7cS(Y*dmeYxf@U%v1@>f(ve-+0#Y z`}d2=aC-=?17t%5aVB;(C1q(T;q*)xP3{+HZ-2-H%J7)jeb{{%%vwP~VHg~GacOC6 zpkaHgj#`lVet{)KtrU^JB3k8)bDf!6&dHf8T6L0;Fjl9J`s?^m`f&hgP_{Q|jSLN? zP_HMn)w?Ro%Mq4UJ*nb>HKa9pesr-b0YUPSvxdLm*xK47j8vC-Ns*VwN#xx!c;FVnu!iXQ+B*Cy7I;U}P8POu|M zEQD%grq-e5CJce|GO6)-cwbmpRJ6Cfy}u3TIQPk7wk=Y?M~XsS|7nQTxr-{}*XfF$ z^Vpd199eIORfvm^r!lW9qWBvZ{QV_5x_|ETtS$>(O#2)1qx>8*Ihe|RqwXr=6=!GH z>G2hHGl;Njc;_Q5q=n>aW#+rwQphfLnN^znyWSSl7PPl@D9DzQ78gh9nW5ibm&g!* zC?$1QN@{js{@Z<(6F;wC^^9r88IXCh%!K+a8LzFdbh}k=eXF~-gsmJL{N|?84r}^` zO+Nqx`ISw|ZEyiWh^cbIn|VHyH|9q*hhBCbE;78{rEP1=#lo@z%D8Gw3SDLFh1piS zOZn&T0E9$$7`WLH+cz0Usr8=xi|G7#U|?WyU?97Aaum#%D_K@;dpqg={#{Ga%+UFQ zuyC!r`ik?l`S}LMu>9quiFy^-z{40$yVZV+&MMYeOE%9sw>vNN^z=s3j)!d71RZ|! z%LI6*n>HUG8_P&-eC9_})Mk-KLwzO6-dCk3%!Y99a=kLYe*Kyx>J6Vyn*<}STw3Z1 zFup;lRpDJLtJz34iBqtS-bRN+^(5-%8Kx%|OvjImch&P%S&#B~1^?;ll62dxA(H^E z!x|VPjmnD$NQ1UL_c=LhQKBO?Rr4psejCA2@CnP_#$cgHv=?y(>&fC+UXz$;c=u#Y z(8*zCxwU&~WWwjLsYRvm?)B?2*RIE?rHkzkiOST2>z1?oj=qa!rh{IETW1cG>%6t1 zrix%s9XMm>R#8QU>E*q+`bd*?uuApkw_j(7MvqC~$($)Ew};RKL~&DAmM4oy#BIT? zrPAxd#&R$J3<+G%)v1%w(={?8ukFk5QZ{3-c~t;|b~5k^sX1t=8eo!{@6}W-)OxLB zwdJASZngd^d#OW4(*gX#yuAGCa!|*Gg@v^0$} z%L%i6qr)udGes4F{3#TumX?sGjoRS(S25|Y<_{`}Vr zy^QZx5R%~j8QvXLM*b82E8fy-_EJr(U!&znDgx*2P`{NRWHFxec^1>AucPBMunlRG z%CrbNB*;_(04C_#5e!x>!FFB+S)W+JWT7>!oUqoNnD#V3axHrh1yScBbeT>3N#MZl+>aj$ zftqzLA45a!Zv$Gy0tvTEzQMwxuc4vg!sGJc_U+1HTxWZGflzC2ulG@n0KwfESGv-E z(3>V6AR;A|T~Q&lu6OiB*D<#{0T$SKCsvtJdh#{B3t{3xCt-b>VIH-P7YcRTz`s)tKG_>` zcc3wE+@3qK<&iS5udv2Nica_;E!^{38B4{RBABRzM1YDYBnJ8WNVw{Yy8uZGwh_=% z5Mtoxu|b%dTUuO%j$Hr1Mmu3I`0i-d!ZUIgxj9655x{yUmsjIfVMg^$-U<5$fk)argIE>Fp zB{k0OY`zVepJ#MLBT`bvdGzHb{cX24f{tLWp)|Sb0vOkZa#Az5rM+&r`b!@NiaQ{G z5qnu4G8gD*N1je}t2)Z5dwEq6QUfpC0WclL7ftY(SJk-blDoPpgyB7WyCuPVMu&!; zHBoqRi!Fef`m7g|kOxL$BOoEAXj*k_1$l&2+;^t)si*S^=ZI;i?02w`UufSDamE&w zLRo^BFr~q*TLt#^8x1F#`x9r{UL|ZVNQ93RPx`DGz3C=&V0Lbz)&VTCuV4MjCOiOe z1V{9%z8ltGBO`0`3$%P@M@rBE#B{9V2;>3ZaCHs>S5ok$?rOxr|2yMbLr-a-7=TR1 zBRvEJH9+p>badJQyL4LN;SaEa%VQ164rml0tQHm{;vO}F#Plk*Wt1Z_)6&IO)MxYO zI=+rn?L0rfY!880TUco7>BU7y7ua3$ZqDT;AdoKrs;@qo%8zbdd4MRMtYPwBc8l*x zb*K;RHm_`#hUy^Yj;Y(#tg(Rd=?X60b_;r)9Mbi#;qxzHn zCq2eU;hymC7TS##u$)|n+Z(H^IxSi~YHG|99coHSi0W9;M^fdaWncCBy^fPhmHpg} z)1GdaJ}q==>KazJ^YpTG$yGWl5oy`k%`_VgQeIsT&g70Zc!qE9m)*oRy(hY0{!CAv zkix49Zr;2J2(B55p`eCfWRT#!HURTRuFFx=Uv$dTmy3=#`eMqH5Ii#{LeX{PEPo&!K}je6+Tk4PoN$5 ziozZm3g^FRd1|#tBp7 zbf`^ZRYb8#Lsi|H3Z_?nSvn>id2mL{kq6ElS5{Gmv4sF)oy+;;31MG`GW@jt&!3k# z>p?7*9NsI9EqCus852}hRrPd+>2@+~p!fH-nh)u2sn#w!YjSgQ$MKpMKq&UwT#$(p zaiMz;cDE0BSwWa9&|WMQW{kJnpOfTI;hUAzAYk8 zf%;;NyG-2T>*;Cm^U}Wro>adevZg806a`+xdVjcF@X?;|YX_*GkN?3zo2?pKA>uE2)e##1>s{)sGXF2-&1!9?(H28+fZRcBbSISJ# zuG&`Fqky6fkBTmbR`MubVlQ;nf-U|P(K~mlkgZSSuvjM(l;uQB6x7!*DJh9eCi|C= zAQ10%Y*PQoLoP0TW5rKE`MoV>d27C}#36AC>kwz>gjF%tem$7%;Jq6wRhv5CSGXa1g1)Mw(QLyL3Tfloz#

<(X@6M&o@YJIFI zYw&Q)a2PshWzBd*Jzacks09$T-$O0dWK27*s%TIz#IGAm7qP~nM-ThJB1l!k1zfDY z8=kA9X0xggjXrtnW@f^1#sgHw=Ju|KdjlM+ZnSc6gFA5-cb3%pm>4kD(q*Vs6`C#_xQtb;q>rFHJze%+Yt$rSsIR<1g@*4?KJ$R7~iO{v7wz{1-mp+{kjr zhW<{v?=Pfh#x@cX<_x5F%6eEjd@0OuHWBvr=y$=LyRCsc_^1mPKKq$nQA-n-vR&6c z)$#;t%<`Dqc{`Q`ZH0Tr0CQ_&AN+;^IC==BA0@8Jq^KB1gr7igoEg-eoSd9M)gyWF zPA-dLZld^{#_M%weXMD9m`{dkeeRBEpabY)vF85dOuxJN-K%Rd2TK7FJLZ>bwbEIS zZV@Q|fo5iI`rQn9t z)YcxjG3g*FAt7IsFFHQn96uMK{xU((JIz5x94e8aYB-bZvS%aOMx?kl!4X zsbSFCAI^0UZU%{#S0hmfr?I-@)!H4Nxa0n&USyo3gs+i%Hl1$8&gP#YCYh7E7Y$Ki z0Ar`5m`F(ATHk8cnHwe~JKi=o(;K2!v@cjt`1pvImjIcU{RFV4ykdyR*6mfG=zN-% z6IR2PZ~S!UKSJ6x{Me9rYd4(IrjI1yGyvnwH2wilwQ#ZTLNmPsZmBDbP11eOKh>rJ z^o_U(M^H--I$c!Yhi-bDF%8>f9N^bqP=nNHOE~=4rJ+ih9&jvHi7I{8*~LYibV(|C z@ormW`YTlxxX1N@Kf)Lnb2*sT;ZLtZR^5r?PnZ_8ClAr@)|13Xa~-i}QqXZZMuFhj z)=N~(Tn_nEc3BlejG79D+#$w*-UR4Mn`oJv7-r9>%O zt0~K2NMSYQmoNR7V?~7&v|A%s8$ivy!`S;G#9TVys82sVXeN94>?Jzdp#9@0M|8|- zO+@%~GnN9dWrwq?E;c3HL4i(}?&r3i7!!s*TnM+2qcPSUW7M1o(_gL2 zKp00}m=(r+>l?is3*`Aa&$b57b9VWYjiJhGg))y;u$ASlTh%ky*Ad|fzy51Urx-&M zONPk=QJRXvY;$0+wpM(dBL&5N1_QMaWB0IVJu?V+>b2{_mrL z%=B6v2aMr+t2`(=4JSYEKqfOC%je?Rb)b#W?Qeis7SC^fV0-XIJ5RX6{yGaw_1wM; zBd>q4i2k*RJ528G?w|s3Gwkkeeu{|DW=~H{9GYT;G3Wo-WT zcY7O7*3&T-^!Tr(8Ia$&nzBgR{RD%QwFX~g4>HUujV+01kUAL`ROdPIbazh`@f`Sb zRW48Z@eVm6Io4*P%=7SruR6(52JKPD=%OjqNdEEi-gRzB@q^YS%f8cCm zBVylFqjbHs{Y@DlBEfuT&FgRZ3y|ED9UO*e*CpQOXJkARzBH%-Mz+K?CXOMW)4``j z#^PvH9ZfPiUOA*ou9hawjYF5bvN8h?nYkt78@VnMa}J2h#uW_Sg#_7H*e-SabY|8P?mR@Kh(UBo~S&S!i+)xEDzu`@H6 z+52m0xyj0A`I+Pl%zk{qu{_ZLcVPKl>RPQeI8fo4F(9WcwGYVuZ5h|XH2U!_%nRJT zf}91-y;D8uBwJlLaB2*08wEoj+x?6i*Yv(14e7{%1yQo4195w%NHqDo4mUmv79D1< zZNH)ND(NvF2gmlv{h)mHi*;}N-?8}3R>q4!#&XjRmzSAFI5^JWDiLE2l|y$P3=DlY zu*h6sa@t+tEr{e(R(<~ZH4adp{3vy0<>9KAfZDaTn%%pmgsM6%W~yO88uT<~GBIXZKT)e_X&iV&wm;##(=|0e7k^TvUcU$~`(MW$J8UPVPkR#w)R z1m)6PH{_ilrZz~O-ELP23-giD_w$b zaC5c$HrYu13IaK|fp&w}?i@G>M~>X$``_$76%|Z@t*K29l*S$!(|pX`2Ed5OEZ@@b zC|v?Z00aVjRfTo#UHjbHiV}rPNc8bEV534! zKZ-2dS|gc=fJP-IC238pe?CipeiJZPz2o|qN~$1AQ4~?x-MwT-g!u;q)OjtxYCL~O z0l;;EjPJ?G`URD7*tc)rZoc&%8P$D;0j|d7`~;u@8W{jEp$X=3bhM%T?6s{67mW7G zndGz7DqZpptn~~?!a&eD8oDm)1BDt+9;d<7^aIgPURqfe*e9(}iHX}u(E$GdoI{f@ zb8}%sKBDXW#y5cAPki^qgirmI^1yJe2AFHdpnFA)m9tHhd!t*uSb01T{ew(O0Wits z#a*hb0(w~^v6YAUFz{J$o|mkwrLOjMVfD&tq_4~7{x5e1Q5&j0JJ;&JGL{Kr**KX_ z{Zo2AH(L4W<&c0W%Fd4cRl`jra1{$hE%`p$jWr<+1?l#Ywmqr(?Dh^0b%EU%Plgc~ zuK9&|;VPjY?QdKf8+`Dgs?2%P;)7=1p|WK&{}EHvfU`AnEYaJR3`mOQ|DlIf$4vw{crwSFdsofovehqzyiL4IlHSYv>neSIgRtVyFkO!NM0;7rz^>vi<6<&S=8 z7U0`I<2`XN$*uQ338Z6$0$plsH{P#Fc-xQff^17WYlw2?@Q<$%5o>E}Ky2ULy#(pz zGaTWw8APhPM9k_J8D&GFZ}N;d)UThba&)w{Rf*2mj$>fJn%USNT{7g1f#8a3hop&= z7@%S0f2iXBq@E&Y=-#VCRkYC2iyYD)88zEQ?hZ>{c+((;($n-#p z=g{PWe%9Lddd8l8Pr%!j;o-kyf9dBVYU|6&o&@L_ceb#6_EUjG6o-ZazkBHEGSIx4 zt*C*Q#VX-pSHW0Het7d=ApaZ6N2MAc6O)ZZMv;ApW~suVW?-P}#*fE~5fXM0toc9e z>`PGfCOgU>K#RO=|KhJxzf6%WV5V<6n-BwD0Gcur1ImBVRtK80M}r=IPY42*sOZ>? zQQn2(j$gklHZ%Sr%+g-G_~gkGVdTnBE* zz|;1@!R#*`JVPL9AVXPIfo$lerlvZSZ@)i7w!aOGmN8%s7f;vS@Yn*nMB#AZYXkaM z$yu&PF6a{`R#t8Mxn4Ad&09rI#TCSGjt_|ii?O!W*7lNg>AveNoN0M%Z0pSYjKF_j zMn3lS^-T{th*;o>QlyhvZ}fK2B=4@nT#v@w7vq6~Ax)W#g$wJ^2#mUTd-VXJQrK>E zDK9P18Q37=JXe9T>(x7P%*=J}ENRkKe-((MQqVmnuT&sGws&-ZhG=hJ;7Umtefr;! z1d~A>H4E5D1>&djU7+v)`Lne3K*Do115r+HeR~z2uJ!8_G^m}HRePyQ9W703C#P`` zD2a&iTPm&T(~@iH=~0`FS7i+5F9>SC<;_D0k>?jp^H6#9Dv2^x1i+e*wCT~oA-{Hh z*;{@t67bvL0y{qrYbE?@O+JE%hxe8u9!i%Wxt57+JP2&-!C$=^&BT-lYBt~lAj9?O z2-`8_Xxrt#d%$SU$Im}flcsq$bi_^l%@achBW&X_4SV35^*97hxW@`KWRP$$lCE|< zblb^Rmkt6x0rff4M-Lu6;N=A&i`f;IpcJY7heXcIk5>Q+i0bnDBv{RMX=$<=sE}Yz z(9-6~2tn8C==EDzk#HM912x~p%55EvqzVUP<&tgo_E;tCq@&R!0uH2v zfa>r-M#h{o24v5GlIPhpM+4Yr-_w(f_s5P^z+vN6J~{1&(Wi7A7(QGer~xux-~^7z z$fW;%!@f*$Si|@BHz3$zwmaa6_oMTf>*qorkx_W)zM?5QIxf!4$jAt^;G{UORZ=_n zCGVc`JRwRXi3IwC?*tfZJZk?YfnbcP>*(yPhKy(=>ly3cP7gRB&VS><`n?dv%tSm7 z$gL6q@Ao>VOo{8@Fs7mc%z5yqYI3z&%J#cf!b90lVK%RjTa81_@BEYPdyBDX7Ot?FCVN{fOQ@P;lMzE)!I#85tOQ{B}RboB20_pN_nI z`Lj|xbZ`C8vz?ug?jf8p+{!722Wc-n?3>ZH(Ak0w6?s|QzbV7Q&HMv@mD38V;Q=26Je+9QdsmR%&E0dJE0?XKT!;G@ zD3tU)DcKW^ia^&|aara&p->R%elr4)`OI~_NbXljPt|ZkHn>ProYxYOda$Fz0mMUu z8*hwfS3l4y*zFf9m!|B}5|w~k2D+N~2AQA(!W!+`dBA8g#h=SlPxUx4Efsa10QI1a zy-{>0lg5*;(Q$tQ7-i|XL(TynJw+WO9K6UJq{)R;NyN7Cf(x4Lw4HN-+N53}zZX;7 z1Og%P=1}c)vDB2?Ah2MZtn%E{{5K)*KSS{PDPyxI>j?CZzW*W{-@_`rC*nFXBURyS zIditVyQ8hOwV`AwR{A)MT)iUcGrH8`>WnyW7>cO=j?Uim?3arka z6e;9Zoqz(GUO4m;Th7+yxsIW}Klv*J@72lzZV**tF_FrvdhSo6pg{Pm(opX7krDqr z@|9EFG%2b&9v-eTw4-y?LjVNFK!B0Zg7O|@j`Hn&l5}($fv9b z8+$U-I40$)ryi|-@FkXHf*!FtTv`5HkJx{>;jpTx$ezZ=rV@XoGpo?6DY(&i>_>-H zh*yTC2W?vb(}MZ!eBYnnzvtRE8VglJ%C9zuNlSaHLH4FWR}^^igbXsO2n2$d614q$ zOGHE@qUy1Mk--sp$6MD#XQQ8IFPGO?crO`Wt&Rjz%yKz3E$vRG9F1d2g{$^_&p@)U zqsXhyE^_|-3|(=WHfoP8McQ-s54I}SXasb}NB?$5E4gRqo@tMmLeKrc33{w(wdkZG z0(JXU!#{s+Pe-4uPEBR>no`Qpf&0WQPPB#$$XXyvH_qyU4g(ZxE<{CXMTP%PWjy-m zXts{>uK&*;yZ)v{kPnbQQ@j~@og*^Sd`m=IdB4m4ffvKYX=@|;f&0D*O!fNfNahQrCcXHp(8=_ zZsSq?Wt66?w~JY_veBtrMr=a@WxMgDc$7udW$y~kqb9_7Zz7(Yu z&TqdWZ^B+_Qg1O7>;B`$W};$zJbP@yLNt5Bn-`YT2k&mRDf<1M6k z0De@+P+eNOweOZy{N9MR*GpIXYNL9F4P#Yf;o+Uy>XKfoeG_?UDaLsyTPG(;|Iu%zIFZK9@xOy*>d+NSz&{xN z$|R+gp0|IQ_3t3cbh1p`kx4RVs0IrVH34@0Wv&tgM9ycn3u=jyu9L-x>|F|_T222l z28RJpk`R_5A0u4t>@|1CZ? zb_lvrzeS8sXC|opsB8Q*WLq$Gs>t1B7N+jQ5c8jKWR+8tffkjcWe*aqaRsTK6Wbt1 zQ{_zD22m}i$x@!Brj{R=itpdADbmo`GUiQ4LBFyJ7P{v}zPV1AoKuT#l`pZ{QUz11D`+71kP0~Kn1a|@)&I}&PLVF zW95qbYOE3%`TTYa2GcPxfW=~QxN=>194n6IGK+XlTtnZA-o94AMu^5t4?cctAMomd2I}%^n1bO#jKfw?*!9aW|G_lS)DU9^yZfuZ?mw( z$EKR<>Df3tuM-F}lXbcqW>SC>&d;olkLq29*dDw0b_Fdg=k=t~iUTZgWb_+6gWC>B z?eBnuI#6H{MUMbx#8RIHOGd(J=`{#=H?RMc3J9NBSt%z>qSaHSQINLcsVNnpS%Qog zUyquZnb}>%ODr;8Kcv%(N76to+bxIz{p3-6XLeRGxGn$V)(Ehpz_@y*yxjfbMcVN2 za3Bf-3(pDS6cEsWtAyYXj*B4srMI8DE#(0q`E&9+N&o}`?>|aPOdJDKr`ic$ENu_0 z+F~!bK}FdH}T&fgaZjbyzRPLqYzT z@usJgAP=ghX7v9AnXb1+c)GdOSMymCO}PIJRmf(Q3aI{XlMmH{{<;6N#pwTTB%-4F z|7ttb>sP0nw}iMi{C9ONwYK5>eCO@(Wda+vz<+I&YwJltC47(5Gcw}*Xt28)vxcj@ zL$i_js)$DA#L#o$sO7H!sI-wN zM!#(*jaRFP`lG);zZf0ER&5g5Mv?lscf*C{v2k%BiVCI|8W>pJd3d zwE?Jicg`G+VGZ<+Aj%GYqm_S39HZL{mdD1#VAe2^;bC6?05WlTq98ybn7t(wA>q9t z@#teoW$D3Q`M!gDjxyltP|%jQgPD7L=5&YCBj^g%0RO$OfelcMyoGkX z*TxuP2Y|Kyr_Iamj1+?^qw`w_G%n&|;H;OHv$;X;u<97P|5#L1aEgsmF}E@GtuHB+xrQ-% zR+WqPk0TY82Ft(iQy3)BCn-n!GxPAmyMuyHz}ahBBjoceeT*J|VBpyqDHS`B$iTpW z|N0rlbiMkdZ_73RyRYir}=IyJpFU-&fE#BziKvYPG ze4AR@P1&#PQvR5d=bj!OeYwvs8mDse^Y?aj^mKJJ`W!e!6{jwAr{0g97!r4NGC3Ns zhR1x@)=^xTTbQ3yFrkYYEwkzk*5zp^}Y0ffm$ZMh0E}Z`~Fn-`_0z;?Bs2z*R`C>oskh z6!~u@a4Ipwx0f#~jJ(AIm)r|68wB7A|(ajdRepBMSf_TG9Y zKeqYk)&rluD#L$}S5ef3!Mxdg*JvtH>9W4y;B1qW8pFcodfZnU<63dHae4fNJh7Y&xW zqN=Q{;<2d9P}6>kZQ9a+fW~>()w!{w53<3xTZ#k1obTV;+Zb#{!_uEe>YR7oex>1W z`TfdFV&1~;@>qA%_;^Q$+n=9}U;gQf?Q@*Is!5l2SN1#@Yia+(c|}|4zG`o8ea1Xy zm+G7fo%0})_Bj`W25--^1KaVW1aSiAmmchzP&@Cs`A4bNq&VS6T6FCk@+QpMx<@TC zeA36h6>|Y$Og1z&WM*NJ5EH+&gxg8gXH-jls=yJg0d^WuQDGhePm$`K!vh{3Mp=>{ zRM#P*4LJXU*}kKd^#f&EsXX?xlga`bVAFyvU2ZOIS^`44`~?S-cz)2^mv&&rAxOWg z{;H10H*9Qz+62hw1ZlE7uyLX={Nu;alg$-#zy&9)%KeB-TJ|BIl^#m$-3GPG_ie{V zR_NthgqP~+sDD1%;Rldf)%OGZJaJXfAS1ulcERUZKQsHy+XZoQit5EC)rJBlU|Rkz z=jRDUd9IVXwB0{mF+VQ3C&}T^_2-V2;$=K^Y6-@Z305+whx}49Xw&3j039-?uQJLN zx8jxmbo4xY_^`CJYHx?g_9P?9;riQ)g>Y=;Mjxy*lPS@#m9tgaj4fO7lKrH}v-8o( z@zv0i=uUlE!SXVIBhDCAr---~CBw^GQ+vC*CW1(#Lz{oTc0KZy56RA!TZ~*&pl7#) zSR2~+fOV2886dr9;>1{?j z1APoTyx@?n9;vVS^=_8)ZE?SCa(4n~aA0?y`u;)#)MITjPZ8$m*8N>JxvOY3SW)d` zsA8_|n~pb6XlS79F1?fnXce8E?((BP_x-}gsoT~ zOT=y2v5|`HaL+rxm#zjkb8s{l%kv&U=VwGIc`006T>Hp*P~3bc&G19nAQE=8aT9E) zv8I=KTtEoqEcDE=ww~7%%?i1~iqj7i6n7H67)jfNPR#S>p^!d!{o_Njm*~hdx80hNl&8RnOim2)t=`nzsWHcAS#U4;o zo4isj!`t22t8eqWMMWQk;erlhLkKPtpp)&6pdD}Ac@UB__XC{TwjU__#R^7Kn%3B zI=!y8aQ69M|kijbUW4C((f1TL`A|!?xdC_gc8_rJ{YtN(}{rZwFQ? ztI!P&KhsYp7@=Ho06hITVYqQa;RXKFr#e?KpF(KUG)`g#{(N>PkB5M%1n1OrQ#Nu_ z#*Xd$+GyC4aZ%rA-bf?!FQj*lJm9D)w4NOg>5RO*N6!2H{ELTbphU%uOQ0fQ|CD=27qhF@^dn|yFQd8+QXtZgrZ?id#svfX+>VwsW zdMY9H#Oe#VaF1QwOOUhJ>aYqlPm!r;PFN1N_>F#2nYzFeE>V7ThQ=}N3Uya>5-5X{ zeb0}JLro9&7G*$WY=9lz_Y^ml@%sApS(?1B8)FUO4XKFjYI0T4;QvUR<2g!-5qN z2z}HiP;LxBh^{cwUq_Wmti0l}ehTHw?Aip(YR%g?y$gJp1-n{^gWu7?aS1;5uH$p}N+4*~WSkDkA<(jeXQ=d4Jo|Kp) z`3rl7_r8=OksuGxo?g?I zwk?eNehxJ%j}l(+DR*}bAF%o-vnMh#(a`YlGMj||k&dO7TR@tJ@X=bI&g0xyw@icU zsR8cbIQ_363c+T!;~7x~$Yj8jlg6f^IVIz^i>;%e&D^)G>`*wGa7|XSd`VS4p5@|1 z+PXGZB-5xml0j{Ihjx$cDfh>vfr5rC?JGkO($a^S8$riAs|QnikiW@$>y6Np=@Lgx zU>WVOr^6b#3YM9``w|coOqQ}rvWOCrSlrWJ{Ibr@jU=F>!5(Jnpf=O;+Xd|E(&F03 z_eo3B!8{RLJ6qd3i5AYzfm54Xa&2mb=o{B%>r19w0R*|xyoRj^^jJ7!eo}J}kd^WA z)nid}Eul;IKQ8rGYJ;tL9iGYj=5+i#JfZ@%0Z5*Bu&EILAimSMax>r{W2wTcTBGx;#P@$l`g4fNC9D+v~-z$IC~tph0z*F{F4l= z-WFsbmoKO#$~nUa`df79j9ucFu1H=@s$t@+LpAs0a>(#4ZKp z+`nJlEYA$J6BTY?9eBX^&^|>>i!ljRtG6PoSKR~rRQI=qN5+*J3feaS@C9b!HQ8Ol zgf5e+=6NTUj0a)`_TJ1xpX&n!#qO(~7jSIFC`-{87#eba=?hWOzhQO5;+8z(6WIco zQsu^rw&{MmG5cnRYpCA9qwVGsDN!WAYqFmJ8I*}fI!lJA3Tyx154qXZ-HjzzUz5F; zmoxl2Ov#02Z{R_x)wr$zgcMhiCotk~N#f%j1ufn8+0p+~r+aDIACx7Ojq3y#vt(w>eU z%{AUN%UeDO|}^9%k1#})uC zz~a{pLO2<$;ixNISTc@Ae*b=2f(UwTe1g_5l5i!PXuSPHOcwB~iqdz%2+E(I?(1sQ z>d@y!XKa49HP>UGtaK0C$_{&oJVQOchP8FB?D$^zZkQJ{`Mla;+R?#5%gBh~ItwyC zRin3sUxd%>{PY4L%WS%RV4w-tnLPj{B(XN8fyBSP<4<@51rWH9D-G;=_<9KG{}>n>gZ2L>l44)y_OT*WrlM?Y1ErUvHLD;|m&huOs9iFVm>z<=#Xz1YV!u|A~ zN4xn+p;w-XlA)m?%%(#~0e^!AF0I5J&3#=_OiOe54v5?rESe*Pl*CNH8#FU>2Cj8e zuPtCssC`yx@9BYB+IXNAHAvxcq-w%YwhE+{H zJ@=MU$d&CcNgtnqUx5@M`~&LAdxVKfZRos@b=c6sxpU0MV(_^CMvw4;+1rdGdK8Dv z&9$AIpJ#EPhu`N%KRe&Q`v0)^*I`kuZ}=##+pVHVs31y;NSCyP0-OU(; zw6xUFHFOOOHA;66UDDkhGv~#<_vicn?{}{A$GOhB=DGl#dDoitzUztmzMtnsz4%Dm zy|Ho6{=_{dhBUk2dJ*qgXE)G$a9bD`*}>;I#zUPE8veyf79<#@sXF z=*nyl|DT9ou|IGzVKs_1;0rbGFZ&9GTFoZsU&& zdDZC?F>!j`1#)Y4$ItP+2hcD1cphCv-S7ScEeFxC@XW`y!v}8pjZpmtwFPjf9;%c3qZJ-7NmARy?v;6vB|fCEqQ0T4{pVu$6UtBZQux*mvc zOha9dF^y{$ZQY%!<$~OH8*RW44$sOY)Ktr44gk+};*$~j*^0TFD*5|!pW?C#4p;i$ zAizGeqvfv6$r&AJn2YvR()p1v6ODU%>Qo64)UlxUNe&yCC}JCOxG$=2176e;-eLwr zAuFmTUqNaHLTmoxC2Jag{QOFy$e<3Y0es|-v`NeU7+H~Y{LX$eh zNxpd(q|uJ%Gd)k&^A(ncQ&=uzf5fx9j3JTYIG}3x7Na*nV&_nB>a|jP`+H6&PL43% zS#(RwxSS4x3F-)(&b=fFg!s3EgJAbjVI8t{ra}Usn>lnI;F-0#4eqRfKqmNc#>t0* zbW*d@sCtBMbAfZE>ftp{6FS{%5LfXxl>?m0-(}78J*_%;nqKDF+89#M_C?;+?kY z3G4M3Z}D$cCr$8#s{Xd|j#By7pTPH)%V!^vmQ`&L5fjrtr^BUAOx*gr>z>vqQIbj% zJ>Bv$iD@GBvo*gz(^;q9p=@Gi)}QA=LfwQ+d2iIaNlGE;hNwQ(kzA6?P)W_b9ZMRU z`7I}bGcJFi4PeZ2@|`|}^&&7ll?%^WVPC>_#@scI=!S;0DxEvGgre6DgbFUvQaYg) z^Xv1pJ^+}JeIutK`Vge|m#}M1X~Mb1(tz9zd*cj!&VQ zv|pg9ce1}+G=!DHBLhpbqD1*3F9)VlMzcAXmq7SzP0G<#B5Lg-yH`&}zL!w^U=v-? zza&$qT#GBOD9g;qs7#UkBg%7QXbemS8CHW$F_bgT^6-q6$2-8-(1>>Jm|`+MWCDm( zd>x4#b@;k$GYAL$ zOxX$NO;sY%ugP*w?y%t#^K{nMJQ+~PDo@?yS5U*Wx%t~Ld*CYM3Dimvi_8HuV>;mn zbTw<>TivNmz4tm$d0brJ6y45Ya*-e?*3o0jTP#_D4m`Q2Os{ss13okGs)1c!Y(T(n z-{(so!V#Mm&&&Dk@R4-fpg@q{TqVsQza&m_3nmrO;&8Cmo4%b4t@N1roxouir#-f~ zSUNCJs&gkm$9d!Otd$m|1|Tg2DTGR4ZCcugRnfd27y&OQ=ewOI%cRRJ9BSunQF6-r zK#;`^VBD@Z3ciG_m!3po;R+MZ&C_x@$u#OKGd2Ec+6DSCY@%LU;>vb*Aa@2|69Xwz z#_6xOdIcGJoBPX~Hy@euu^_Gj^zZW0B9)-q(Y_5d_iXXKm5p2G06$kY zsJjOci5F&*{TFpAKx)7xUg0&n0ogr&U)DOtapn};Pc6-0_1t(FUW#z!rlb^us#{?i zhmrc%!0MBT|Bc|W6hC10*yoU6N-emIEWq_GYivQyfA?3%_t>`(EF^6=hi`) zQPm8Vu&r*8gj4&BLQk$TRuHU2N5#a%cB%|=Id2PIfUMe=5G=9Rw~nkvKg=(zynt@e z(9yM+tv=a?o^_8aOcc!oGqSKSvBv^<9|pX8I=lC1&e5eQ|O@Cuaat zFy@*8l^UzC;h|qabc1MxsAnTA`xa{4wA0S&jlvtcFfv-7Do@&t*vw9VjDyQAEpt9~z+sU%%UjJ-GETKrhbrmm5Lj5_;4_q ztzLP5X$kc+0X-a!9BA_IujkBui}v@)uIQg4JK;h$#a^Mq9P`$DvhBIxj1R;C5tSUv zHdr;Z;h%*^*(c3GPVq4Nq{?GF!wx{k9d0clq>r3gnVzk+M4&Wkk6WHererFHsfjT~ zv+3@fEYDm5z~!>F6b+B86`EY`PD0P}r1ob=(OW0tHq!Hl!`%!q8X>@RCCVO{b^X z%S<{k7OyD&ny5XRl&V@osOfRqAusuD~lfNRjZ^U?&{en+6#cc>WZUEkogk4`fX$bDS zts2&opYHH`cZ@5hV4N`dM+QbmJN1D^{^eE(IWGgR$Zr6PM4I#@3pwcVPuC&y(qES` zr-E7!gRhB>ewUj$VtOE#W&lk@u3bYTonG*|TTdH;!G)c88R(8Q^t|f{9|-CsfYAQN z*^&83R)xJH`15$ia}^-xJKxp;{Kk(&KAQ5H zDBU~STNx~~DB6Mmt-rx>$N0FMU6v0dr~~91J?%UZa+{l0Iwxnqe?fHcN#VtD@jrz2 zhZ6gC+9FMQXM#gJ+q(;La*9jK5B6-g$B|w$dctqskaq5WNiI5;-mZ10ez`CZU7WyC zf<&GI0t&EtdN9hQ@ygASnwq${t)F1|fZjFdYiTCVTl|EeypWw81?nDd8OcVc8lLIK9y+n_a7`I z_9pbL)=7fFlpG&V;dt&_1*wVxH!#jaEJa{hS<(TA!AV7IilF*H=E0 z8=UtKw*lnjBP|i~g#Y7NhTZYoT>!7Uu67duNer+CMmzPdfQ|a(LFDyMi`Xsum6i6% zTF+dy@fH+u?iEkW(D#o_gd3vod2F!_gXo-{y%gXEefvN^9zpN3$zL?|K7^O*1u}5i|8=wd#3Up8 zzy7zM;C~*ts}aB(|G##&*Zv=GXKNU7EhiJdsP21ALf&Izqqk4DE|E4Al(480dTt`8vIXI{l`b>AzZXf$edt;7% zqC3~-6@o)HSy*nwd)zZ8JF%Vl+=D=5pzBuhD0J$_k&+x6tFw;}c1JSqL=EiQL#Hd{ zIfP0LmzmNScjKNU^d|GW&@eq?q^^02O37%NY$}h7j;@S!xXqp{QgFkseEogaj8$BV zMX1^{4n7{Ias=r2I*d}7@ISj}Tbk?H?Pk~I;}5-fhhxV~ zEZeV%A{39x%E+{tS{L#XS`Cm2`gC{8$_R8AaYKR^JxwEYvSzeNquvy7Jld@xB4hPA@$NnHKgYK%Mo>Qlc4Uin z5$P6yx=e7`2)@{wkattIGXpPkcW)Wjv!yBURgTE8Pm!NIG&=%Ah$SnN`5=_>jw4P_ z>G`ZKog-wFO20IG$s6$@&DCkwDpf?u^r9e6)GH|>vcSQr<7aR~ zbT2fgyPh~EisfXzl8eeyITzNEkNHY;J$f=N4H`;;nHddOpt*MKr*OuJvF<^Qew5K# zeVtmpza;T^!C9Qdt9dn8QUY>2=fmh^?*wN`=GstNj;R~+&1q5Z1pD~W#xoAyXH(oA zxYhYGY!iAA3&bsd<*&roiM7QHT6(?~5-8EB_rJXV_Lnf@=?K(^urx66NBNLarPZX} zZtaKK{It2t5DWaeOKRD_C#f`&gfukt3LnFU%fim?1!%{CrcA85y2}J)!o^mCE*}1S zceU-EkC0@_I$wJz>mD)1c3L=e=YH?B$$E||-?Ncq2h|D}b8~gnzsk<=RFd32j8NJX zLQ+RJiRnXG$c3k6m}r@=Chd-q*%L6f@s_osYU`+@wawhZ)7bV7&eFRL&m2ON+Hvcd z%-uciq;RxNx=3JV_KnM^y_glEi67-+EGF`Tn!cF^Y9LM6?@~XPl0pazpe^y7Z@AN5 zRO^P*#ChP?bp2rXtxOv7;5zYSV{pbmOrbRj)C8b)N;u4N1H%pORL*g62~m(om(ebN z#Clxp>{=L%s7CAu-epzju1!k3U2b-KGW#1|#<)QADDhjB^Rs6xFPDsUrQG$DQ`ML% zDvC`P*QP~NHoLrUBuwxPIvhXXX%ItiMN49DpaDiXG6O=3lfI8 z%JQ0(X2Rz}yrx>+4+=IrdyR+}nEhtrFMcLo|B9Lbc6KocOf0^}@QsOE^)HP-;F4%k zH^t?Ao5A10t9BEv%;`&yoBsj&?1jd?H!;Yq?=h!PhHSj5ko)4!zID6E@1ho}2^M)` z!x!z#TLnUH=j)WBCL^lbP`Yp#HIn4ZPM7l{U6p%EoBijpt;xf2OIX;;G;&x*t{AGB zY$QiY)vt29{2XZPjQfxc=7LcfZ%LBmoBw!v zlU2l{v)WLH+J^=86l<@ASu9R#Bg*x6)O1;E(uZ}xS7TEgsjEQq7^;|rp@II!qhP9m*ZteU(Za1o=w1#yhu=Y0?hI7C(RnZm3P-TD8kqk z2a9+Iqgz-16rhRv$5XcBTalmgJK+UaltC= zppA8qV?0RBHbS0upanuH;;W4v<2Rh8rTL4C;%$rk^gTvYf~t-*SwhUxY+icC75kG!Em>yYD z2U(>3SX#Ru>r1Hd86(9z%%c+)UJT7edsSaLBw;!!mhy~{2A(Ve+G|HeXZq+wYpr-b9X4)${ws=IZi>u492?k;l&iCZ(D zu1%=AC1t1+msf88kP_zrrMF=v}LsM4bwwlY{9V!u>7mzMrpGaB1dTchDGrQ99> zrw@1cVWeYqI&T)N>@RdaLJ60=dZLh$ z6Vy*Wl#5g;RZ6VcFJ67WeL3#XVJb&!theP&RhuRvVt9^e64hw$?AQ__h$eY7^^LLm zeq_Co%|5+}fC720ov%^usxo^mggiEdJY5UUu$DE_iF>bt*!Q5o;~`W(Rj-lmD3WTA z-FI48QA00Y6_%uGODkkGueh!GX!K~y_$Rm0L@|xqm^*T#hbACD<*mkBEpi8D!`ADq zC5@BIqR#+d;T8BquA3Lm`pY8jud}8QA|^R`!)Yv?=t<8*ZG(mGhPH$;f2H`Um+1b_ zu~`|CBF7qLDuMQI#I!zwm)_fNBb`bX1)eF#VrZ52f~J9}@(KEyvh-YetC$3rgs$#4nZZz9-(NV-?(Fx27Z)7KmxwbDBHD>DBdHC{9 z7sc7aPIo3Ixk$DU%Yz%fk;R>dzPnX!dANBVeN;Bv!J{pf`qpYYPinoyoGJz9 zs&nTTw)=0z<|VorCwsjGi*3 zEbSKZv?U&@p3W#2`yt&$S4h9}hPKn*56&NU%amyuC#h8RXCKb#4zn@~XIhFW~mZ3FAc_8+Chqeqpi~71ARi^2AMne-gBn zW4!VnQ=i9f+|o%%7@4BY6lX()M_jbW;JJ89&_;w))a3Mzp|^XcVS)FS%>i9xtJEg! zB710*)A%D*C1`C*FUx8=X=j}iO%e+STwef+wRhC*)%;K} zPNF{1qwgo3oM_WP6|hjqqo1Z35C}OOQ(yhVQMf8dKZY5OoBNz|oV2a0TnPm`QXKi68piJe5-Ur{6Q%Zk&kkcY?piD`&M8711< z@~*gRloX959`WQp&@mc8Z{5`kRlKreF4$%1kV9uj#rAfLb+7Yc-ZTMk z0mTTirfhRlBokt}NlEt7Q7xNI4_~pz+4&J&))P16ZSmLBGgVUkNltd!tEKb)Gl|@g z2~5WFqT>0{J{e0<=|s5f-gI=vZ9>UA*tMv}}T zx?I(am6el`qo}H_W%5fW;DcV#XgYFf9O|Q`;trJK6JsuplX#62@Tg|@e*Uehxm?<{ zF;t<$bxf%otaRp30YOo&SuNpEvrHh4Jb&MLXwtetll>5SDy^q%cTR^LeUeZ}ajzme zF>)ES5D9c-=_F%l;b?+--uNhV-f!Y)%Y1F_Cv}b0Pa*&bb^Y8wF-;H+T74m^7esJVMNoyl{Lb~pS!xVc@CsS z+nC9@&lPn>ZLfg*5Wg^Ss_VLZI`0AIBq$aBl4<8tXd>?X}plBuFLzESa ziJ|LH?obn5CNpcDOdr;cd2R(SIhcH;5~esl?d=GsK(Obf5fED;^RfJyR^3N8Jou82 zzctCYudma(iNzAr5TnI(DPbEzR=RG+&Ica)jE0(HtvyveFunOsN$;Jcogaf4$~}Xv zWD{c@dwZRnrGbXli;#P{3{1*1vXUE3ldXg0r<=DI;tN`JGec_Ts{YnC2oI+PiJl`! z^eD47qVy{2iV&2rz)u}}^U@ud^N#d9{Z1;%x~XT`m67`+WlOMcIAQWi55<{%#pvj;xLDYaoJfFjlDX6W)hXyMDW9AmMz?qzmuHFR96bbOEfl!bufk*vg;^Fz)mdzg7}$acf@)^5D^Cfb(3f-KBIm*yQsHT#j|F1 zhh)l;MduJTeG?LWlUNQW1y^*l^3YShkz{^#Myngbbv}fjVS4f~-M;O*5oYs|&lyR+ zi^mSI7knJV;zB%*KAbRWpOuCA38JzF-Z8A5TQ*l24J=u-b@P}l%6djCf2+h)NT zDrj#z`1kn8baLl%gmN;qFuUfVer1i|6ZfNSTC(1x!t%BgwkcwR+VO;e^w<__2ochd zg-2x=Bir3heI-`xJBc=#aq=*mTS+;ynT$mlQ;BGwwiM|TG}l0=C0e##@9j;ve2Tfw zT8D8tZ*Lpw9^3b&-(&ZWq07EQ%@inK8V0kA=Z5d$Ap!!nC)i2(${vcp1QDr+0K;~} zW`ZX={(8k^;GD+v@swns(q87HB{l*@HN{=;O#2B1Iz@bZL%&C@v>7&2d5={INe#Xa z^-MZw74FH*U9NEfKuH{ji-2#N4@Qu@|7FgpjPRM^vRL=;zb{p~ip@)vS?}nb6ZMsG zN)M;jbQBzS;Jf>!3~huLk+vx=`L66msW7g$2i)r{$k zht5PiZQVf}d{Hz2bNd@UWV7Soyty>8chS9GA=`c*wAA?WL>wo~*yV7OBUh(auoBLc zRMED$caio%GCarblzS3?DC2p~T7O#fvv`&E`LF(re}U*9fHdGIh%MOIYPnSCwHK_cUFU&5mgu67>InLn_S%W4gY9@aHkbr8`7HGV1s&RpG0VCqVWf; zs-3tI6^y;zH4u>^$O!q{T=qqN7Z_iNRe=Q(!K9cso~VkbtCqp8(+)MA7T$)Qf3D*( z_W$0=d^d*G`BoFA|q#V=oj;wn-&3 zGso$`{Hg>jv`Qz#HMp-r3u!q;g#D;|fuS>+16U9cr3O+mMySW!RWs|%nGOXJXI#8T zw*A*^!^LECCF$&Ux6~#nwmb`ry(jhvXj8Xqu`cnsmC&9NFxJDeIx+VUm-(7cp+cba>+pgjqBqClt7!LIx&{ z?Ov%~BCh-nG9E58QrYT{&EU%|vscqt*L8bs-E;Q0nz`-l+Ei`AOF^a^OmGkMjydE# z_mcf)FXO{(K^v~9aAG`dx8tvEgomwy&sH99WL_1Yq0JR&|C_YxmItma|M*qgVh@y|Tunt#=Nu*t8o&PUME>6;s*+F3MRu{x}5 zIAP)YTbXa>qzl{Vn@8r<9V*H+ z>lYLIDq07!VQ6GqOiEdY0iWQxjj5*T?=>y>RdB2GLNqN}7B zzuOp_@Tkt(Bv(^7dfWy=1z<;6EUmn*yL)EtZk$UCaMsMswA__8S#P@fPO8pC14$RM zf(kA}qCPg;iW@H;PwhTg#nbV#u$Ayrzw-1UNfH(^-aS95MYpDMSHwOxB2^)tIt=U2 zg-2^pX%9)fUQSHsD#PN6s8cx8UT3?E6scg&w*zRD2&ghG?-XpE)FAUPWRWpz(;J0H z6%KcCqbNXrnrMl{{rlRrs@tPS8xbexcNbAlY6$ohC{$PTc9?#=9F5I1V({C%b+G3) ze_A~%d&5$)GjC@j-9_Ljx1Hq@a!0lkQ9#UY6^YSHKzP@DCTJ?r);`;mejq805ARUf z5{i&v?XuA2Pgm}XnB|fEEJu7la2r0BQvbjxjxj)~x^8@^LkZdu3OaajXYL`QBsFmi z(kx%)OWjuYfV2K+x8GlT7Q23$$T_sw0Aj-JMoj==-5g1in|mPtAR^GZ<&mT$bCyOF z*uzP5{9@Oi7`p^R=6oB?MI5VtJ)G?McJUXoLpl@g_BS`>(A!^Lc|}mN#hvitygPAE z1W1%OM-9z;MPA$I!GxM_iRt)I`41sX$SM@MS{z>+f0w7VfAK2<7!Sf)Jl5#1O4763-bSTTy$6eg1 zXmUlxiUWKSxp>Y`bLaXF)5 zk?;J5U2?HH6dKZYs@k87f>KKyPL*6oPib_hzq?%ul7=h~`sM_EH%SnQtA_;1sOkA? zMn$XdzuFRdNDT0v(E9yVSboZj<1Sx8hg-LXAXy_zgyu=nXvOQfxdChVvCp%o8Sll4 zSNKY)ogXoThG<&3ncAym!#{(N(@iukyyp)i(Eg2C$0iT<9jg z6w*Z1kY#&=j}}%FWkLG~b}-yz%galD#fQ?u6NviSU*jHXek)+vloi3g~5cVF& zD|Hf%Fe(o35N_pyvLYXQrRUdPNXhM8hip#M%yk$jNBO|a89B|6ib{ui1ZntE)ft!p^*l+!Px!#iqRKOgZ=JQ*i z*ffa41txojux`BhF6{938~~rdZ)uakLX|s z<+IamRWfaZ{%$^zZhmhB3zrUPt`QTi?npy--nM!|g$;CFTin}-2br>q4u^`_b;f12 zxBZ8vZf4qcy5f`BqK_b9!=-IW(#XAyNdO2h#MgfxymIx;htc<*tDe16*vf^;FKu@yCxR^C;{INZS;G;pVGtZ^2$TxFjR7 ztx6cr3A$#~$bfhOI7y+%Kp@{J$=BeH>*;)TTE?sY&RQa2JsbY*V=VMD&_V9 zNvV9ztsWqCMb?*XqOeTyE_q_s2)pCN{%q{{AUzS*k9HEC{=A@aZ|t$$7mRzw9>KwZ zN@jhgBry8rLgjJb83nj?Cgt-w_4!De&-<%z(hYn1<4xO|f@6SVM``Jh`@pHBoC7 z`89bHiL0`JmblP~_(6WY8YbQe@$Nn@?m>JQ$xhf4yd21g$|0HVN47zx+S?ZUx4kki zislQ6)!8kUww44e%=b`a*4Yl07l)GDxxL1sRbf`WY)-NB75qN$;M#Xb@irhAGtWvd zz9iUqWjVZFnx@20zh$x|oW?S6_OVcG0fH0Yx@QD7As`6W4AC8)O)*b)0dLA;Qv@uJ z_&0zKmma)%NPoML(i^Xb@rHgM9D0?Tb=MF^xKL*VqQ{T6vbyvEsna(!tywXgu14sd zZKyV}mGpNXgEEK{%9f=<&v!n@Wwo5$!cV5)KtzTmgPr?KNYElVhb6WLK5nnK)eMDP zc|QKqN43K@XuCe?6pg2GJe+~;y&}HU>U#2G;hbMdgY*>e8~WYx7G}m8Bxa+(I!$D1 z@cp_KsGG96jE_}w=Te5oT9qo2ay&9UNp%5ohM}bL9C7&w^0kLG>Vjh?1dsE{r}GD1 zhrvO92lN@Xj@g9*=aP=4CF)kF!lM@Ft)t&xxtK$`*ce`tde@veJDeNS2-uR~0F;)! z#B_7yY+1?D!{uSMHG9aWRe|8tAzsD)jfa!Vy)i$246RZQ_iH3(hgF&u{5I<;M_QEK zkJs6%FkQ76ELfT$Kne*VPbZ%|*i5Db1k)Pe&8Cc9Ht|zq%)kHDquLr@xucOhm?gc; zgx_?z%1^5EvGl;kU_@z=L*YYPy2E4PrgCK7nJY+nAjXQhmtts=Ht4HH{ozCs5i;>-m%($`P*y$6qnL<`E6T`v5 zHzC1ZiHWVwTZs2fEBeRVxm#H?$rxK>Z;YQqHW7BK%GEnOcMO0veMdbb#b6p8Xj&E4 zhSz5vN{8y>1}Vz7voj%H@EqFOZcKGRHLzk7wVvE;S3b*7&;{bPcGADoXA!VF=6e(1 zy_EQ&!a4}pcv%>rmaIbOwzHG5cH2Dw5B)Hx8cp$IgpQ9`u*~^)XJ;s-p2O0~;lgv8 z-12CP{eKALQ5_4-Nr%NUwxRszD`xL{MW`&_#j0whoVdt^P4ld2FTLP5-jq zKjrETew=Q{t0{O6g?Upm@!QGwjd3Z5^QlIb;o^357{wT8^&p%<%mJi6XE)0?(H*G3 zYY}>=coiCEE-qky*y>s7j#n}{K2&bb@f(S5@sD~)&K*rqCDxXE<-mAO#o#@OV^TR&F1V?BdA1?fG~?i zWO`1!aac_q0RUD$Qk7XwF3;m>55b#avQa^Yu?tj*WvjN zx(jlf>zINW8T~Xv#ev=BocfZy6)(&UYq4l4HLP(jh5yIi+d z9eL|LE8cu)rm0akKK#fHu*`2eIuru^8cM?wongE^#c?`i4jr>_C#0?0`!DMLyf3}&3~&M26iE8P_owR#1tbLznaHeSvXs5sG*Vt1r<1opZJkWrLm1mG;;lGl68%-q z(CC#RO*o&U^k3ILoudKH&hL8Yj5p(D!?LvkiEK^P)MVB1z9|0S+Pl8(G^}^#;m-ItJe>5+-Zg?NP_HOQl{nF@6ba6TicE!-$-2CVvV_VSRiuK`!G`ccd8qtSb@ zx9Pd}h-6Myi-IBkM2!XY=RXHfH#5bRfK1D}_AHe!4VZu)g2zwO(;wBV5)rfdyeKWl zIoY=`6x4a}={{pYX}k)NqL;iY)%(G-m!Odm&>G~{c|}l?A^pn zeaypV*1)PNv7D&Rr425rOX|GwFe0{yM0^2;#}*pOb-bTdyFzo^b(7jBw&Fjeg$LPxT%R<4qCHu0;D%0^qM8l*&jjxCSB)olIMo^3A7^MS1R`t?mAa`GfA%ZFuLn#lr~ z&iX09mg;L)Y#h@kD<{nijaF$gW9thh4t5D}LXoGd!yy@e9t<$pZ2fak3Y75|{AhE_ zD%ob@m#EPo2pK+SN)3}uuob|W6dx8E5JSmgd?7z#u21H`1 z!CB>pJG!VHbr0q=#(z(K4M!|e|E<%;yHTKb6aTj3bK+biR3y?!r6ep^Z$Yq#0$L2O zX)gsQKaj;=$+GUDUK%&7Kg`8| z{v5NEfam@1jo1GFd?c+q&>m16sGs+Y+ogle%FPH^s&1LI_ zhtoC*wMy^DaO90=KiMX zLKa~zhnfEJ5J8ZV(p{Xy9+UOvu`ht===@^rImRpCsxa=)F)OmbZ4^8h#S!>>ZuDf0 z&APv%^6;1n9?Ie5;^eU*mK$dV5+r*?SS+ZV9@C2KQMe6uJpK2>DeUJ4msge8fddQN zD(SMRSN#UAj!%B{?6JgB(`_hn4O{;5rx(?=`ENL;4Tj$L`YH`Ky31M98gGaIwOc+5 zPhDDmS5yD5t|>G^$>kg)9AarRA}#}|gd~gbUqCOrCR-zg>hxtZDw&yNSXpI4=)k6+ z|D7ckxh>P61m){;U)WnPS|8-y_^zqO&>Bsb8AX=~QnPk~`nmmKaZVh$vDV(Z)Nc|g zx+fv%>?K&;505i(ZR41shAZH&5USML|N40rUB-mhr~i#k$ND?g=!qb~V@sz80r!bX zw)8U@&2sa2aj7RMXs)bbgAd-$tY4lMEfOKHG*+(Evub1Ge(Zg*Q7h1b-Bk&QNlaxO;P)*h=IV$#255&1}I zl^^Y&uT-oZfgN8bGgsUl-K;4Y-HK?gXy>aVC4fmmK@)!&hTp;Ot#N7tq+LKfs(l`I zT3^!!_ad{hXcMuo77~=^u$)_X&zHiN>z)iKolx)PpfANTK{55LBHmMN9i`*VEd1&? z1|JO-K1{Fw&t<=QNXyKih7TGUks7zLe>U|M^WVKoMr^7xl>A1o!{(@IhOMEj4X zflh3yZ5&>00hCQ({C7A81=3I<(*@u!|GUr#u<;qj0^Kv?@H-O3@Tb%UcF9;>Wi=Kb zknnum;@{Ytl{jqP|5|j#^!fpk8_?R{GmZ;^R5sIU+-Pfl6}xT{$l;R4Sg!|8m%heC z=Nrx!iPgIb-;69o0bCJKVrOP=zVvzii6Cea!}9KQ+;Lmt+06R?Y>{bNu+NxFY?pSZKBZqGY~6bMci$>_t`6L z?Py)Gp>VcEIZ4qg(hGu|Co`gpNC~ zw0tuK5d{3$a#we9;@A~hYiN1yI0&_)4%ez6C{9a{Iw$?t!f$C+U4+T?6RDT1n&HM` zwMiyeBioHH4?@hr#?QL$;|JR!z07RSzFxcL;!PoP&iHYICJ@zCCvA>+K*KD5F6I@5 zrDKI?KS)+nR*Cmud+E??u;@BLnZ(S>(An85;HrCCG&oeUfMYw8{*U9B?0pjw;!$pj zW>2xgesZRN)=)oDR~HzkYF`S1aRw~ph35fodaJz8!o<`>4Ebcf}dcYsIEOR6*A ztNXo?3+UcJ13}zQ{bbi_m%L9^!R26^4s{7_K z%bTaA#HDmfe9=(6ZiX2Ro-88)ERguOFMZ#>xfQ9s>nl~r6TJL*L2r1R8W^F{TnucV z(JOg-!m4c$pI$wZTP8p@R(u&Y+IJ69 zmU429yi7>t^_pFyUEE3k>W7cFVQOYP+%mO)Y3A~b!9tOOospSF>1t9Zy9%3Ii^%xO z24A8}6_{sSu$sAoc5X-(!*lONRF9g{_4u~MFVFv&1O`{`n!#7K5|fupGHua~ux@g^ z`qidz4`0~@LJ3n9AX`__yxfV{i zs@h3^%wcaBIgs`_A0rwf9yAL9JgTg+l=CDD2>qiQ?N(Pievck-qsPSO=Ya+JdD($5 zi8Xh;#^!Up-s^_K=x&arzyE0Vd#~?#tUui!ofYMhp z>11Pg8V1Jbd-|{S+A=jZx;(yECb>M1O@jr(Ig+Zqxz2a1b70SWeh8Y50^t@(bj)39 z`@6W5m|cqy&^me0|0?Jpz4hOtG5W+awRNH>M;b^DYS{HRPLG&9*!1&*Hmms@w9bag zVvLQ~BHx5;d*Yf}7|2or&y^zfPetUl$wvt=#B7m1u3Vrm;iIf9d6*XAX?M~6nTx&s z$LF^ z`o9I5vj_^3-q@H}W zr$`Bm=Z_C3kx#;eL|m%~KY$XWWUC1+wRcQD3{0#^*yW-%+;?$tVe11^vk8yHGZsD; z3Ar7_Ue@pwZpPg=JiA?fWiqezz5UlQ+v@BaA=A{x_1BYY zEb)6&iS&rSlcpR_KUlh7St#G!xY)1rKM|iDZr$5;a;Cm&3eZ>l+#Z>z_9~snbwQ%} zXSmmd%vI?uI8p*;o1c`Ix=_Q8% zi#qY@~Fy1^4FKR>X_{!2KYN;dmtqn2qn+b|BJHk0Bfr0){UZqB1I7qK@dR! z5m0&$f{1_^k=~{EBE5y6B78^{=`9EX(xrEzs30};7CMC9ODKWlPW=A=lzYxS|G7TE zs{}9XST(mFA^~yx<5t_gUeBXPUzOTw~vD}6A;?|2V^-Z^FhFJIOs_2 zD)C$n0ouB%l1=UP2E3;$(%x*E)5vgpq;5lOYO`i9JX~}F?n|t%cfIq}gG8@H2g#m= z1Qgw3=Uod_%d`q-LX1|=E)?qd@30fCvo>@1{4k_%0tV{4zgC9yy;JG89@q*VsvO%( z^s!KrQVn@@i&cDQj3)IJ?l(aJD?G5G0_2#L<)uX;M%5)XOQB^O?{m1dJ0*5d{Ju^U z)4nug`@!C@VA_nHMTU)@4(#kPeE$m+>SSBAU3zs5Fd6z9O5&;{hwS% zCF9NgUVK!N4@O^?Y~)pJ&?(fbC6*lz9BkXnHn;S~(8~GPF5ApT zL~i)l{>opcfP7P{nv2e-Nysj5qc6F+aAq1=>ytodKG>U3WKcaOW zG-w6{l7@%_DE+-;M8>ozOrDz{Jv+oo!pR-|?2e0Qn@~o&yO9c%Gx`6plLj=pZsiDu zl(ICipT*_1dA)|;0iSh+@W*RvRg;=l@vTd{S2Jguc%fTD9`L789FFjrl7sCQvbvJ+41q+{%eN|+-<9S0^XzXYrM9g74)b)afTdTQ z3^%2u#N|)=OvYxCQ@5=TtQ;$;7bm$}`r%bpj6->MJJ)VZ>KLhyZx3%TOXr{XEY$D= zyI1-273%VxVFgOxw`J{co1djg|um4QISe-aeYeW}BZT!{I z!M+Bp&33cUc3i=;YXTjGryycYdP}CI=qudhJT?OKQ!mflUCsJb@=9&1)=InB&vEbl zyA`8+yD!N5=8JU1leM?8#d-PowUibX-Fj=orNMD3s(0cw>OqmmQ%hEXaoa1LEbs-@ zNj!LQsl!Aby*t%jA;UzeYm4_J{)mS47!y4dn!mh!Jt;|<$3wc&ZA%zcd)U4|Q7CXK zfIYLm_6W@{#bn$tT^~xVwOxyGV)XhEY(^v;toCiKdF$%NcS>Z?)A1ud=LD=GB?$a>}9HJE?SN)_^1%{7aVv{$GV(i2uR4}KK&dRe)H9G5UUP)3-H-{-z>Hdo@5 zEV9#y?&i*g*3Z0Ut;gn+I6mMtHhRi&`S3e-=yTTR+MK7=vzD<@=5dC;naYqwBRy7> z3Y0#5ZTGne|4^#$G#L`V96zzOE*Oy<72A`)*A&TX$SikojN!u%k1WvKW&w?o8ZvY6gnPs9_JhU;P!$*rIxHAkGkW{rTJ= zSb-djuP@wyuYCR-!75zYVz6DrP?Oxnu z#Y)tEidUQOGpVfbcZi)HCV%j)$K5R{xY8WT->@~Kg7IonBYWS7I23kd29OH&@;AT4 zI)cv^0sOQojkNPZAIyGS(+IrO-L$WnY5idDy(&}7MP|PN8>=t3K$k!`dj#? zEO}6Q(xz*5e6Vb{&wae3$a=JXf>846%4x)<4hv~Gl=$nnIwAf1*4r%fJ=Vj57N?qI zzTV?#$dJx}aZV%ayAFt-o?`Rbp)OgPhZl_l4?TI(|P;Ee`c3-t14#Hf6_Et6= zLA+O*dgHQF5T^Mnx49~G`zT&XMEH}WpBN9Kwkt^(r^#)`|Gp%?qzqU zFAnJ{-Y>V_B(or#iEDLt^Y)4S`Syz5`du|9vx70U*80ja^=>_?k;KeTxeppG)&_b1 z;GMA?8N_!JU{)PlHz&7*4!CqtXM+SzKe|G)|mIn0dD$MLghkg6{rCfI>r(7UTr_`y{>cx-I4c$TUq;uHt|K;oPg1(NJYx~B z;R9D;$W_lTag<_AVr{>dK08%g8|++2yAaGpO9 z<;-G5bVLE5=Kz6CS(HdA$ z=yv2V?zleXgxG#G|0TcC`8x2K{Rl)fqWhRqdrHTqKEcm%7j~IBq-QJ_2 z>&eO6Wu>z=%43pNX7^@6uVZQWe`_TzAy@_!rYJU0rXw~c5d!Y!>>fR}?AZwyX!JSpW~kWuX5nQ27k}g; zG#y(seR8zHUAeV4h%Rd=6=;+qB(MQQ2kHJ*A*~ql?$(XphM&1*S9YI&di#u7`VLNE zVy|k_WxqZL9L1*w90~w)=#%NLVG0peu|uJXGtT!B`kQ|CUI3A$H_2=yyfS5?88fQj zqyT$?9cy@?lAMhOTDm=LN+7_Vz`s*&4FCXLDGAKiO?PcM_A-saXmZ38^?*RPn=kpn z(57tfHCRR)S8j#B7WEUW^qc(^?|*n>!+WRjWAssS*PzUb<`j+#^ z#>KT|x~Z4{0nE8*%K?9~h28CM@Q| zFygDnp1=qc7@9V{+J)}Nr|Bge+crPI5Nma${Dw20mz$5L(8;=^y(>-3d>D9TESwW9 za6|gr_YT7v`@?Ke({}L&fnu4pf=brp*bkSC9j$HcYw|z+>d+i%@VT+M3Gle_UMOqv zXeoE4PfyI!WZm&~Pr#rr+EPDI9zkrJf^jLI8dXSdGJ1^TAN6V97&U6-LQNqi_8N)a zLwJoFvY3V6RT=?6oB zK9fEPU$|($8c7>CFCVn#I`t#e_y{eDpksZo3hhE20NjCiv3|sAV-CzqWZF|o@j>i*gR^e=c8)L-y(l*hSZHa8y zq$}sM2wqWPQN$k1b5hvznY}PS&k9I7x5H-dRD-IsEJF`YNFnAlv*Bra$l_lu@qCT* z+M*k%6X32VcEJ1Rw)bV6_bRMXhg>!`kco7xZYv-c%L%_Wus%8VAHfkHFFylVy8h0G z!ZzGaU16oBJI!D<%YFG$2q(gbPQ<=SyVJbU!-0~HTIZE4M=C_~&Vb>(oCO&V1=&PI z6v#j%C$AUs1UjF5ugfW42QVMuHV};R+ZYEUiCpM+l?aLwLr3A6tAIWt#nzECQn@P7g*pYu*Z0{B#7PP-X@ zIH<4gKAYcHcVA+8=ykSU=(-J1QXC3w|4X`0=NQ&3n?vl_RAzYYaDA zDKu^#3JVE07}Prgi`&uSHLiAafSUmhJhrUsU~6Eg(q(G2P~V@}k>Z`NWj|WdP*h^_ z{NXgnAdQ2mGmSx z566K?@OL)m8wNkh?Ad@G3>nMdoYh@j{mej=pK-`?jpI+PGaYe29?Y543R%aMm(=3` zx7K|Yb1&(x>}9$y5c17f-wnd}80P?)TSZ)}jG_%`91L^5XS_SjA^rLS2P&+*obGXW zny}+B$i0zOwuhaO$C-QE|B8*k-8YnQ1drA}NeuA~ZdPhCtIz>tUgm^e&;3(A6r`3x zh&*|(4vXaF*QvT=a>xoje(84zc!mu?f$0BUs9DV_7h?r1VK0}Qh+2^%4OP+hVmI(e= zKhq#M04VKl?nm5^LNJ5AHY1U9b3L<`P*G2uE4pB;{=s!B`6HaPXVcsYy3QK(e%hZA zlYbY6pVyukHoAKL!>Pl=KE<%wafqBf*9%e=ueo2fmmFt$`Sn59VV{RKKC;5 z{WD2m(Ess?AYHAGy(RR@g=@gc`-;4x$d-aO|*_}XQ-T)v*7a6_!aRXJ9mruz?VjKW) zg2H<+k|)cDx+5eWsa{FFnVQ+}DN?4+))tU)Td63bx5{Cc&TGuVF&@R|t?2I&AxT<* zH?p-a$dXfN1T zKh+;oan&9KPri4sv3Jy~UatUd(jx~nm~J*Yz@y-P>gRdTX4t|kn6--uu-lZQALq2AQ_8@1S=Hme{xokL1;@FRQ&9cptI0tV`Tz$1{ulos*aK z@sBfy^G$s92SFd|vzX3V4-!K_F^#L&rh2$5iR)y$eAViv-$>030zSS{^cHSN!6$3% zAAV~^d$q;!1iNFWRXO%TO<%vnW)|d#q)uRSQV1XIioze=ns^=M_d?^;*U0}i?T$ET zdD9;g12f*mG4VI{Q)T!j!Y7-kFK$*K%5q|ZqE7t;6MCXg>+n>a#K@TW-M$j?2uv#m zoDHymJZwK+(74WVTUsN4Ea6{g+sZ0R(##C0GN%2r#)tFjJVtQGy*7ZW0JWzvI5@w% zEL=|vP;U;aa~WiuJ$GYvu-lN?=XXz0PlM%T)pICNVWl^yh7?`RB2g>%ZFdV^&BB;G zkLxwto&4um9i*K9_LoX6d`G#IlQ{5QNyGta9-JSN7Q%~^G})eNk}hkdzQ=X&Q=Gv- z$l_?z{{*-@aQ|ZqLK)O=)2Xw^y5JjosId>RwXB|r?T{3a0MoTRsP8{Ck?CV)IqtQU z<*;;Vn*?IPxAS6r#G_jugQ$=~_6Mh+EwY}k%8x>S7V&6->0-&(SLA;1AL-#vHNV>y zvGfS|*5Sc2)q-)q%Lea|(WHyJvk4My;I@?SR8IhOsHy(l725_jHf|7Z)9KvqDTZKK zQxm=~<^jMh{rvJHZo|%vc^Tcw;cNP7VAu;vX6aZiJCsQIOuclcP*MV=nc%tS88HW? zPf0EeqNdydsMNm3!&uSEcm+9gS%1RrrA$8P-fmelY;|~qMEWKHCVB5@htD?4S>9}R z&=5=NW3aJ*nQnaShpiA-zRmf-&%iYYM2PJ&TOilgBurMYR7B9e5-g_o1>9^F1ZLCr z;&=z&27Am2<$|%u&SxazfNm*wEsX-_M42|JrFKc5MPqp3hTt=)mnHlYu0sHVm7$wI zwuIqvzQxSK>A&@_L`MTuF`DC=83Vt-A?1B6b-sOt^w>&DF};t0k2j7n-t62r>X{&O z`X}v+TKn+yvEFF63|vsKw?T!C1X9vfsnsJ~zHO7&51xW$*~^t=*ZsQGS5<-pqDP%3LLyc4z=?@pvlN?N1~I6i`#>WHF1u z>e-;(CDEJw2yKT>L7sTT!UiHDzV#{ET0Fkhy=rp%Thn0)Ho1uK8;pv*cN6Fs6t!vZ zw->a-Co$!Y0MHIypy$68!w6O$tH$$$_(V-D%^I}(PFX2AJ#I@R=+EGo|MddrZ4yKT z%6P{-PDSoWenY>`V#kU9N~iJ%cK^3-7~{3~hh}Y*sr2hzEa`h3Ch@YQHYlhi?w-GW z>x|a3y)9S%Cot7L%RhHaUr;V(wjeU({+SDQhsYOi9yDH7QdK%cQ}Wf2C_kM>5TLpM z{atK?h+5}9p9RWMlK5k^(J^%ugk<{xXR!Bf0q7S+dMc@<;gSvMrI-g~Zm^^?%j@u=(F)!yU{qq7LCEHgCar-w{ppm4^_26un}Vh-B*%LKP?+A)@1ks9F&b; z%%Do$n?c{@tMljy^14eWcg;Xow7PUX#U0~_lsR6ybfR~a<#_`>A6>wgH#+WO_RvuN zq2Z68z3){9WBm5RO0vX#O%s@my&R4!+i+UhB>ya-g6X>gvT*G?&rhZMs=WEsknK3W z7xM1c2eLI~rwF5}uU^`gM>DSvx59_`xh6POZER%s)>jy=B33$H1sjzJkh=OP<4S<} zEzm%nM6H^#nXZy1x?DO=F2$ExtRG45K1a-B`b z1BxK(qC<3C?NB_yyN+qLU3kg&#Hh%zWVBA9#2C%IRkJY2qOC;76o{+xG;y4mn_~X5afaY@|uJ**RLa=h-#nzXx`BGY}A+yTC<&%o>1nzvKUgJrWp$HXeJa<8bXx zaq*H}i`JHTTTv?m>KJoEf>|mJXVLP zEHE;gWkenlLuO|4tEl(AA-cf8A8=m;te`RTU@@Y-p~rd`Pea&?kCYzkuU}xd_`k`3 z&rCjN^!UJvbVPG2D4Rd+5R(f$)BwzJ)Fkr)Sj$A%LTkAT-;n~T9Y^&1jNlil3B3~Cpc6?&SRlVXcCc@5PX%)L9`S9I>} zC1wVNpZwY%^-WvG=2wmHKks}mtSVfwbb_BlN|}AbZanSm6yI%f;|IoDt82%>C*Igy zycjF;b-5Fp64V@5=d!Y2poR0<(Z&TVk0Vo1d=28)<%Uy}wv!XtjTjSHER)IzCTc zU_yFQYOeoQ!!i3B>b2Es;;k3GeJ*#x+7qQ@JZD}B=MOnFtf1GVl#OTv9hctdGLU#HBk zvl%(Ag>uIPu^yZ5s!lFCo7A#{HEP(v9X;j{eOtkM0fn1Sr=nBg=}pQkNz?#8>6G_{ z65^?Uhw@$-6fVb>zpFSOx+ZL_;vwJn-O|XWebe|Mh_3;U0K#=TK3M}f7sYh2w%_8f zp~3R~i!E1hHd`i+#gVQ(U#%{du)T94oZYp%tsh@+#H_!LjQ0e^kQz$G^pjqAGOyB( zw%t$OI^5&f?hQkRecvGE=lH&lcX-SnZ`6((>H4jDU zm6@tyRaAV?31cNqe_VL2qdLd=QI1#0G}L;GsK?JZ;4w|Re*2eTBrSvB*v-19(K5$<@?fT(gcFK3+ zwD`obU&0^ax(dRQcKhqcNh4LPQ@h}5uIjUlD+n7jMvIp%_Cy|dr+nI-y3FU3zVs8K;d8KOZufHLl1grh(}gCLq*hV$J-?!4h{d{-A|Li zRg-F>e(tjo(g^1rP~kr@>v?mq4D!%H$8nd9ot^bsLLX-&+3l5{Zrb*v!FGvrZle_R6)gHz6;auQM5PyG*Qq;phDt)CN|tir6No`3F1 zxO})4b}4jXIqkCC@Q2u~b~?qu9F|?tgZ1(l;}o$xDZvKb_=v|%>}Kx?GXBpo#tthV zT37K`D`r1k?mSzZ0i%u_&+I(fR^g|K}XXu1>!Su8oinC!UsTO6n{uf-mJC4wm!QMU1N% z>I_Ap>OP5I&V2?Ok4~=1GhVMZfs0}-D)$GLL8{)(2R#{r#yR=02Rg6+Y~2vwijfx| zkmpD3F4g%nArXyPiblFZK!yO=fmsICL>5f+hsq(=k(7VQsg7skM9k;7y6J>8wcgS8 zZtvhGW8;P-18E&>ZHo#+Z*#5PeJXkzjKG%)*;;oy^Pco$4~uh9Nt=H3v+D!OuM2Au zJ)bcRHkYmaZmZrn_`ZJwDrV9e)F1PUY=?oSV`KNaHC&5caDWW*^JIBT5gb02m`L=* z2Gd}rnA$4Gom+ST=gbxeG+)WZhl?Y0cNFD!Csm9OSI4V{L`Ad)2f*vwi;;)T`=El$ zAjf6jqklVavvA)~+-EACw(0O=wvq5k2T2ZQPZE2%M!w-XD;6~$6a?`18e}pE+QR_@q!DMh4PS*DI0lV=| zlnqvjXi$DLE<2D`#<^{2BgOlL{h>5xngCFX0`VT5+!wnq2(K*_AmlNjFgbo8ug4-& z;DkEZ1_T&@kb89%7UjkPWXeh~8n#E1N2sbSYq-2U8%tby{{8V7kd?9z9d}<*k|E{N zVy2brcb!y9qO0C_{!{5Ciu&aX_tiyKWjl?K=wbio4_O@;Rj*A2yl@4ZTEpw^;wA@l z)WBIIpR=>gEj-fy(P|H_n?lKv)as(a(71m)8RbGtoKx))JD|G-1Mh!^O?thCN)tP{ zGh5Q6OB7*FIJ1L0HUy<+i9db08u zaa%9t$B$j%@PQeN!?I`AHTla}aA5aHOZd8uNiFy5YmhZ-ub+g9l`la5$%ccS#hocs zyi>NJKYWzJ1A&yA0$HW8|6FayZrEG;-#}ETJ!GUEV@x|K0K}$fNvT2WegC$L+R$JedrTVJFkA*m0| zGJH!lB=LVm3J+xTM4d9N9P>a|qNG#A{eBPY`_oS9xc#cTn}*2sIb0qhZp6K2^>gw; zHUU5@N?5J5c6G(Q%Jc;ReET9;^`2KkDu1`7=Zmkr@%(u;&1d!dI$oV5XsZ%vy;p~8 zX4(Gefo1;nabT~4C4Uu}yPLwcD_o8^hbZWECy{ zRabJ1TlY|k6^$NZb>&7Bh3)Idts+eaN<^V7y8x~Kb`*e zsZv4+Dt43PLAnjE3xi50&ykUfZ9m5+8t;vvFP0KbzbnPt)Ui zyN9d4c#W6kYs*ym{AzM99uV$_T@%<1Arf$>D2Y?wQ4`0%q14qYf7in>TB_G_B)7`pf?lR{o#@8SJo{0wu!GgIZi>TM{tiET}N9BUPSvmQPiyoa9RjDI5 zNSTG$X$NLj?o*48fW&NI#R6kWi5(c=FVNjiGO4mOcFAcv=L>)Bdgk|RQ8fuz4X?zRj+SGXn>nn?^qkGd;BcL7pDju|ZFkf3({5U1X2`g6ZkM#{J?AIUJ}@4q`C6F& zujH!`SrtVx@_@`I>2eE96U9Uzll9cl^q3KEiMT!*Za2s6^xnKrVIoQzZp{zVdLg4r zL;Msg>QSKJ03nB6m0pyOX}J(dzpHWXU|W(BRhDjd`Aheq(WKE;Ij-Mbn)KcDdJIX7 zZu}*pwmeQQB%`<9u%3lDYfxw`niKu^VzosIXxLH&EK^1UG6#yfo73x($85F#1lZlp zJCTuZAk6eYsACOkUnKT@(sd>mh>NcobaP7cx&y^9R2sSgxKawXr`JV}?diUJ>#^3h zHj5I{LuR%$=XMCNqhY<~!Jo+sOq6Lk8HwvK64w!Z-V zfrGYIKWrIuKLLu@OMB;opG3&fSOKmyokudidUaHRsW&-ui}x+h$~o=9)G|lRT3;|! zI>Q)9yVHI4wUN4p8WnZe<~OxL;0t8e@dez6>x^b9uDJJBx7hr^Ye(XNFzWKS>rzJE zxPoRKXS~K!zTAUW?iP$|_`ep2n!J2l`g)!k=-9Mqu{?cuQ0|wE{Q9O|o&%k{@h6acIDLI&ddm4oLyR zXr;`SJfX!$E?2Cvszrb>?^gKyIFr?|oGmSz1YB`H+1LWxjmO=(jvUwmpHP4+)6Jor z!W9x;Bue$5wC!M7-Yj$->iC>u+K{KxV0%!^@IUaAR*rm;Iov(8EKJfn>y8HcTjJUU z`<##>_1w2X(q}gMl~KQKXXUya#fj~5j{SHAP&x1{gv5sBD#1=>=YXEQo^aoLQGBJ! zLPVq`^cPvIXYnvIbPh%rEag1*?Vz0t{nU7bzbaJQ5R-X@bq%uk=FHu0b+u?uKtf>=wbNLzeImL+UzV3>?(itolw9L~!(e*tV3J>9@@HC# z7UO-jUX6_WvXJ)}cr3@&-lZdTpZKXnL9&#Zxp#MB+_^8a7+0Jd8T33Yv9S`?)yICs z%iShM5shc*Wa;y6w7z_EG+a6B=ckq;C{f(NCN6oh27QXf&}SKJ zO(eD#HT2*w1Qfy2Pb(Fb-BG!CBZPiK>QqX!?9>1Wnny@>+=uSDLn86n4jNdhx zZOf~5!iK(>`b|n|oC7s?m6;B-RBq>(6u)X9k&bs(?~ptl=R$F)d)wr{A{^uTcmlgi ztCT6G*K_4X_q#5#-MbzG1sy*4eag&3qosY!^LQ#QxIZVmgK5xH#c|#<&EOIy`g(Z8 zh}uB2S~{JrZ_+L0QzYTtzxUZbcWsBY(duyC?C!E01)6HD46Jd_iyCuW&U0Y55ZLM= za>YCnZs&gh&rZCTH&Lc%5H)_}{cN;aQ0C{>*rk&+)=|z%DJPxD^I5T0^x%a^0!c}* zVFSWuLI!OU)1fXe4S(lEUp9|J&)7<=^_LQNxQkvy(O6uK(Pf|??4|>Ypw)fmZ--FX z-+V};7&lZ89~h*SvnVdF%cfrOsNhw8oeA&;PzxT-8OL}M?6OL*ARO3q_6$G03x1x@ z@;!H6N8)Dj-p-ZEYuYsxrWHtBuVe8bp%>-Wfq*q5~PYLa>S zU7ET*D^ZO%NPTeqkA1jPjy?{~?d(}$D1e0nnI{S?CS@Y_KnbiskNMPBY^$4=nI@`6`sMg=1V0TU#NEo6)TC%q=Hb=vAm zeBoj|6YI9!;0k88Zsr##^B|c<7dmE2fx@@A{?$s@CD8Pp7_GF1#)4n%Jb&>r*JU2D z^P?KQo)bz54eh0ZZGwHS6G`&7jjCmYE!VxY1zO8C*P z<;Yl>FB16G>m>n@7Ig}H?_Z?b^=h7di6G1OFfOk&f1#v+96fqMfhiM+yNZ<-sQrSy z|JC6`Pro1Ts1%fOg0=cV9C=JCL|_u)E1aFbr+hfk^FO%AHn--OQtLiMB1v)!cnchU z+ggvN#+6D-nmHg-ynh_=ZA{`{YBL(>dQQ|j&?)nP;F2S%=xLr6Key|%mC$r3e*v?{ zAmy9@YLK2!nFxngLq7>m)&O+j-3glfvNDqppl)o@ZT+TL9?PLoCKLBUx zNEG!AAfX17Y(m+O))PehdeVRP)o=W6JHUppLl0+;;^ZmI%O!cS44e^n z(PM&v+TNB&p6F}%EN{CD)ItWsNiyQlB;h;PZTcd+nM01v#=fBHsx0jaR@`Pt&EB+5(Li6TN&s#006l0CVs>O4t!OPMO8ut+@;ZHV(Vj- z7Bi|a!^r{eZ)jGKsRPRB^d~-OfX2*>22E_Kl++ZqOt%-gi>1?&2>rtO_>wCzIJlj z(f-!k%N~3oL0=nh`U0Astc6O-lCPGj`Ui;5mDd+` zDN+S=O|2LFLQ*A(Ch@j{KM!Z@{IL&RHVYL=sdJq4p4Z-8f$#T!;)RCd;C9xfv`wI- zH~Ry;)(~WAR7C}nuJR+QIT1!Z#Z{OKhM#nS20on}Knhq7H_453#1C`tuy&;hnd3gQ z8Rm4-F0^Yc4g%(8mzfuTRWy55xXn&Ibl~73S7HADv%HS-0uZ!8UA8Cgg zFZNgTZpi)4bvhbzwS)Rje{M6vws~WT%SW(?NXOwV0pZJ|B7SWZ*nI12x|huQY($mY6EhV;Y1X zeJusc_<4;TD&Ga^|AH6okB(A~vRwzAe5a}(uHR#hZT{AKFX?s7cK+5AYty<9aM^Go7VedCOeq7+P)prHtq6af%ihAzPVqQ~TTwvVd^ zJFcfP%IOGkAwU5SWXK^7tYw*_4)53}#d>d^4gfWKA$lRE^k5*;d*aA+p|qt@c*Iw2 zU_Gwxecpb-RR(iTsR1P>&ur8{uEqLa{%D#tI{h@!f{no5wzBu&h6V`LlN|w9N6--kZdLRVPw`KZDIhqY=r5-$ zXAGbPX2y~GKh!W=#{aE`37zZP8A{!5ju_r96nqSFaDJnEM)Ut)Rm}%R)pw-beYGC@ z9oBq;cirO)Ei~F`V*b-Y5KnqCj(}tgPzl7+4Jo9ENNFsj;ABz581pLZ`sk!L~WFSLP#gwiL-xzr3 zPIXwvl*ImFdAFapCNcjA+XYe~^@&q_h6Yh5#thHutsRyCUVNLk>?16!ZSlE~|F^$~ z4sszFXQKkUo=#TC5zY>=rhI*&TV0PL>4K~;sk0ydOI-ch@Q=6(Z=reSmG*B%HKz@g zLRHua{@@|r&FB=hy1Wve0x}BR0|$cgq57%C`y^Jw@N+|7+E`g<63$K%tJSBGeB9I=ty}N@Zl)MLj7IE$Mg{X_+7sAg> zQFH_C?cyye(&m-03ZGH`bgjz}W(jbegO_yu_n+$*99Th!XZAi#f0J?avL3~^52Vt@ z)M90TPz+@13)uURN?byjAx9OsWkq}Arcorq@ta!4r!$aXuv=;|U28k?Bu8m=*jtDu zfbSWQGfdsD$ip2{^c*QtJKTnd34qzP@?V@p>F^-uK_i#{B)kbSFIUTMn`CDcf_A|E zDQhGy`zBx!tI;h!;uoU)NOZoN7H0V+l2MOX|LNBmh++V^TOjQCeB^t0HA5CaC8T?G z>%S;O=!xlMXY_zy0f(Y29*fB4Fy|`)K^kDM0)UT+DgZr*XIDN|3y$j}lV#v!0QGf{ z5HZ_+G(#4jrR$3Ey6RUb$lsR zWA0Y)o@k8n;w^_Ft+srjKY#L#ZG_k^16PNPlESphF|-mL=K;SN6UtnoFVDcx{^YU@ z*h&czdkRy_li7j1Zp-%8(0 z(ttSklXmPy$N&K3c*LsBEnczA zBacX+p5rPcGPr#u*7-dLyC_K-$KXdFuT41%s!Ka3F#A}=y!|u0e+1Hy^n4Jw7w@IY zWDF6A#Bvz}Hc-SeCe7wLkQJr>pkC?Va(MdO4II>#UI(`5LEMJZ42(|=lO0TZE;I>DF~+i;^gP6(V2GY6ax zWHAn;In>5hF@uy>Vr`L-ys~gfd$J}0oMPCvf(GewjOD5@Qd7}+j4;L^W?N)ncxKogqC!AOzR_U z`o-*81uJl;#pyqvJO-a!b$bjHt?LTlIig443Rp;VO3__sjtZGkBg}ax7k1aSsCV-D? zdegWhw+!AuNzVQVJQs@!*hKAr@-q{huE6hLtJG-ZH%-$x;?iE!+{`do;A7I6=Zo~6 zham}mVCUS?nSdfCZF*}4b|0|k_#xQ74tD50yDp9Ef6$UD;4#+zYo(F+&6q|@;C8ft z#ldfI$cQ+6Yzu7sz@R)Q=C^NvW1{A@Bc)5h_6Ghj*P%jIJk8#Dh@qB#ZQj9rJj*lE z*r$Uz^Q~DQqb!!(ZyMfi>rB?;7PlGRR74$z$LWI&-$%=mlVIEU)(>(buDhMGJwKn) z#qGtZ0KUS#qPs!F4D#ilU0fx@E=@N?^Fgig2{ovDjo|+edv6^VW!tukqCP4Hhyh58 zfG8>545HE{DLEqDl0%O{C`gwyNaxTEqaa<9Lxa*ebi?dx2A}VJ*ZS7}e(yhf?eE=t z-9OZExaXcL&p6KWJT8Tp9q@s{{;`_m+>(1HsaL@VfG)w0k4pJL4W{ZIC;(b8%nFgR zq~+t1wZapFCFfifQObaU>Bj^0x84f7-(=Z+h(Iw1tpDx((sOv@AETERX|c-#E9}Kh zTn*fYgJ?YOd98bwLrGk6uyGdMADf{s;o%oTv%rc0bN}`!qf&C8YX6|#R!%Ia-~dEE zO`v`(UmPzfwhHVFZ4hqzgO0yK`4MIkK%MO41~w5DsiMjQuxFyW-8mpalJR(A zMS`IXa1dtj=otx>W&qI7G99gDciT&?@V%CWuc2dlRQ#h;ZB<)?Rm|R*p>2Pmr`%d^2D2_8Tokn?>tkju zctQ5XAqbToPm!kB97-(vX6!B7f4dQ^CL4)3+S~)FHQw_Hb(jfHbj_P}M^xr9s2WTD z@Y}3y-@xY>q*$}Ggro!MD8#9&)^%1#u{~$%-VS<%jlt%QIvs8QQVd!^?W#N%-~W~q z3;c2(hN^kX{}~QkU0<%*@J2P+HhyNzdHe%7%fD$1fkEPhLcJ847nd0=n}Kx)Fo9FI5hDLnpW%z@|!sXxs$7f`vSk zT_?pO#!|3J4S*mo!~H~-wVY*izv&x==7&M|YWo1l=YF6@H8iFga>zr|5!PX`7r( z^$?l&DuaqjkT6Vq;?_+yfucyS#qKsS6SI5-A`WARpNy$y|CIKKBf7zNTU6g1Eg9mDc(jnvF(IhDN3hItqP{b@RKgF+9VO%L@hK>C2JMcGG&tLs!1`|1P-&#BDpQv=n=um1E z2@vzH{k#G#!ICJY%`~c{mlx9G1-lahovc#6vZSFxqs`VWhx8a(PY;*E{}pJ~U~KuQ z8)1P@yPcDjSmcNT+;msEch`txr`zE%{PYP?pkO7+BptC56}x^N6it}a)``>|{v4{k z6_uFCojtp>yYl%ch=RMOW^N`x$Us}`5-(*iR-;m?SMymB(54d4q|F(ki*r!H0IWRh zWB{v4N=Bwyp?bOyu;h4~B$7)Dt(PwoZ;`42%H)Q?_(z(WV5VjgonBA+cLf@zR?}f4 zuXtjY9oIl7WSD-pgW{CHv`4gF>ESIPgrVqRAZh7nC;hx$%t*;-yd|Eea8@^Z5ksln z|1NRu7$q~UQhA5eklms|c#Bo2%W^G#mdZ{{5Gd$3hX2cC$We&3S>qZXFDloS)l12F zeWPxv+}a=cc1c4+9a&}9Os5!l+!ZBP5&9R_sBq9>bW$VJ)4L62Yh4_$(~w?E7`Vym zN7YfWsO2zeXWsCt)@|yw{;}*kUQX zDJ4f4jfk4@fQ{Lh?OM@v_41di;{b8o_OMte_yf|t`A;AnIw@Yk7_0!^mY4YYAO~RU z=^a)#q6)@!OA= zXVu+KkKcJO{1jc6I%__d`AZX}8zy8VzESR}Q|zic`!%iBM9v9{D`>%spZaN%{imE| zWe27$&#KNv5qrDtOPr6$R#a*ok0na);y;ChQSTr(pf&0Ab;pRF3o*(ghcH8eM*M6Sz}j@U35JW-t4Wvm2yLF5d7btdIDoonWW!e};+me>SS534q`LDiV6%yU zZEolG&{?4J5qee1NxkD(D{+C+itLn`vL;8yV<~~?z7_gTs?kH`s7U@C(;*?u%K^T< z!Du>APbW8|e$aZb{JO7kzMn6?*mD%%<0F`L5Rzzc8faqQ`;mESh4A{Hz{1$kr`EMM zYr(g?f6dkXclzavw!hOaM|kKwENc)oVHp{7rRWsTrlfPF7N}-r%0GjZ)u5}!==TqxCJ3oll6nU8#8kxa#h?5 z2RI5UHG_IZ{C`=BQt?lvD6Jj{N3brFnA<6Ngc;~kD(mXZH;Rrm8CM1joeu_x#6gPT zsApNzreB-4|5JPqLQ$%d@BKEcKqBmRIjU3%VWn++t^wD8&BGSzqd~(>pro|dn9gFt z@#`0BLJ|k;O1a&-<95?n)2q>rNj(h!GL|2%VlEq_{3OgNxG}bgo!FJn84}w$3lY2C zCa5M}!f~}&?a2zolCZz9`zEPcYB8ec$m>ycg09l;p5&%tJXs=9opnP~+eOfrX?iSt4ZtlR{jO&sdMWD8Y&=HG%+lqx*>f#> zZr(5%Eqe#?#e^3@pk71Pd2ztQoA(wS{_DSLA>=@``Md02n|84&FV50{3O zPfx}1HEIs$__Y9+(8_0^t)_k9uBTNh6GJz(>l~Xtf+bQr94f8o0Wvcg=nHl6usMB|I-$^v^C)_V$*HhsvEV28}adG)t5zN>K zgmvattwEI2rLi}_O-p4l3&f49c2==I(%UsQ`~HIOn2lt4*4eFVuYg^;E)rwRnS#6X)0B5h8e<)Xe;_ zq0Zy2?)*Y<4U( z2iW&YMFlPkyGW=w97b*etW?|O92O{*S&Up|z|N!80}plqKKr3yE4T;FQXZ+=uJb;~ z=(zjlKxmXFMLO3_rfb8%jf!jil~GYMmD>u&(+X8{Fwm}nii+ex^{3Cx83$5oy&3uB zJs(dktq5Ze1qc>M8qN>k8esniunLx4;`x9_*!mG@|5rB|Y$Ofo)-=T4sjZ@fD!@$q z0ZbjxHFTu{uHP+2f9GH*g$fKHmfW{ad8!uB)0Z7nDCmL1c<aB(Z;PQPO-W*$_V)$3QP$LJSYEhlo$?!AuHfm3&o{_W(-3sj2szOVDSs z0v^6%T@4ntmQ%zvM!k|U2N1M+qUKl|6O+T)J4k8>5z6fb6<#ZoeY4>KLWli>&5`AE zyRK%)bKigeYcCql2Efzs%=PFQ~%q`=zlBS2i=qJ7xYsW&q*o;y3@eCc8q7D5h) ze?A8a$v_@SS-7Q?yr%GDILNnKbH0Z2?iK*z3DR#>tBtcWRIqvq4@lZ~f^1<2m?-#yJKc2Ezz55^LY z|LnQZzmS3wAF)pbU=Dcq&_T_N4(DN0vI3+6uj;C{ct^z#7@gk z)>2#bh%$@zj$MguFa`^SxV9Krp?!KuF1M@V9)Ppw_ME+<+$n-OM7(>gR4nqJ&{l5g_p!rs)})^}2M2&2;E|W1`}{hR1&o`wZz!gUXKp zGsD9%)!_xl;m85OuT8zB7ti5IX%&8+d1c>^R=Ez|(*mI?(7V!GIUsWy+=4AG#1nbH z0)WQ1PxT2w)4}x?XQpCb;btBC0taqkU*Ot+eIbRN66_U>l{nbz-Z|{Uz;!kT;KhH( z2lc9+qf%GX*c%lH5-BnLRb|CM>-7SV*b4nhUGA!UEYf*|>%`mC-~Kj&rtMu+=#xcE zoq7>Z-RwXB0$H`Oa~Z0&j_UTzf*D6L;f4U4Nx6`;jNab?g_GCp?CK5@n7a}lKVoH= z1!x^|cOJs#Qo9d0+8x_VQ91WFWmhS<= z_=;tp?$l@Z0S61xy!}Bbm>)2%fXVIWO@M?v@OqqC&@FKLLHU>{0NJD)Bmkq@5ow2p zEbiI#1OVwM#8Dun7IdFh2R>Q>UUPwUfYv{n`|)^Xg{ZzMUTAl+)NDSGs2&vZ0YY8u zCH*u>TvCq4))HSIbYGE8d&KStB(q1=>47>GMAnNxUlPErCUzO*n*((3x-KAX!BlLa zCf&P<=jFb{MaFCbrdrI_JVXLe90UyV^h>7nNYVFdYUlvmYCr*pTd}MgzH7b;{$jl5 z3S4Qbo1dOppA5lk=I3Z4SC}xMgwNlmTdunT0Dt+MFEHi;U>87#y0eb6%`Y&?{u|QVNIuti;l`0`w|3qXzm9^y-=-NdkbY zsF|_<3TrwSP2q!ddt1N&?r}0svwv6(`Dr&Dbm{^J49EpHmvG0^y-n3ub`|Uk%x-Ij zdN6psY3(O7zkqGe+f-Tr>47Og953FerXZKtrg`%#e31z^Y2Z_UX?Ou-*1xPZ+5%LV zMjhXtEM}R>P7ba2!B4hYjbs7cD;$udE@owAD6{vF(RigDh|{P~1NWVr@RB(e@vXXj~~FT-xIt=Iz&epDi?2e73vowb{u4vZF{ zOFsle;_&tDVk}(_xTL#Nw{b82UAMUyR%C+fo+hw^Ygg-h&DC?wva{sTlZ==dvI}f* z6LV_??ZENFBV_o*(YJFjf`g9cLmg-JdA&e3tilABnBB;K1s<-hMNQ-51U3+KUjG_M zN73DfjwdCoTZ^G{5~yOa$BKLGr)7*MX@F@?4KW@fwQjkq8Xg)FBIH>A(Z-!EG*tPl zNV;lg78};WeS$uA?(neffZdnq5P1|z^yx>r2_a4IkyjS27bqQem~6KNwI8@UrFb0? zR;Kx&AZ|t>zMwmQ%rQF)tP zQUk7%g`0Z3P&u$Z@RMyd8c(aqnVFvAhxGS{=XMar37}^df%oG>lpU2Xd_F2=8Eepi zE7R~ne)|C5s$_-%GU+=l7Q_7K~ARP|doBs?4x>p?L{?WOWaiDimx5_aa( zIe9D+J1x;X9~LxhH!1>bBG(-PXrN7&X07$OMqN2T)RqU46RMg6I@%iLr#qd~8~i0%S*IOoPjXQvv1 z1>tIRaIQ<(z2zgqgZJ6}co%^x;XhVEYZjVe-k6Z0Xcci|AopdTs&>>w`D08tZlks@ zDiS_$8!K!+MWdWLW1+R~jvi*3_VR%5-FmJi`C>-=OXrEjknQcRT1wr-GlS!8Y6O?9 zj)H%y0-^DrGQ7Q!JHP=MBGd_7Yr&r3fW7`dXv6#w5dZIY%`E%v)9sle2Dtm~V$@)5 zSxS{T;MQBDlkfm)MvuX{3jfsPVp64IR!qLZ_w}VbkqXj^*`4r$_ zacyuRb^J}NfpB|T+UMx}^gTAfH&u8Z{MGtCp^1{o;Td2LB@$P$-vTBXAjS7|+AO$< z<Grqycw@4MuKX^2=Fh?etPrQVzRj8XEJ{NvNv*x zbw?}YI{R~xx;#8gS8*K$;rwJeKtd;!ne^DJ0GyBH;bb*&Fc%33@Q86kA!Y|m+b7sc zdVGDrZp+QXS3Y)nWRh@|_VWVdm6%@LAJbI;YiVPIF`*ODn(chGww(yiqU{|5z+4Uv zYr&qGgl#@iJzVe2r@w7;?7S}lB9rB2%rqVH^Z;=4iB?*gwddD@?qhh0i* zE7Oc32aw6rVQ}Min-O4dv*YB-Af!0>&ea9jKdwjohw=4B3d;j~@PS4Rq;G(Wp6`#M z|Mz#1{@aKbtSb$!|2=MGa5`?S;z_9kc+I97og71K5o2x+BFL=}m!`Mvlh|`9_ghR$ zoUCjc{n<5GHD4f0OKj)!!#nh%>3m4yTl7r5PQbbW!eRchspv^O4}qMr1#C#v0db-33X(Ck&4y0l)p z-E5}Wxyr2~et(9QlrA8wlToA9|38W3;`|~4o-WO!%?}I3Erxm%@ zwl+#pdQ8dfBRvNvdb)8cX&k}>zCTpn@_kszam@)VZFYx{_E~I9jK%V9khYcvv?y;o z0qIM0N$(^flJ3*$qxC`lSMm{R1Bm=wU>XNXCQu&4v#0H`I{XQ1fTy*lS0>YX5lIVu zgH3Im-tRf|3GNHD=T$@9!60K}vMn}E;+33F$&Oa!|0P>_>+QMg;~)PX->|cw8t)pC zS!!aNSWAws?%*I? zkIYK=@jn^U|J>gPyj$)DF3(jQJ4LIVY{md(6kt?nB3lHdwmU^?ng$Zsn5lAc@yL!0 z=OqGsn>2nLQzdrhL*?f(e?1~2qxH&eY~LML+u6*a^#XQo`K&OmTf4}Cc6B2VQ~nP(La~P!?U$OWdKQdPMwW7GO)8Z- zYi{t_`0?_Y(D1n?KKfk_2sc7y^YuI$qNcTXQyGEmh(?(VMM0w1*TJY{MIkKReU(&6C8Ht%@IXFQGFf-*x;_j3pe2|vG1x&UA8*4O?_`sSV zNHhZ%4*ZJ-gpPG7SZ;IMmdAg;|9FNp{vLV2T4ughh4m_yzL=Z*j#8wAGzY z*O?x=AC*Mk@Yt{bw9Hl`g(rN9$FilkuL;)|T8J|^@zxzE(?>d&wLdWXQ7hOMiF4+MNA_Aws z?I(mO#B)0g0j|Hej*g#@RGaGsuT7r!bYBWBBSd?w__sn*Bb3eFmzV^U|)(P?<-=4bPrMTuU24OOwF4Fk(Pf zTuG6@{H8+%EI{sYiA--#2O^Ik)^hUFng!OYakH{&H?b?}&7FvjI>^XaoQP*f0sB}m zN9IEYIy%^r*Uia&9eRc|kUQHJ73r`>MtEZ9V2i~T@{_@NnFvtm4?+J9m!hxp7)vtf ziU(f(`nP)00bKazO$lI}#9(&UgxUNyZ%pl9Xz`%A`X2x4gtPzNyFHKC7^CRuG);Ag zd>pA)J|4OtU3at+UA~w& z8P{!%$D_MC!elkPiFhOlZZ&r4lk*R7INC6HIWNEq&(ABJ{!%)5q4e&7byL&K%={JF zDqica+*y(ae6&9zTHXwvMVRp3nlLcluX@O?h_Bz99oC)wrhzzE5`~ zpInToh>CmYRlvc+#mB82GxOSn(d(m(0vij{>*r_N5YPF6-SlnH43-Xu&k^1FOQEI~)uk{^cI_h_@;}4uMldtR2 zK15%ADm7cBc{=30iHhp$zpljj#o*eZ@j(rbU}kRq8tt}ucoAk{d8rKwdKFDZuHOHV z`ou@1dD+69eK(r4W2^RQ-Q$le_pd!8`N&d#)r&afBJOo=26J0Cskmu`6QPPM8g2ij zsK#Aihr|9M=1t~lh4{soU=8`rt5;% z{xNY{;AVI7aYORq^i^Wh`g)L07ZS6PI+NPp!6^Qbd_=J6@Nkx<1^pwLG$~TgK7;+} zx|rd)XCz`T&)qA+yMIk7YVy-la!g*}r$VByQ$5q6VOOakH^ogC7fdP0Rr@L1Ae7mV zt4u$={PW_8?$5SPOAc$wbf*PL$=wm>M`fp_=yISS5%WplL?-MtHA@e_M^E;Ps@8i$sXEzXH|c1+QV9p zRQ$TdXu?@g6f{M%XAl%a**x-+8ivc=5%P_`O|?fq6+=}2)vXz~q8jWAR0^*qMF(9b zK=lmPQQE?q2IYGm-LTu*>h|)H8(a6#fAM0}m8~|0$K*^uZT>==z3 zn`(X-YU+75m1@86S8wz4`%GRPQ)cy|X6qyorbDxR7;k3Jlkh<+enYr7lsG5iocZt^ z++N3Ul+h5}HP^T7X#2Av z);lzBELka)cYUapEViCx?da+$E#pS_mksX!5=?&(!Qs1k!*}U3^BGu)okaTe>t_}d z_CHtd{L?R#{wO=}ez4e+Gp7%{7CV3O-~R*R1nTG3dt(GF^wxJ`8?_xA9rJQo#$$M? z1>8Ss_ceu{5L;U2?yU?l!Q-JyZ-ls4wAMQ6vEavC!hP8Z%v(I>#EzXTbHYrI)`5pE(>VAu$*H0+BuBvKPq8V`a(Y=pqs5 zMkjLeYp`>9NkAxT$G2T0e`j0pk=M(|a3=)yc-w-D(01UwE!qNQlb)JRLQJfwsl^Mk z7(pm@J6UN(%=0Um>o7a^4G!{haO6X~E2AXKzh5|9T3EP)X~F(&reQ^P$|MZf>(o|QV%*&ga&q(e(kc){44zAGe^h=i~kBB3uwzX+5qbj%d6?HjI zm3k2S*m}Fad@>hMB#J($<)UK}G#<}R>l}}{rE{++o4-i)I&;!!{$Dip+A#zJo^y6GGYH}D{@j54OLRASgGcjsPpuT^z!-ve!}W9_EyJ? zjf`5I&V{5|8{SkGvlXbU^gNxPK?J!R2(Omdcs}{Q#6vM-;!C{7_+VU7PSAE!bv|$v zf?0t}`1nMW$31`kJctHTN#Z`OwH^>Ani0jWX_|y!Pu#x8U^JKOa^=b=zb-SgMN+nl zIu&;wzg|cR5_JyByMEul<$Y>%P+~_y`D2k2+MM@O=RSa-^O5i#q-vFG6&Qsw^PU>DYvrT*Fwcpq+Lep@Yezev zd#@idDeP>Bv zabe-dj~}ko5SN98jV)4k_~zpLvO4PkxTg4%WXg|^kN3Tfs&^^*=NvZ=JGDnKLpxnF zd@~-N`x{YSUe82CYGD~Q!zmUfDk^;y`}tlM6`C@3g+@uK$= zjSoCuTA5RP5Za~NtI~*4t3KroinNT%TMn!HS7=yS^AIE*@!T6~mybrOc+6jFec4;- zaqNUmYH4W|ybv?YqR{jrPMQI?(*9UazA##5tw|pi&+n3Lm?&~|qqY0MbMsdTyJ+m} zVD6{FkdSwEo(-4sX*|{^G_K8=TUhw|j7J!avw;&OT?hx31Ts*Vi`EmX@#;{NzN?=?&@>-KfTA?Q>jcq1duou#&qnQo4F+dzHs?B~uyM zz{tP=C8$fzOB-lUsgBMv6Pp?^V03@*nMS2XQBQ9oFy76JsMgJihS%^_YqZ1a*hWN8 z18j8yi3y(1nU>K0605P)w$fYK(Acg)4LUYAx467aOoF|x(vtI7D%Hf6l{qzBVhefE zqZW(+D_Jx16r4(}c{n~96csCKU-s2wX@<{>$WJ?int#P*svyH%`-mEnG)VOB$@fI| z!v~Cv<@@qFMx4WsFvZO^?rUQNKrV6faaj!4TOAL*AkfuS{jAcM`uVe*+H0l8n>p9N zKL=kDL8ZxKzCZ*P+LI?2-xh&wo0^(>m6mdONMlXl{n!}uJCqmEg~MIr(hn{7bJve$ zWMwgiU8FC1IQ$}8G8e!PkTe>sP)ieS*({8{BEsWx;KNf*dgf4c?KM*NFw7TSh12z9 zu!)#O&h1o`qt`?(`x$Z`Knz|_^V%G+@`~zrVMJMGS>Bc-e2 zirj_Bn8-CCUALpR>`l5?M%$l8@T)Oj8lWm%x(P%wKJ!$g#FLBPkepn__2}oPKy=mD z*I&xt(JnQAk=5YsJITz%1jrP5EPl_fR_}caS@`mWoX5PPs-LDGsfZL6aT&URTf5#i zkqAG(Kn@;`XbvlxC5dAIWps@o{ImwJp2f_?HyTKb5P zTEKaKXft9thSz%L6;zv3VZiT+qh#*+zeq!#8>(Ry0B$8a53dfD!kY7I8?~9<;0_+BxNVjSvjn86Ft-n=XOrK~RF}c%T@9eigs#E(5f^KfrM*gIO znaL@ppFaET3cASPbHKYWuDc6NHYIs#eP!yM3RGhe5|%^X90!UJv-9r*t`nHl@G;Cd zy#(T>UE_VY!qKJ43I@2VNqK(!46BEm+x}tvhKR@9heh3d>c9q|EpG@74UKSZqlp=>OsOvHnY$77EHwKhFT$WLBGw2wLArKW2E9Y>&=@X8yi$zbv z=N;l!bx%x#z9PH!sMaK_=FI-&`D9G=)^kqHjKH-(hO53Fu5OgO+gIysWwUAx3q$~gU=KqJF9%N2zxvG+qc7jm0RdG`CZeVlm%Lp>DWGJ zw9U?VaO1AOpP$Z>5W9Ab*y*F@!oUVbt_%}nql383$!ZLdi6_qJnS1?u+MB2mu95jO zGkH1f?YA#q_|u#+ssd$LfkF}g7@GVdyUKCtVd+cFmn{Z)&ISq!fr%K+jc(mfGeG{h zI-y;s3g&^N@awDF+^XDkH6+6th9jdPs zZuQ|$iY2!$mD1)s5i9Dr zzbC5iDvNaaw0}$7(9k$i1atiU{GUN6tj8*yY*B3IwUU$0C*7eM8>QUXuDVSf5*B9g z>XnjoO26arQO5nU(r~0WbDb*0CG>kVln5Ah*<1%fKl^&ABjdtjBJ&rNcn^a55)BPi zr{1@sWwz<*QqV~+7&1s&4fQ56^x!+hw9ovFcb;B&`XX6lT{iph<`uB|y@>G`Msm;NF+B3j*;weAZlh=%EQ<^XC?&f8Nb`Ur z)$ejxOms|vfEd=0?^+TR#HX!a^NEVPL++%3|DxQQsx`U=UgFHUZ~UB9?X|Mqy5pxx z7nh?pXo=Z)ef^jYexU&>A;WQ^(gY3~oRFYb&PJ8vOkgrUgHK!HrEVrk98GkeogGn7 z5Ud*l^wUQcVXIit&iqx#Q9QruF(i%gIV#(%*m$;lch_m2T~DvT^iwaQe;H+eOI)Yg zv$$=*yKN-J0ji;9y5CQnI8iM8+e2E@KjJaP$` z=FxJ-1_rCkj!KKnwE$BZEw{WQ{-ddJvc^%HIsy?$1&-kmn1wI26o%Rhxt;!AtG(<^ z1axX>;=O#CK*C+|kBnqw8t?NT$qUZH)baI!fwq_te-HHe{&Jept0eZ*%1K@COf!Sx z-D(EH-BNa)tNXweIygA+lLof-U~li$N-8kZ8a(LyB}w`U?400*1HfV(0j%KCUh72ub~DR05Z zDRoxzSdFEtU8DW+)h5dA=+Ma6Xm)01KT!Z;^BpDr(T6}Mg1SVB33x>(N85MA-D_v6 zq2-w9;NX`^N?(re&DN*AJ%F-BVyK~xoY8Wp6bI)#+#+0;FEk4hcE<3Mh*JvvqO)-^ z>iD?3X5~ecGia?-<1pK)U0z&N3?wbuOVS;=)_f7l%F4uKK9>1Cy!?Us3Ay#!VE@E~ z0{+H8IWwV~tB90zYiFnbXFt6{*V=MKBGGFxJUKlZEqpUd7c@r;jOBBUNt#4mK89f5kD&CHnH8>YDMFyztFv#D+Rr^{#F#H6oj$$1V{+kLi!vWpmDal0 ziyqAMbpw;TPO~c8xjNn|&pQ#G{d+tPa`;B;ut5^2R`~cS_Bg0gh;CO|ryc#`hr3E=LL5A|AbaCCV7!*}_kUF-%9W zOKH75JqLCZ8O4yJ&5dbBW+di`nRzvDo*o6nCq5Xr?OJx0S8WeFDpfMN69hffCpaB9 zDQ>sb@>u2OsU!C1riZuK_3Abc>TO8b;}|DxOzI#!TwGiIi^afR8>ush>V=kCPUO~4 zzB=2Eu5-XvJ2!9@pk|m}0NL>H*k;YI^whQ|ojWn_|C#mv$@pmL!EAFWd}Yj@VI6eY zf$nLF1c9zE0ol(yS;jJaoN0cM!Xih0Sz~n={<9n7?(Y6xuEl(G*lD@fW$epM04p$c z{Sbq~TYq1y%Qy~7T(1POI6lJ4z(5z7&^Pt&gq+Vhb8xeTCT4XoSDNbbQO3^VVvp5$ zQzBP!(Q$V?8n_hr$vHVk!z)2PH^`i~D`3Fhe3WIzsTmTHmZn|&;N)nR=>Y#{Hn8!R z2NMU;x@)8ot zc>Q$q2wit#I#?rn?wubOvZ|^I+J$idYZs_x-!VeB?e&o4x}c3GhC1NTkAeZvgy^`q99F-iZwAadr_P&zhx{h1hWH(;4^LQu*3&) zt|k(y4jol$va!`XYA1+DgwFD!qcVQstJ4)dk*bK5$NIhf0C*#?Q0KPlmD-WU6S#KI zB%~P6j%2-NK37KO^U37(DxiJi_t(`kt5JtF1NBg{4;?6lc60xrPCg)~Y z-o`zzKd&VHFcXj6|H`z4qQ{Xu?K6FSePd&FIm@w#Bn#glYDIZjXod6eHjFWdO5}a% zUk^Q)PhaP(cBvYRuCK0(eyN;-_4!c zzyvI|d;9x*zK*<%XHr^PsrfLf;nIzQ;=Ht+91A0(;ffa@G$aV1TjS;Nf)yIW^^}lb zbY6Ew9bSzNvichq-DXrQd?(g*9{gax)Zgo35|gd>)pcbq4vy0$|8Bj+V?EcyhDprW z$xc)vaUxS#tHME|$lR>`ytT8jae5_4XC<+oZVohGSW>PTZLzgU&47$mx#$7_nWlQz zYmBhHt1Vx5)LLv(58S0(wPgl=lP_}Aoq%+}czz}NYAjEo5#{>f(zX>U9k}0-RZlNO zFtLd~!ThRcv(3rN8t{1r7$gFD!D_DO$pKB6P00u(l`IC#z!(G#gC+23kn;aH^KXx)Xp0p){$_o;{-$brP3d zo`jlY3UMn*Yn5BFVFCr=zxY%h$XkwOKG{gzkJ#?qR0FtJ`hy2t>GF%cSswxeyMQOg z1F@8;NUV6XQ50Z-WIpO{pK|Qgz^BE0wJ8j-{ zdv7mI8}vsbBPCc;EkEDv_}KuPq=h1WNzn87d&$@OuMG)$M>nXWIacF;bD8~tA~f>I z$;i8$ZDfFH8a%2Wh4|}{D8sYojrg<=6z-x*?Im6OKX_PyF&wXURPT`dko->O_lhd4 zqb1pGmxl2ARq(^h%WK~MtyFs2Cr^vlrZ3yOH|&;;x7Vdh7ja(E>_mP*O;3s&>8KV@ zh6g`0_sUvY`#UP3lD=0Q?`_6pq@?=$`j!Vv!)nrOY;J%Nsmthge|$m(fb^>_#o1nu zu@debb_}??G-HeiTB#Deyh-^@M9^2jnACtM3k{yWq3J_-NJKm<{1<%AJw$FWE2a8( zUzvE~1oh?(8ktqZXXk3x!SoG#fgJY>MW1MQ(sTlwf zEKd(tU@{_Yx#(A{_e-Z@0R{lJ{v;dpgr3e|`(yH^HAmrw_{XLI^60g6ui zYtnhKF=XOHxW1VgEj%P7T5~sXhV;(EQs5#*PV6L_Ag%yxnLk%+cPBj&UQ2e1+?HMW zG$hR`HqxescVacMI^6s*x6;KKn8Mwkh;pL)!oMeer!l8|HGof4d91&!+nj&chJ388 zsTqE(4Xm}_lbN#Qe71hQH^R%T^)$MNrSX5k=G>i}0s$y_v}@%LTj(2|27#BPobNY4 z9`(pjIELW=z8h)@UTP9u*+pJprRhn5K<&qGIp4+cFL(ff(X~keyjj{GUbZP2btlG+ zfdgZCi-+i_jMVh;8pk8!mQ~rP3+$Kw(!Eks=Gw>y;*&Z3$?yDrFDb(ZCAo~W`mLv6PQDI_(DdI&-Z_z$keGauBBQ_>+FDbiSbMs}Yc+8Ja*2@of;(%C(tsZvDkoPx@w1K@A zw5DrZ!w16Q@<%tQPPP!Wo85;IBHT)}OZ|mW$d;pdnboR5Lh+9U)xQ{oKuCvJygyHS z-pqcq^kiOn^mbV2hqTw<+ebAvG@-JlB6-?mVZfFVxHbbOTIu4S7PW;HBt#bX%j1k2 z=Z7NeMesI^-A3>Bwl7d{x1m%yrs$r$;>bw#met2LG{`nRpE+xJZ=%-L8`++wVg{sa zrzh)AFCQTx8!*#J1$+?ZWqJIt!nwOZ=oeU&-bC>u)j2|K8@EL{qdS_KMo`O1z!jFk zSWOnaz`qFs1b{Hk1%Baz2kkRsb@f)w2#zaQqYGycsRe@pds`)tBS;ULjK`pZwM?#F zyT-*;{QCFr3m@{S1nuNJ<5#x#(z)N{Ar_{AOk4lvDJfhFvH4``nOKTu|GK7u|BEDeC)gK$jmAk!>NG0>4RhR4dCnSOid z3Qf7OycdxpB_(hXU8wh*vgR$#%#zhs5N%)n!mcCiu)Jkz(WGY)8;?_?x&>zP?JewD zdJRx=A*WT=VQVw9jg8NHRj}pUe$lOytmx?I428t;Is?=$RdslJ8-h|p13+qs$>Vhzq})MtCca)llnHrUT3dbi0^yLxr-h3fWL?@p+cVSmqqm1|WGD`xP|K*%>FsEH@YN!Yhq*Nk z{jD}~cOeq^FPiB8Lkp_Pciq=Nw=|b)l2Fj5AP1}i24EdHFM@z%%v}m#$?`cknxV=Z zQEXI>7X1%PN29hNR6h}WJ<}5a9i|p>H#9XhH87Z6QbtOmy4F^;EFN#yA0#FSzPI^cC;Rt1)@5Zn&5)exZZc^#%_46~FGfoFl zW7zVT{ws_qaJeuCgXUJHW+&MoMtOLU7!;%!d_~hrXcUO+APTPfA3t7>_slBFI9Lp( z27Wl{JKWx0D6^jKj@HyXo$uOPo$X1p_pL032)} za*HI%y32L$O3uoUXFybmYjH+@KFjjO`fN&R+y6u?*ie*J$w@P9_>Us)PnmaQMxE=AEZ^6mX-Rf)Rr zeobP`ZF|A?#fuQ{N=+a)SI0*IGNx~8y54J@vXn@b&G~LK5D)L0{X1-amY5odQGN0Y z6nQhWwF-l53(R(3&2r@uO_=Vn8R{`$d`bMxe;)fidFkOIS)u!hwd+e1=v=MhUn>Xx z($cL00s<~BF2G0cN=ZPvha{_Unpnx37<&t*0HthRRz2^NDkp{<6zyn`Tq>u57N z(lw~A2t@-(?bppcw#$D}?d@uAHKePvj)N4Z!d0+;u{oh2OHB<#>2uNpI=`KItgKi< z5=0>E?A@qt_%US!Q^NPSzIa>Nq=sgbys z(m#GUG~8-C{F~j%$|7RCGOi&aJ2P_uq#1nrzuG&~a46WmjqC17T7)~xjUuVUB*~Vo zDQ?_|$xdiwH@57;kU}C$)*^diY?JK!T|&08n}m^lUlPVT^ZuvjdEeuCzrCNIJ$q%}J`FnH-g8y3 z&?C4iBYC7tGxEb17H|Iu{SGuhz@7zhz@Eq}?*r|(r%b&?^)y;SOd`J3GWEWIe3Y+> zQVT0PYjtIR@{I=$EnCZKDak_Nrh)ul5o62oxZCk6-HA&&kI0lTRAqp~b_lci(%g8n zvT{-_zlV7h02ISs`ySHgfX2!wE_TUJaB{8_{fH~F*qJS0Es^yfE%4EfmLuI<&e+Ja zIodh+ctJfkI&S}#E5!;gJ}^dn{J_G^)m5?(J&-#d5E=GdwRKWJlSSdpjSng)6n;Bn zy{lmhR9s0>!76VqE(wYT-{+UufTUM{B>Jsl>i83P2@R`Xu@c+6V83<>cgR*&q=>ut zX@FY6ZoKJn3TlV|tu1{Ys^s#UR8zFqXrYK*SRqD^AFE^gePMaA z!o2l-lTnoh^1uh^tO>X9ut>PU(&NiDFm)3&J=%F>c5b)5*$F?Nm1zvn)OP!$>{N_k z5u6PnSy>xOV))Y{Q-jjCZ{N{Spr1(c)?1>0<{$aC0KN_ujkyC^lX+1@gm%p`9(XA# zG$#1hrGHA)*<}22fqya2yVUtM+D**O$waQHWT{dHhG5L3;a(N+i|U_FB6J@toG zo)pnjE1jle*z92{&zXCKxZqCWK#S~~JAu(7ZoCx6rL`Gx+3TRWaDu-r%u-)JVYJ45 z3%verExZx0Act-(_5j?9rRC*sWsaRJhpjFf2@00oIq$5yV4^A!518bXNVWSy`pQ@y zcROE&qJckE(ON%YKaGi3e*Gts5;A6xe)QTz0h z=vUR*qIl#f+kHFa^H~FGikP~r@7z*}{uP%*R#&~^Vq$zYzCIV5y;^X;{%f|sd?b>N zuGWFp6dns*jBSOn=eOVRP86$TW(5O-3Y#dWb=3xHl`p7X^z`%=Dom@RB{hYPg)d%o zz}`jElz`scD?<{}_MX?~Jjb;HaV1~vRg8DEv2fhnR#gV!L3o$~%F&MJ&_XA-equ2l zl(-pxiXND_Y4_!4I+AeibxOE4v5`K%21-l1JrTyUOB#M2w;JxV&&+^wg>*hR0AmQ8 zmQh?$*2Kucprh>6Kpco%$`NO@yM}P*8qSSV4nWS#&PM8bN$+iK%?%G*G`F}pk*ekQ zCOM%5AgUbloI?lfB}t48Ie0vo6jveV<1Qh2z1B_2ym3YtfErz|;3cGcKdxx96z2>! zR_^=vYg4#BZYk%8ut=;|Gu3=@q9uY;)bR?Y#F@oq&=_cjZ_zqp|IiG8~w0>!T zqb5^X7t7yv_Quk@8QOQZzZFCiDM`v+?ef}#r+%n3>3LtDZ~YZDs<&VNRj#C7DZ_3=bSEEe&Qi1G2TH639!)CHQ89g4f`I4HElmxcso2 z)x3SZy#f1=?0>|R(7Rogu+~Ez&pYGX9(z{Y@UWV$i9*>7wKV?S$35+)HVNcN_zTHy zVu164G0$ktADmnvDt|id+u0A<{Z;7YzP2U`NFLKW4HsvkvbwG5vFd72-k|kC8FpWs z@o2)wYXw{TLW$44`RkzMF+aTcC2Q>3G}Czr_nQY(he7)ZW}au_J)q2@6+-bG9eN|9 zUv8lr&}pfRCX1*P%8yo^oSfRt>V1+4h@V=1H$ki)t?|T=3R;tONDZfXTtLKatr2K7 zJ22icNeKF3L3(7L88(ZP;90Som^iKZMh-F(UTw)1DpAo=)%|#Kr$Jg5#o@Nu z-L(YAhIW*2HZUjq_IPIViJ)R<)b8w=++TF##iNDM)4Etia9GpUk%*w}?Z$anv|`9G zrCUPpL#N3JU%aSS=wbfAsorT*TTDl12Z`pCkXzE+Rvqi>5ruE&<2VuIvc3hr5;ASC zmD?GGx}z2u83|U@+ZA(z{?I(Z; z&H@X>;l0gqx+mA;8f3INIdF3R>whvlz+kjpdy_Vkw|Hgl)mni*?Pe43*5yiaa`M4@ z>#E?Ys{ORlYWFsg%i9adY=MEq^%KWg_1qX|OfVh#%G%xMWLa}2xVm56`zhj)ZMRi_ zq3Lt;PFmf$pl!>U4bY3)xRWLAN?qp-Kw?){^S{ft`j@e~J#<#n1zCiQ>>y5TXOwB; z^os9ZDvw;chzcJa-F+e2BydYlteaR-Rygn>csO@lk(E4n!;50>@e}Xsd*@UVM7xqP!j3wQ+S`8#cV#PvdYBS-|%Kt!L{-_ zAjRimTY*GVKDF0{{aIF4N=mxHckS{GAsI0DjDgwG3Rd{ zlUbYbk)lIj(Hjd5Qw7`k%BGz>f5@qObxxBRSuNolx~F*MuLiBnk(x$EUOa7XPtPpk z(#jUsUO&&)+Jl(aLQG5lFMSGd!fHuQR^xf!~n0vzn{Yh!0ur)$X&^8)K zv3*@fM%z1uJBii3;=X91f+7SB5~#UZCm&X^I6oi3D~kcehHX8v+H?7B?z=D2veH{i zJpub%Y4aL?x_u)wdTugJhG zuYzmp{ba(RRidwP7>mtTWoMs(h9pFx9~cd2LuM{;YxS%wne4{0?QYIj?Ib#~kk2bR zJ(B>xjK+pdPkJdci2BW)i{$aPmtqAq9pZKGMvDX#yZnUDm^&p`m^{&1+|=~B=@=g{ zL1)gM={J>R3_%#{TPMJ)e&6z@E}T7kmY+WhxY;kf3CS;CL$U_3TrM47`UX15Qh@_x zqOz@8Pi5DaeUM+4-0-o|jj`lfL9$(^{V_iFe#-2x;^N}kSnPA7W@J5!Y@*G>DW`U zi^MxgiEpp>Mti)b6>Uo^Jxo#wyMm>a&FO#pT|a*L@)tu-jSU$HZ#~bhWWJ1bF{8$i zlV`p*R46WgQ&v_6B$$B@Sb?@-BEG-Dhgv5HjyL`vrhDIGr4fn3`XiV-G1iT$wMG{F ze1AfVZdDBq4mL6}g2VT-b($Jr@(o}6&qETT9HJ1~zp)V$tA@27u%zPNv*?jR^T5&T z22>1{a#L)wP7tj1XHh=K_`nta^72p@4mA39^qabeb|6&`@FgHHg!q@NLi2xXM`+i| ztS;K2q!3rG=*lx@>zi^wsNu6r61FTS7Oe{XF^>^TDT9C{5lN+0feg{g4gtOXxy*7kh5Y6S;IPR2; zx9JmmZZoj3a6_%TV$4UHK8Ywi_h|; zW;?$DJ+T?&QU!(_b0py|wm9E91wIc*O@X#<>7Ia|&_6Jsp`kujKZ*a~J2Op7*)hhY zxw;6RyCz;x@N=8S2w+7kyt-EYH4;`RDCikoi+F}m%sI6J>l*xIGOg*<{dH-0YfqCe zNXJwK+#rR}q7<3A!Q3BfJy~G5ng+JpnVFpny1Ypdg0l~>YcTUM@@drWHk)%-XXo$^ zk%FnA>aq?y4zOp0vx{^Q z2#;opO|;zfr3B+#Y73tn4j1ab_eto6SzFpMUial{?JiT_cro>sEpTHXk^%=cykd5C zc6Hw#tt7lZRKfv!KzPOH&&??SnlSWKJ*~;q+H8VP>Vd^W=vc)*)Y5?`!0&uMW>Eno zEO9xYxk=z8@AdAc>{O=+F3{*zTbYW$r1horLYn9^GU{2KyY$1oNK@$2qGDooX|LRS z$?lw9r3T$SJ+G1EfRK<75Rt91<==<#^|l9HSi97Ba*gs47H6N`tn=~S+jiW2WbE&M zn_6Gj(b;KkK2g5ErE=@L9^dCFDS^xS?Ue?~6w2(1W5enlH8omVszhfLA}2w6WZ=}_ zBqiKj%pyw|YPQI19WrnN5VVS$ix>`iSS@1bCGc5oM3TAGkcWb~Ws-Nf@lb{T{vj5d4=FXWU zX}9rl!)vf$%w6!@TBl&2O4J|UiX81_Ywm^6(?j}ZoP3^!!La)&hEhka;MUEnGG_{$ zv@cJ38y|sE9BGZYLOZ^)UYG#cB?8=c`{ALjxj`c|8hQS@E2NDf{*S`6&s5ni)Kxi}BsV_J`&MKsTjns%?sqT)Lj8|+e86QHxSWIK zB{dHP)0ebL%Sx@W8})&KlvM1qnGQ5mg*18zMf0nnkCaq#G2W_rz(Iwpx(%7kS0JJ z-%E(VI5O~k7U0#4ick-&J=%JNcQy!t@`c(f5%Z!Be63hgJE?^{7YEgJDP##>t2R$_K0 zEba@`VfR;*lx=1$lapUpN!ZKyOhtodPbs1M_O0ZmPXk2u8dajkYT0(z#uY6rCck-> zr^r?@ado3gropuT7}IpZbD6};!}9v&%M5q&BQ}WFeCp`fc($5&utU5FHuUr7s<6;9 za#guuii>Eo|Us#P^RxGYOR(=NQVgTqLH>Lf5dV7eq0+<~-$qhSfnxB-_$Rn6bb z)~x$|VEZerM!?lphP?iQ;RY6~H|nH3+D!hGCQ6kpO7SB79UeH5KGrQKpGHF@L3#4O z%iwf)&WjF4x?7ur4(;Rtn+HxBlu+T{bYlQirFI^C8X0luiY$aaQNWz5^i&-sRFm|Z z9O6EyUR~c04Xd@Bl&{}j-D{ZUI69I?+q1ayS?TD${J2YdmOK64*SghoW^*%m1WSkR zmIdggHCsy@3Z;&tPiQZz2(kYv!E~v`Z$?J4S~N{+{ghNxY(BrgUBGO6%fIS*7!mi- z@%a6d(1jBH$Ok|pnwl;Q4dy{!!E4cTZg9*K_SV6^K5kKmYxxY&@~-^yIQ|xkO-Q(j z*-s7&0J@`}P|luzgbT4F7q7kdK6A}Hy44cYVh~XU%NFP6Iw;=CXOH^ogMOWB_$#0L z*yfJ0dcJjG66}CJlVu=q#_4b-=G|5Mc%r$j&9x~;380N2l|OHKM{D0Tv~{!^%Cq_j zP-b$K(7AcrL`7herobdnv literal 0 HcmV?d00001 diff --git a/examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py new file mode 100644 index 0000000..343880b --- /dev/null +++ b/examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py @@ -0,0 +1,112 @@ +"""Wan2.1 1.3B T2V with tf-kernel FP8 Linear layers and dense attention.""" + +from __future__ import annotations + +import time + +import click +import torch + +if __package__: + from examples.wan_video.wan21_1_3b_text_to_video_sol_fp8_h100 import ( + PPL_CONFIG, + configure_attention_backends, + run, + ) + from examples.wan_video.wan21_1_3b_text_to_video_sol_fp8_h100 import ( + get_pipeline as get_quantized_pipeline, + ) +else: + from wan21_1_3b_text_to_video_sol_fp8_h100 import ( + PPL_CONFIG, + configure_attention_backends, + run, + ) + from wan21_1_3b_text_to_video_sol_fp8_h100 import ( + get_pipeline as get_quantized_pipeline, + ) + +from telefuser.core.config import AttentionConfig, AttnImplType +from telefuser.utils.utils import get_example_name +from telefuser.utils.video import save_video + + +def get_pipeline( + *, + model_root: str = PPL_CONFIG["model_root"], + quantization: str = "tf-kernel-fp8", + sample_solver: str = "euler", +): + """Load Wan2.1 with online FP8 quantization and dense SDPA attention.""" + return get_quantized_pipeline( + model_root=model_root, + quantization=quantization, + attention_config=AttentionConfig.dense_attention(AttnImplType.TORCH_SDPA), + sample_solver=sample_solver, + ) + + +@click.command() +@click.option("--prompt", default="A small paper boat floating down a sunlit stream.") +@click.option("--seed", default=42, type=int) +@click.option("--resolution", default="480p", type=click.Choice(["480p", "720p"])) +@click.option("--width", type=int) +@click.option("--height", type=int) +@click.option("--num-inference-steps", default=PPL_CONFIG["num_inference_steps"], type=int) +@click.option("--num-frames", default=PPL_CONFIG["num_frames"], type=int) +@click.option("--cfg-scale", default=PPL_CONFIG["cfg_scale"], type=float) +@click.option("--sigma-shift", default=PPL_CONFIG["sigma_shift"], type=float) +@click.option("--sample-solver", default="euler", type=click.Choice(["euler", "unipc"])) +@click.option("--model-root", default=PPL_CONFIG["model_root"]) +@click.option( + "--quantization", + default="tf-kernel-fp8", + type=click.Choice(["tf-kernel-fp8", "torchao-fp8", "none"]), +) +@click.option("--output", default=get_example_name(__file__, "mp4")) +def main( + prompt: str, + seed: int, + resolution: str, + width: int | None, + height: int | None, + num_inference_steps: int, + num_frames: int, + cfg_scale: float, + sigma_shift: float, + sample_solver: str, + model_root: str, + quantization: str, + output: str, +) -> None: + """Run dense Wan2.1 with tf-kernel/TorchAO FP8 or a BF16 baseline.""" + configure_attention_backends() + pipeline = get_pipeline(model_root=model_root, quantization=quantization, sample_solver=sample_solver) + torch.cuda.reset_peak_memory_stats() + start = time.perf_counter() + video = run( + pipeline, + prompt, + seed=seed, + resolution=resolution, + width=width, + height=height, + num_inference_steps=num_inference_steps, + num_frames=num_frames, + cfg_scale=cfg_scale, + sigma_shift=sigma_shift, + ) + torch.cuda.synchronize() + elapsed = time.perf_counter() - start + save_video(video, output, fps=16, quality=6) + peak_allocated = torch.cuda.max_memory_allocated() / 2**30 + peak_reserved = torch.cuda.max_memory_reserved() / 2**30 + click.echo( + f"quantization={quantization} sol_attention=false elapsed_s={elapsed:.2f} " + f"throughput_fps={num_frames / elapsed:.4f} " + f"peak_allocated_gib={peak_allocated:.3f} peak_reserved_gib={peak_reserved:.3f} output={output}" + ) + + +if __name__ == "__main__": + main() diff --git a/examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py new file mode 100644 index 0000000..1c61b5a --- /dev/null +++ b/examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py @@ -0,0 +1,257 @@ +"""Wan2.1 1.3B T2V with Sol-Attn and online weight quantization. + +The DiT is quantized with tf-kernel FP8, TorchAO FP8, or bitsandbytes NF4 while attention +continues to consume contiguous BF16 Q/K/V tensors required by Sol-Attn. +This example is intended for H100 validation and keeps the quantized DiT on +CUDA so the quantized Linear modules are not repeatedly reconstructed during +CPU offload. +""" + +from __future__ import annotations + +import os +import time + +import click +import torch + +from telefuser.core.config import ( + AttentionConfig, + QuantConfig, + QuantKernelBackend, + QuantType, + WeightOffloadType, +) +from telefuser.core.module_manager import ModuleManager +from telefuser.pipelines.wan_video.wan21_video import Wan21VideoPipeline, Wan21VideoPipelineConfig +from telefuser.utils.utils import get_example_name +from telefuser.utils.video import get_target_video_size_from_ratio, save_video + +TF_MODEL_ZOO_PATH = os.environ.get("TF_MODEL_ZOO_PATH", "model_zoo") +PPL_CONFIG = { + "model_root": TF_MODEL_ZOO_PATH + "/Wan2.1-T2V-1.3B", + "negative_prompt": ( + "Camera shake, overly saturated colors, overexposed, static, blurry details, subtitles, " + "worst quality, low quality, JPEG compression artifacts, ugly, incomplete, deformed limbs" + ), + "num_inference_steps": 40, + "num_frames": 81, + "resolution": "480p", + "cfg_scale": 5.0, + "sigma_shift": 8.0, +} + + +def configure_attention_backends() -> None: + """Avoid cuDNN plans being selected for dense SOL guard/fallback calls.""" + if hasattr(torch.backends.cuda, "enable_cudnn_sdp"): + torch.backends.cuda.enable_cudnn_sdp(False) + if hasattr(torch.backends.cuda, "enable_flash_sdp"): + torch.backends.cuda.enable_flash_sdp(True) + if hasattr(torch.backends.cuda, "enable_math_sdp"): + torch.backends.cuda.enable_math_sdp(True) + if hasattr(torch.backends.cuda, "enable_mem_efficient_sdp"): + torch.backends.cuda.enable_mem_efficient_sdp(True) + + +def make_quant_config(quantization: str) -> QuantConfig: + """Build the online quantization config used for the Wan DiT.""" + if quantization == "none": + return QuantConfig() + if quantization == "tf-kernel-fp8": + return QuantConfig( + enabled=True, + quant_type=QuantType.FP8, + kernel_backend=QuantKernelBackend.TF_KERNEL, + ) + if quantization == "torchao-fp8": + return QuantConfig( + enabled=True, + quant_type=QuantType.TORCHAO_FP8, + kernel_backend=QuantKernelBackend.TORCHAO, + ) + if quantization == "bnb-nf4": + return QuantConfig( + enabled=True, + quant_type=QuantType.BNB_NF4, + kernel_backend=QuantKernelBackend.BITSANDBYTES, + ) + raise ValueError("quantization must be 'none', 'tf-kernel-fp8', 'torchao-fp8', or 'bnb-nf4'") + + +def make_attention_config( + *, + dense_timesteps: int = 10, + dense_layers: int = 1, + tau: float = 1.0, + threshold_type: str = "diag", + kv_splits: int | str = "auto", +) -> AttentionConfig: + """Build the official Wan2.1 Sol-Attn profile.""" + return AttentionConfig.sol_attention( + dense_timesteps=dense_timesteps, + dense_layers=dense_layers, + tau=tau, + threshold_type=threshold_type, + kv_splits=kv_splits, + ) + + +def get_pipeline( + *, + model_root: str = PPL_CONFIG["model_root"], + quantization: str = "tf-kernel-fp8", + dense_timesteps: int = 10, + dense_layers: int = 1, + tau: float = 1.0, + threshold_type: str = "diag", + kv_splits: int | str = "auto", + attention_config: AttentionConfig | None = None, + sample_solver: str = "euler", +) -> Wan21VideoPipeline: + """Load a Wan2.1 pipeline with SOL and optional online quantization.""" + quant_config = make_quant_config(quantization) + module_manager = ModuleManager(torch_dtype=torch.bfloat16, device="cpu") + module_manager.load_model(f"{model_root}/Wan2.1_VAE.pth", device="cpu", torch_dtype=torch.bfloat16) + module_manager.load_model( + f"{model_root}/diffusion_pytorch_model.safetensors", + device="cuda", + torch_dtype=torch.bfloat16, + quant_config=quant_config, + ) + module_manager.load_model(f"{model_root}/models_t5_umt5-xxl-enc-bf16.pth", device="cpu", torch_dtype=torch.bfloat16) + + pipeline = Wan21VideoPipeline(device="cuda", torch_dtype=torch.bfloat16) + config = Wan21VideoPipelineConfig() + config.dit_config.attention_config = attention_config or make_attention_config( + dense_timesteps=dense_timesteps, + dense_layers=dense_layers, + tau=tau, + threshold_type=threshold_type, + kv_splits=kv_splits, + ) + config.dit_config.quant_config = quant_config + config.dit_config.offload_config.offload_type = WeightOffloadType.NO_CPU_OFFLOAD + config.sample_solver = sample_solver + config.enable_metrics = True + pipeline.init(module_manager, config) + return pipeline + + +def run( + pipeline: Wan21VideoPipeline, + prompt: str, + *, + seed: int = 42, + resolution: str = "480p", + width: int | None = None, + height: int | None = None, + num_inference_steps: int = PPL_CONFIG["num_inference_steps"], + num_frames: int = PPL_CONFIG["num_frames"], + cfg_scale: float = PPL_CONFIG["cfg_scale"], + sigma_shift: float = PPL_CONFIG["sigma_shift"], +): + """Generate one deterministic validation video.""" + if (width is None) != (height is None): + raise ValueError("width and height must be provided together") + if width is None or height is None: + width, height = get_target_video_size_from_ratio( + "16:9", resolution=resolution, height_division_factor=2, width_division_factor=2 + ) + return pipeline( + prompt=prompt, + negative_prompt=PPL_CONFIG["negative_prompt"], + num_inference_steps=num_inference_steps, + num_frames=num_frames, + cfg_scale=cfg_scale, + seed=seed, + height=height, + width=width, + sigma_shift=sigma_shift, + tiled=True, + ) + + +@click.command() +@click.option("--prompt", default="A small paper boat floating down a sunlit stream.") +@click.option("--seed", default=42, type=int) +@click.option("--resolution", default="480p", type=click.Choice(["480p", "720p"])) +@click.option("--width", type=int) +@click.option("--height", type=int) +@click.option("--num-inference-steps", default=PPL_CONFIG["num_inference_steps"], type=int) +@click.option("--num-frames", default=PPL_CONFIG["num_frames"], type=int) +@click.option("--cfg-scale", default=PPL_CONFIG["cfg_scale"], type=float) +@click.option("--sigma-shift", default=PPL_CONFIG["sigma_shift"], type=float) +@click.option("--sample-solver", default="euler", type=click.Choice(["euler", "unipc"])) +@click.option("--model-root", default=PPL_CONFIG["model_root"]) +@click.option( + "--quantization", + default="tf-kernel-fp8", + type=click.Choice(["none", "tf-kernel-fp8", "torchao-fp8", "bnb-nf4"]), +) +@click.option("--dense-timesteps", default=10, type=int) +@click.option("--dense-layers", default=1, type=int) +@click.option("--tau", default=1.0, type=float) +@click.option("--threshold-type", default="diag", type=click.Choice(["diag", "exact"])) +@click.option("--kv-splits", default="auto", type=click.Choice(["auto", "1", "2", "4"])) +@click.option("--output", default=get_example_name(__file__, "mp4")) +def main( + prompt: str, + seed: int, + resolution: str, + width: int | None, + height: int | None, + num_inference_steps: int, + num_frames: int, + cfg_scale: float, + sigma_shift: float, + sample_solver: str, + model_root: str, + quantization: str, + dense_timesteps: int, + dense_layers: int, + tau: float, + threshold_type: str, + kv_splits: str, + output: str, +) -> None: + """Run Wan2.1 SOL + quantization validation.""" + configure_attention_backends() + pipeline = get_pipeline( + model_root=model_root, + quantization=quantization, + dense_timesteps=dense_timesteps, + dense_layers=dense_layers, + tau=tau, + threshold_type=threshold_type, + kv_splits=kv_splits if kv_splits == "auto" else int(kv_splits), + sample_solver=sample_solver, + ) + torch.cuda.reset_peak_memory_stats() + start = time.perf_counter() + video = run( + pipeline, + prompt, + seed=seed, + resolution=resolution, + width=width, + height=height, + num_inference_steps=num_inference_steps, + num_frames=num_frames, + cfg_scale=cfg_scale, + sigma_shift=sigma_shift, + ) + torch.cuda.synchronize() + elapsed = time.perf_counter() - start + save_video(video, output, fps=16, quality=6) + peak_allocated = torch.cuda.max_memory_allocated() / 2**30 + peak_reserved = torch.cuda.max_memory_reserved() / 2**30 + click.echo( + f"quantization={quantization} sol_attention=true elapsed_s={elapsed:.2f} " + f"throughput_fps={num_frames / elapsed:.4f} " + f"peak_allocated_gib={peak_allocated:.3f} peak_reserved_gib={peak_reserved:.3f} output={output}" + ) + + +if __name__ == "__main__": + main() diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index c38419d..be6c3bd 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -121,6 +121,36 @@ def _resolve_attention_config(self, sparse_state: SparseAttentionState | None) - return AttentionConfig.dense_attention(AttnImplType.FLASH_ATTN_2) return self.attention_config + @staticmethod + def _is_sol_active(sparse_state: SparseAttentionState | None) -> bool: + return ( + sparse_state is not None + and sparse_state.config.sparse_impl == "sol" + and not sparse_state.should_use_dense() + ) + + def _prepare_sol_projection_input( + self, + x: torch.Tensor, + sparse_state: SparseAttentionState | None, + ) -> torch.Tensor: + # TorchAO preserves Wan's FP32 residual dtype, unlike autocast nn.Linear. + # Cast once before q/k/v instead of casting all three projected tensors. + if self._is_sol_active(sparse_state) and x.dtype != torch.bfloat16 and torch.is_autocast_enabled(x.device.type): + return x.to(torch.bfloat16) + return x + + def _prepare_sol_qkv( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + sparse_state: SparseAttentionState | None, + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + if self._is_sol_active(sparse_state) and q.dtype != torch.bfloat16: + return q.to(torch.bfloat16), k.to(torch.bfloat16), v.to(torch.bfloat16) + return q, k, v + def async_usp_forward( self, x: torch.Tensor, @@ -185,6 +215,9 @@ def default_forward( sparse_state: SparseAttentionState | None = None, device_mesh: DeviceMesh | None = None, ) -> torch.Tensor: + input_dtype = x.dtype + x = self._prepare_sol_projection_input(x, sparse_state) + projection_dtype = x.dtype q = self.norm_q(self.q(x)) k = self.norm_k(self.k(x)) v = self.v(x) @@ -193,6 +226,7 @@ def default_forward( q = rearrange(q, "b s (n d) -> b s n d", n=self.num_heads) k = rearrange(k, "b s (n d) -> b s n d", n=self.num_heads) v = rearrange(v, "b s (n d) -> b s n d", n=self.num_heads) + q, k, v = self._prepare_sol_qkv(q, k, v, sparse_state) if sparse_state is not None and sparse_state.config.sparse_impl == "radial": seqlen = q.shape[2] q = rearrange(q, "b s n d -> (b s) n d", s=seqlen, n=self.num_heads) @@ -208,6 +242,11 @@ def default_forward( output_layout="BSND", ) x = rearrange(x, "b s n d -> b s (n d)", n=self.num_heads) + if projection_dtype != input_dtype: + x = self.o(x) + return x.to(input_dtype) + if x.dtype != projection_dtype: + x = x.to(input_dtype) return self.o(x) @@ -439,7 +478,7 @@ def reset_y_camera_status(self): def enable_quant(self, quant_type: str | torch.dtype): """Enable quantization for transformer blocks.""" - from telefuser.core.config import QuantConfig, QuantType + from telefuser.core.config import QuantConfig, QuantKernelBackend, QuantType if isinstance(quant_type, QuantConfig): if quant_type.quant_type == QuantType.BNB_NF4: @@ -470,6 +509,38 @@ def enable_quant(self, quant_type: str | torch.dtype): logger.info(f"TorchAO FP8 converted {replaced} Linear layers") self.quant_type = quant_type.quant_type return + if quant_type.quant_type == QuantType.FP8: + if quant_type.kernel_backend not in (QuantKernelBackend.AUTO, QuantKernelBackend.TF_KERNEL): + raise ValueError( + "Wan FP8 online quantization requires the tf-kernel backend; " + f"got {quant_type.kernel_backend.name}" + ) + logger.info("loading weights with tf-kernel FP8, start quantize linear layers") + from telefuser.ops.fp8_gemm import FP8GemmOptions, count_linear_layers, enable_fp8_gemm + + include_names = quant_type.quantize_modules or ("blocks.",) + + def module_filter(name: str, _module: nn.Module) -> bool: + return any(token in name for token in include_names) and not any( + token and token in name for token in quant_type.skip_modules + ) + + replaced = count_linear_layers(self, module_filter=module_filter) + enable_fp8_gemm( + self, + options=FP8GemmOptions( + cast_output_back=False, + fp16_weight_storage="keep" if quant_type.keep_fp16_weight else "discard", + materialize_fp8_on_wrap=True, + ), + module_filter=module_filter, + ) + if replaced == 0: + raise RuntimeError("Wan FP8 online quantization did not select any Linear layers") + self.tf_kernel_fp8_replaced_linear = replaced + self.quant_type = quant_type.quant_type + logger.info(f"Wan tf-kernel FP8 converted {replaced} transformer Linear layers") + return quant_type = torch.float8_e4m3fn if quant_type.quant_type == QuantType.FP8 else quant_type.quant_type if quant_type in [torch.float8_e4m3fn]: diff --git a/tests/unit/models/test_wan_video_sol_attention.py b/tests/unit/models/test_wan_video_sol_attention.py index d0c8afd..0510163 100644 --- a/tests/unit/models/test_wan_video_sol_attention.py +++ b/tests/unit/models/test_wan_video_sol_attention.py @@ -99,6 +99,35 @@ def fake_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, **kwargs) assert captured["sparse_state"] is state +def test_wan_self_attention_casts_fp32_qkv_only_for_active_sol() -> None: + module = SelfAttention(dim=128, num_heads=1) + config = SparseAttentionConfig(sparse_impl="sol", dense_timesteps=1, dense_layers=0) + state = SparseAttentionState(config, mask_map=None) + q = torch.randn(1, 4, 1, 128) + + dense_qkv = module._prepare_sol_qkv(q, q, q, state) + assert all(tensor.dtype is torch.float32 for tensor in dense_qkv) + + state.update(numeral_timestep=1) + sol_qkv = module._prepare_sol_qkv(q, q, q, state) + assert all(tensor.dtype is torch.bfloat16 for tensor in sol_qkv) + + +def test_wan_self_attention_casts_projection_input_once_under_autocast() -> None: + module = SelfAttention(dim=128, num_heads=1) + config = SparseAttentionConfig(sparse_impl="sol", dense_timesteps=1, dense_layers=0) + state = SparseAttentionState(config, mask_map=None) + x = torch.randn(1, 4, 128) + + with patch("telefuser.models.wan_video_dit.torch.is_autocast_enabled", return_value=True): + dense_x = module._prepare_sol_projection_input(x, state) + state.update(numeral_timestep=1) + sol_x = module._prepare_sol_projection_input(x, state) + + assert dense_x is x + assert sol_x.dtype is torch.bfloat16 + + @pytest.mark.gpu def test_wan_self_attention_executes_sol_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): diff --git a/tests/unit/pipelines/wan_video/test_sol_quant_example.py b/tests/unit/pipelines/wan_video/test_sol_quant_example.py new file mode 100644 index 0000000..2e1a8db --- /dev/null +++ b/tests/unit/pipelines/wan_video/test_sol_quant_example.py @@ -0,0 +1,139 @@ +from __future__ import annotations + +import pytest +import torch + +from examples.wan_video.wan21_1_3b_text_to_video_fp8_h100 import get_pipeline as get_fp8_pipeline +from examples.wan_video.wan21_1_3b_text_to_video_sol_fp8_h100 import ( + make_attention_config, + make_quant_config, + run, +) +from telefuser.core.config import AttnImplType, QuantConfig, QuantKernelBackend, QuantType +from telefuser.models.wan_video_dit import WanModel +from telefuser.ops.fp8_gemm import FP8Linear + + +def test_wan_sol_quant_example_builds_compatible_configs() -> None: + attention = make_attention_config() + quant = make_quant_config("torchao-fp8") + + assert attention.attn_impl is AttnImplType.SOL_ATTN + assert attention.sparse_config is not None + assert attention.sparse_config.sol_tau == 1.0 + assert quant.enabled + assert quant.quant_type is QuantType.TORCHAO_FP8 + assert quant.kernel_backend is QuantKernelBackend.TORCHAO + + +@pytest.mark.parametrize( + ("name", "quant_type", "backend"), + [ + ("none", QuantType.FP8, QuantKernelBackend.AUTO), + ("tf-kernel-fp8", QuantType.FP8, QuantKernelBackend.TF_KERNEL), + ("torchao-fp8", QuantType.TORCHAO_FP8, QuantKernelBackend.TORCHAO), + ("bnb-nf4", QuantType.BNB_NF4, QuantKernelBackend.BITSANDBYTES), + ], +) +def test_wan_sol_quant_example_quantization_choices(name, quant_type, backend) -> None: + config = make_quant_config(name) + if name == "none": + assert not config.enabled + else: + assert config.enabled + assert config.quant_type is quant_type + assert config.kernel_backend is backend + + +def test_wan_sol_quant_example_rejects_unknown_quantization() -> None: + with pytest.raises(ValueError, match="quantization must be"): + make_quant_config("int8") + + +def test_wan_model_enables_tf_kernel_fp8_on_transformer_blocks() -> None: + model = WanModel.__new__(WanModel) + torch.nn.Module.__init__(model) + model.blocks = torch.nn.ModuleList([torch.nn.Sequential(torch.nn.Linear(8, 16), torch.nn.Linear(16, 8))]) + + model.enable_quant( + QuantConfig( + enabled=True, + quant_type=QuantType.FP8, + kernel_backend=QuantKernelBackend.TF_KERNEL, + ) + ) + + assert model.tf_kernel_fp8_replaced_linear == 2 + assert all(isinstance(module, FP8Linear) for module in model.blocks[0]) + assert not model.blocks[0][0].options.cast_output_back + output = model.blocks[0](torch.randn(2, 8)) + assert output.shape == (2, 8) + + +def test_dense_fp8_example_selects_dense_attention(monkeypatch: pytest.MonkeyPatch) -> None: + captured = {} + + def fake_pipeline(**kwargs): + captured.update(kwargs) + return object() + + monkeypatch.setattr( + "examples.wan_video.wan21_1_3b_text_to_video_fp8_h100.get_quantized_pipeline", + fake_pipeline, + ) + + get_fp8_pipeline(model_root="model", quantization="tf-kernel-fp8") + + assert captured["quantization"] == "tf-kernel-fp8" + assert captured["attention_config"].attn_impl is AttnImplType.TORCH_SDPA + + +def test_dense_fp8_example_forwards_torchao_choice(monkeypatch: pytest.MonkeyPatch) -> None: + captured = {} + + def fake_pipeline(**kwargs): + captured.update(kwargs) + return object() + + monkeypatch.setattr( + "examples.wan_video.wan21_1_3b_text_to_video_fp8_h100.get_quantized_pipeline", + fake_pipeline, + ) + + get_fp8_pipeline(model_root="model", quantization="torchao-fp8") + + assert captured["quantization"] == "torchao-fp8" + + +def test_wan_sol_quant_run_forwards_explicit_benchmark_parameters() -> None: + captured = {} + + def fake_pipeline(**kwargs): + captured.update(kwargs) + return object() + + output = run( + fake_pipeline, + "prompt", + seed=7, + width=832, + height=480, + num_inference_steps=50, + num_frames=81, + cfg_scale=5.0, + sigma_shift=5.0, + ) + + assert output is not None + assert captured["seed"] == 7 + assert captured["width"] == 832 + assert captured["height"] == 480 + assert captured["num_inference_steps"] == 50 + assert captured["num_frames"] == 81 + assert captured["cfg_scale"] == 5.0 + assert captured["sigma_shift"] == 5.0 + + +def test_wan_sol_quant_run_requires_width_and_height_together() -> None: + with pytest.raises(ValueError, match="width and height must be provided together"): + run(lambda **_kwargs: object(), "prompt", width=832) From d8158a5f5daf29539d61ed8f8d907c54c3f4362c Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Wed, 12 Aug 2026 08:41:57 +0000 Subject: [PATCH 02/15] Fix Wan FP8 test without tf-kernel Mock the FP8 wrapping entry points so the Wan quantization unit test validates filter and option wiring without requiring the optional tf-kernel package. Remove the benchmark PNG asset and its README reference while retaining the benchmark table and reproduction details. Verified the previously failing test, the 18-test focused suite, Ruff, pre-commit, and staged diff checks. --- examples/wan_video/README.md | 2 -- .../assets/wan21_fp8_sol_h100_benchmark.png | Bin 85923 -> 0 bytes .../wan_video/test_sol_quant_example.py | 32 ++++++++++++++---- 3 files changed, 26 insertions(+), 8 deletions(-) delete mode 100644 examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index a4f719c..3be130c 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -231,8 +231,6 @@ seed, resolution, frame count, inference steps, and kernel warm-up policy. ##### H100 benchmark -![Wan2.1 FP8 and Sol-Attn benchmark](assets/wan21_fp8_sol_h100_benchmark.png) - This generation cold-start benchmark runs each configuration in a separate process on one H100 80GB. It uses the official Wan2.1 T2V-1.3B example prompt, `832x480`, 81 frames, 50 UniPC steps, CFG 5.0, sigma shift 5.0, and seed 42. Generation timing diff --git a/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png b/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png deleted file mode 100644 index a8bf389ebc2fa0f566b782513260fe6a7a2ef9ff..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 85923 zcmeFZ`8(9>A3yG#ib~Q-2qzUqNVe=n2-){NWH)4A#!?BDkYwMNv5hhIZ7@a1zKw0h z*w?Wy!x)V3?VRuX^B;VF`Cgyb^}4#IF6On|_x*f6w!6<7YKn9hnJ-dNQPC;Ae4#}} zb^a|C)mi8Be}jLK9%#v=`iqK6>BVy$AM7gL+W&?k`_HY~aOkJ&^xvs|;qiDQqX!i_ z8JGv^_yQCUjJgGnH~stlvuod}uFk9n)-Ja6&;G&B^R>(WgWllc7g>=IL{+0sQkg!R z@G*i5{CD)JDPQ~FXDXeDcc(Y=@A2;PS)KoVrtf=LIUN%B$($-cHZJ`bsuO zM{5#BYl@1BJ{*1cKA^?JqhicrT)j}t`u7=>$NE$MQ4LyJ_wV6n(RtR^w&`L}>8GRt zB8zCYm8E-``}Xz@OsfDE5)uLnfx1^S1O`H|BziWXjF5Ep_K;AwWw$@?gTwf#&A&u` z?(XS%$iq`!Qu5Z*b!Rx6}8vX*$EdalRNwgNgiSHHx_kPU>En8Q?@gB z@#2k_mzr6inZ12tZEbBxNNskumAk0^{zA8Lwe#|1ohMF&(w!vim@g#eL0Y=Llcvdj zGS;@y>+@FJnwn16Z9VFH`@uP?eQ{6`lD@UD2bMV`KY2xK4Q<=?SQTOw5>{7Ty zMft%gedo@DrJ2djkCl}~O#5{+6OxG_^iUixn)u2~SZI~>S({5Ot?`zBK~d4GfV3cr z$rHQRL2?k+j@>YR)5`*+_a5sQ*PYNI&-f{q#®5|J<)c8QHNGGGp`g!4c0vp$S8 zCCqYtWj}1LHlGOhHBz#%S%^%9RypBZm)$;@@QwH=yjh&PcmKY5jgQj#t?9P(!)j;Z zs^Csr6f18mA)oH{Xerb}Q*&tN<-Kd}?t-+mDvmrV^z`2B>>E`O>?P*W$u?X#Ei}Sj z<={U0=&{@HlEV1o<@m`)%3fQcI*$5buVPdxX$J1n+=SF+PXG4^nzh+oZ#a}8k;>a+ zTG2=75a;6J2-|sLVhj0j58lhEo}`g1Y;0sO3XU<{Ia2j{pL9tOS{i~oyM#U?CRU3+ zrnI4XMsD1=k*}_sdl>%{%l-Klnl1eI-E0B?_ohf0L>+Metw@wbMzThfPei%og5L+ z<37B*_>MEjcHf!#4EJKMc|J$R_t;oZL2-rI9QWsph3E6<=H?XSP4(F)iy_HS@g$Cq z2Bi+&VeCO>2>I=uop_`{gVo3czs#ZC(zoZ-lQ6K@u7hd*)RllqEU&Hg-8bw>^5q*0F(ll~5ary= zFB52MYir!e`mJ+bUcLpKiHV7>)idhpVtKbFb0bdNX&0hc1;51SMVOn$FK%yddrzLc zoK-UY{E8D-W4{8vuaPn{INcm7MC#aZ36YlvCwLRl(VAa=y0=BHrTZO~Zq2qXnTU`E z=6z|?)YW_QTVuJN5E2&|>DWc)4qea{6(#>1vs_4gRYeL`*DE%I3*0jgN$&3#&CZ7I zuTEh8=1#75Le3mFUa_2AK%79ge2?`>!x9@vqwLJa{#tECVu6l+XG^k?gXqcPV{y2n$JM;mN|G&M`fK8&`Dlw-2c zmZX#?d9_F^%CYFf>nkJ0^3v6tu!G+E3<26cAB?AOT>G0H21%s)Pq~Mpq8rP$jLu_K z*%tZ}XG1=k1W;Py(^F2IY;0_9mdh(Bbf+BqRAxO?rn$_Pw!iB>6yO~t&Jo{f?t8p; z6F&UyE*V$h3y)-~a=I&}1fim0z6kJ$>ZkJZm{#T$%@G%{677t+mzv}uG0x*(dpbG> zM2IPIZ&jk$@JVibf8LbI^EWW0e|z@cX%oy9dH3T&P{>x$fe=CUG@)OA{(Rb{r=vYt zj=%3&_xYISGVjqO&pl7=y=ks=c4@aFYQ`{qGpLGfc!+1+Hs{Jz=G`bs@*6d^;wmOH zKVtXHOcIUqX_>NFqaLUgFlg{PUf{*Oq&Wu98BrIQP4K;T?OJadEeGP|>8+nVy>%6J z)mWW5H(!l!!};yCJK_YSU3$*_jDe4i`44U|G6rog$Ny-hR^Kd$;j(w)=Ds)QP!VUd z&aI@XA`!4q)T%U?tHv>ZbGQhv;c3NOS$*;|GvJW^J4SV|WlH6w*ub>hy0f#>JZMEK z@zwA@JT5a$NO{TX&HE8Cva-cybae1}vOdyOokO%HD{IE3@i5vWKQ1D@r>?O)=>^7lCMCym>P$G(^9&DT9_#*vs6+#KeEzA)oAHZqCld z)pqzu7c@MfjG#&B2Bu`x@>n=_m> zDCo{agWm&*IgLu+6|Vqn=R_SOuT|rD*xq{z!WeJc&KLbs+{;eJr|;e6wKJgm40TQP z3=EXz=Rc5^-s-A#@6<0EW(ZMMRSEy}wNOtTBV1xJ$dGDUFiX+a)ZA*tg$t*JUcP)8 z*;w)R&$s7QA|kG?)p|unha1XtnbDs<{gF8_6WM5QTk)O0BAfEm*Wu*7+}%)mgb-zS ztQ^b0B>txFx;r}rUkF1WRNa&bTi3aQD8&7_xf?_=qzN7Gg9jyLWgMw2eW3O{c_LJ? zX5-{^^mF&&-7xwc7ktOB%*4;1l`JeQ%*-eV5@|)$IW$Z$MS|VqxV4*ev`5R(aBPni zMND)bNekHE4@_?S@Nx~>-*+cCD_d5HhN{NWY^6}y<>h%a#t4}Qh-IYQ+SZ`D>l z++k*{PcO^^WnNOqiY!O3<~Y5WCq99}R5-@>n4&z=BwdeI+tNCEd-ct$>g$hww`*Iq zeSJbQQ-Tfr{8@E;xVJn~-J990ctW=IHO*^YpwkU#WKI9}`1R(?h?vORqli+vI`XSc zdu>j~OZiUtD!KG`9Spo{0YhaHWa{2QWfM~q244U5Or}B8lvEzovZ^ZQ!B(^?%Sj@= z6zM3O-x1FF{YQ!k{$0m+^FR%{Z(xc3ZkaHuw83MEEW$lOGjk7%H}}sr$1TR>^}J9} zD2S@As2H#D3Mos!7j!br{wNTKZ#Y;q56qU{ne5b)JRn(V6;w#;PrTz!HSw%o028;S zrbg=U@=@b1vMGxrBK&A;WoNDXD~ z2PTg9=;<=^>3|8FosBytLcw(9KK_jZtpz2zF;pIA`3m%UN^qF>#+a?8<$#CQ>eO_$ z;n*0t%h14}#(RVEFosL3>vS;_m`)dScvMt%c)7cS(Y*dmeYxf@U%v1@>f(ve-+0#Y z`}d2=aC-=?17t%5aVB;(C1q(T;q*)xP3{+HZ-2-H%J7)jeb{{%%vwP~VHg~GacOC6 zpkaHgj#`lVet{)KtrU^JB3k8)bDf!6&dHf8T6L0;Fjl9J`s?^m`f&hgP_{Q|jSLN? zP_HMn)w?Ro%Mq4UJ*nb>HKa9pesr-b0YUPSvxdLm*xK47j8vC-Ns*VwN#xx!c;FVnu!iXQ+B*Cy7I;U}P8POu|M zEQD%grq-e5CJce|GO6)-cwbmpRJ6Cfy}u3TIQPk7wk=Y?M~XsS|7nQTxr-{}*XfF$ z^Vpd199eIORfvm^r!lW9qWBvZ{QV_5x_|ETtS$>(O#2)1qx>8*Ihe|RqwXr=6=!GH z>G2hHGl;Njc;_Q5q=n>aW#+rwQphfLnN^znyWSSl7PPl@D9DzQ78gh9nW5ibm&g!* zC?$1QN@{js{@Z<(6F;wC^^9r88IXCh%!K+a8LzFdbh}k=eXF~-gsmJL{N|?84r}^` zO+Nqx`ISw|ZEyiWh^cbIn|VHyH|9q*hhBCbE;78{rEP1=#lo@z%D8Gw3SDLFh1piS zOZn&T0E9$$7`WLH+cz0Usr8=xi|G7#U|?WyU?97Aaum#%D_K@;dpqg={#{Ga%+UFQ zuyC!r`ik?l`S}LMu>9quiFy^-z{40$yVZV+&MMYeOE%9sw>vNN^z=s3j)!d71RZ|! z%LI6*n>HUG8_P&-eC9_})Mk-KLwzO6-dCk3%!Y99a=kLYe*Kyx>J6Vyn*<}STw3Z1 zFup;lRpDJLtJz34iBqtS-bRN+^(5-%8Kx%|OvjImch&P%S&#B~1^?;ll62dxA(H^E z!x|VPjmnD$NQ1UL_c=LhQKBO?Rr4psejCA2@CnP_#$cgHv=?y(>&fC+UXz$;c=u#Y z(8*zCxwU&~WWwjLsYRvm?)B?2*RIE?rHkzkiOST2>z1?oj=qa!rh{IETW1cG>%6t1 zrix%s9XMm>R#8QU>E*q+`bd*?uuApkw_j(7MvqC~$($)Ew};RKL~&DAmM4oy#BIT? zrPAxd#&R$J3<+G%)v1%w(={?8ukFk5QZ{3-c~t;|b~5k^sX1t=8eo!{@6}W-)OxLB zwdJASZngd^d#OW4(*gX#yuAGCa!|*Gg@v^0$} z%L%i6qr)udGes4F{3#TumX?sGjoRS(S25|Y<_{`}Vr zy^QZx5R%~j8QvXLM*b82E8fy-_EJr(U!&znDgx*2P`{NRWHFxec^1>AucPBMunlRG z%CrbNB*;_(04C_#5e!x>!FFB+S)W+JWT7>!oUqoNnD#V3axHrh1yScBbeT>3N#MZl+>aj$ zftqzLA45a!Zv$Gy0tvTEzQMwxuc4vg!sGJc_U+1HTxWZGflzC2ulG@n0KwfESGv-E z(3>V6AR;A|T~Q&lu6OiB*D<#{0T$SKCsvtJdh#{B3t{3xCt-b>VIH-P7YcRTz`s)tKG_>` zcc3wE+@3qK<&iS5udv2Nica_;E!^{38B4{RBABRzM1YDYBnJ8WNVw{Yy8uZGwh_=% z5Mtoxu|b%dTUuO%j$Hr1Mmu3I`0i-d!ZUIgxj9655x{yUmsjIfVMg^$-U<5$fk)argIE>Fp zB{k0OY`zVepJ#MLBT`bvdGzHb{cX24f{tLWp)|Sb0vOkZa#Az5rM+&r`b!@NiaQ{G z5qnu4G8gD*N1je}t2)Z5dwEq6QUfpC0WclL7ftY(SJk-blDoPpgyB7WyCuPVMu&!; zHBoqRi!Fef`m7g|kOxL$BOoEAXj*k_1$l&2+;^t)si*S^=ZI;i?02w`UufSDamE&w zLRo^BFr~q*TLt#^8x1F#`x9r{UL|ZVNQ93RPx`DGz3C=&V0Lbz)&VTCuV4MjCOiOe z1V{9%z8ltGBO`0`3$%P@M@rBE#B{9V2;>3ZaCHs>S5ok$?rOxr|2yMbLr-a-7=TR1 zBRvEJH9+p>badJQyL4LN;SaEa%VQ164rml0tQHm{;vO}F#Plk*Wt1Z_)6&IO)MxYO zI=+rn?L0rfY!880TUco7>BU7y7ua3$ZqDT;AdoKrs;@qo%8zbdd4MRMtYPwBc8l*x zb*K;RHm_`#hUy^Yj;Y(#tg(Rd=?X60b_;r)9Mbi#;qxzHn zCq2eU;hymC7TS##u$)|n+Z(H^IxSi~YHG|99coHSi0W9;M^fdaWncCBy^fPhmHpg} z)1GdaJ}q==>KazJ^YpTG$yGWl5oy`k%`_VgQeIsT&g70Zc!qE9m)*oRy(hY0{!CAv zkix49Zr;2J2(B55p`eCfWRT#!HURTRuFFx=Uv$dTmy3=#`eMqH5Ii#{LeX{PEPo&!K}je6+Tk4PoN$5 ziozZm3g^FRd1|#tBp7 zbf`^ZRYb8#Lsi|H3Z_?nSvn>id2mL{kq6ElS5{Gmv4sF)oy+;;31MG`GW@jt&!3k# z>p?7*9NsI9EqCus852}hRrPd+>2@+~p!fH-nh)u2sn#w!YjSgQ$MKpMKq&UwT#$(p zaiMz;cDE0BSwWa9&|WMQW{kJnpOfTI;hUAzAYk8 zf%;;NyG-2T>*;Cm^U}Wro>adevZg806a`+xdVjcF@X?;|YX_*GkN?3zo2?pKA>uE2)e##1>s{)sGXF2-&1!9?(H28+fZRcBbSISJ# zuG&`Fqky6fkBTmbR`MubVlQ;nf-U|P(K~mlkgZSSuvjM(l;uQB6x7!*DJh9eCi|C= zAQ10%Y*PQoLoP0TW5rKE`MoV>d27C}#36AC>kwz>gjF%tem$7%;Jq6wRhv5CSGXa1g1)Mw(QLyL3Tfloz#

<(X@6M&o@YJIFI zYw&Q)a2PshWzBd*Jzacks09$T-$O0dWK27*s%TIz#IGAm7qP~nM-ThJB1l!k1zfDY z8=kA9X0xggjXrtnW@f^1#sgHw=Ju|KdjlM+ZnSc6gFA5-cb3%pm>4kD(q*Vs6`C#_xQtb;q>rFHJze%+Yt$rSsIR<1g@*4?KJ$R7~iO{v7wz{1-mp+{kjr zhW<{v?=Pfh#x@cX<_x5F%6eEjd@0OuHWBvr=y$=LyRCsc_^1mPKKq$nQA-n-vR&6c z)$#;t%<`Dqc{`Q`ZH0Tr0CQ_&AN+;^IC==BA0@8Jq^KB1gr7igoEg-eoSd9M)gyWF zPA-dLZld^{#_M%weXMD9m`{dkeeRBEpabY)vF85dOuxJN-K%Rd2TK7FJLZ>bwbEIS zZV@Q|fo5iI`rQn9t z)YcxjG3g*FAt7IsFFHQn96uMK{xU((JIz5x94e8aYB-bZvS%aOMx?kl!4X zsbSFCAI^0UZU%{#S0hmfr?I-@)!H4Nxa0n&USyo3gs+i%Hl1$8&gP#YCYh7E7Y$Ki z0Ar`5m`F(ATHk8cnHwe~JKi=o(;K2!v@cjt`1pvImjIcU{RFV4ykdyR*6mfG=zN-% z6IR2PZ~S!UKSJ6x{Me9rYd4(IrjI1yGyvnwH2wilwQ#ZTLNmPsZmBDbP11eOKh>rJ z^o_U(M^H--I$c!Yhi-bDF%8>f9N^bqP=nNHOE~=4rJ+ih9&jvHi7I{8*~LYibV(|C z@ormW`YTlxxX1N@Kf)Lnb2*sT;ZLtZR^5r?PnZ_8ClAr@)|13Xa~-i}QqXZZMuFhj z)=N~(Tn_nEc3BlejG79D+#$w*-UR4Mn`oJv7-r9>%O zt0~K2NMSYQmoNR7V?~7&v|A%s8$ivy!`S;G#9TVys82sVXeN94>?Jzdp#9@0M|8|- zO+@%~GnN9dWrwq?E;c3HL4i(}?&r3i7!!s*TnM+2qcPSUW7M1o(_gL2 zKp00}m=(r+>l?is3*`Aa&$b57b9VWYjiJhGg))y;u$ASlTh%ky*Ad|fzy51Urx-&M zONPk=QJRXvY;$0+wpM(dBL&5N1_QMaWB0IVJu?V+>b2{_mrL z%=B6v2aMr+t2`(=4JSYEKqfOC%je?Rb)b#W?Qeis7SC^fV0-XIJ5RX6{yGaw_1wM; zBd>q4i2k*RJ528G?w|s3Gwkkeeu{|DW=~H{9GYT;G3Wo-WT zcY7O7*3&T-^!Tr(8Ia$&nzBgR{RD%QwFX~g4>HUujV+01kUAL`ROdPIbazh`@f`Sb zRW48Z@eVm6Io4*P%=7SruR6(52JKPD=%OjqNdEEi-gRzB@q^YS%f8cCm zBVylFqjbHs{Y@DlBEfuT&FgRZ3y|ED9UO*e*CpQOXJkARzBH%-Mz+K?CXOMW)4``j z#^PvH9ZfPiUOA*ou9hawjYF5bvN8h?nYkt78@VnMa}J2h#uW_Sg#_7H*e-SabY|8P?mR@Kh(UBo~S&S!i+)xEDzu`@H6 z+52m0xyj0A`I+Pl%zk{qu{_ZLcVPKl>RPQeI8fo4F(9WcwGYVuZ5h|XH2U!_%nRJT zf}91-y;D8uBwJlLaB2*08wEoj+x?6i*Yv(14e7{%1yQo4195w%NHqDo4mUmv79D1< zZNH)ND(NvF2gmlv{h)mHi*;}N-?8}3R>q4!#&XjRmzSAFI5^JWDiLE2l|y$P3=DlY zu*h6sa@t+tEr{e(R(<~ZH4adp{3vy0<>9KAfZDaTn%%pmgsM6%W~yO88uT<~GBIXZKT)e_X&iV&wm;##(=|0e7k^TvUcU$~`(MW$J8UPVPkR#w)R z1m)6PH{_ilrZz~O-ELP23-giD_w$b zaC5c$HrYu13IaK|fp&w}?i@G>M~>X$``_$76%|Z@t*K29l*S$!(|pX`2Ed5OEZ@@b zC|v?Z00aVjRfTo#UHjbHiV}rPNc8bEV534! zKZ-2dS|gc=fJP-IC238pe?CipeiJZPz2o|qN~$1AQ4~?x-MwT-g!u;q)OjtxYCL~O z0l;;EjPJ?G`URD7*tc)rZoc&%8P$D;0j|d7`~;u@8W{jEp$X=3bhM%T?6s{67mW7G zndGz7DqZpptn~~?!a&eD8oDm)1BDt+9;d<7^aIgPURqfe*e9(}iHX}u(E$GdoI{f@ zb8}%sKBDXW#y5cAPki^qgirmI^1yJe2AFHdpnFA)m9tHhd!t*uSb01T{ew(O0Wits z#a*hb0(w~^v6YAUFz{J$o|mkwrLOjMVfD&tq_4~7{x5e1Q5&j0JJ;&JGL{Kr**KX_ z{Zo2AH(L4W<&c0W%Fd4cRl`jra1{$hE%`p$jWr<+1?l#Ywmqr(?Dh^0b%EU%Plgc~ zuK9&|;VPjY?QdKf8+`Dgs?2%P;)7=1p|WK&{}EHvfU`AnEYaJR3`mOQ|DlIf$4vw{crwSFdsofovehqzyiL4IlHSYv>neSIgRtVyFkO!NM0;7rz^>vi<6<&S=8 z7U0`I<2`XN$*uQ338Z6$0$plsH{P#Fc-xQff^17WYlw2?@Q<$%5o>E}Ky2ULy#(pz zGaTWw8APhPM9k_J8D&GFZ}N;d)UThba&)w{Rf*2mj$>fJn%USNT{7g1f#8a3hop&= z7@%S0f2iXBq@E&Y=-#VCRkYC2iyYD)88zEQ?hZ>{c+((;($n-#p z=g{PWe%9Lddd8l8Pr%!j;o-kyf9dBVYU|6&o&@L_ceb#6_EUjG6o-ZazkBHEGSIx4 zt*C*Q#VX-pSHW0Het7d=ApaZ6N2MAc6O)ZZMv;ApW~suVW?-P}#*fE~5fXM0toc9e z>`PGfCOgU>K#RO=|KhJxzf6%WV5V<6n-BwD0Gcur1ImBVRtK80M}r=IPY42*sOZ>? zQQn2(j$gklHZ%Sr%+g-G_~gkGVdTnBE* zz|;1@!R#*`JVPL9AVXPIfo$lerlvZSZ@)i7w!aOGmN8%s7f;vS@Yn*nMB#AZYXkaM z$yu&PF6a{`R#t8Mxn4Ad&09rI#TCSGjt_|ii?O!W*7lNg>AveNoN0M%Z0pSYjKF_j zMn3lS^-T{th*;o>QlyhvZ}fK2B=4@nT#v@w7vq6~Ax)W#g$wJ^2#mUTd-VXJQrK>E zDK9P18Q37=JXe9T>(x7P%*=J}ENRkKe-((MQqVmnuT&sGws&-ZhG=hJ;7Umtefr;! z1d~A>H4E5D1>&djU7+v)`Lne3K*Do115r+HeR~z2uJ!8_G^m}HRePyQ9W703C#P`` zD2a&iTPm&T(~@iH=~0`FS7i+5F9>SC<;_D0k>?jp^H6#9Dv2^x1i+e*wCT~oA-{Hh z*;{@t67bvL0y{qrYbE?@O+JE%hxe8u9!i%Wxt57+JP2&-!C$=^&BT-lYBt~lAj9?O z2-`8_Xxrt#d%$SU$Im}flcsq$bi_^l%@achBW&X_4SV35^*97hxW@`KWRP$$lCE|< zblb^Rmkt6x0rff4M-Lu6;N=A&i`f;IpcJY7heXcIk5>Q+i0bnDBv{RMX=$<=sE}Yz z(9-6~2tn8C==EDzk#HM912x~p%55EvqzVUP<&tgo_E;tCq@&R!0uH2v zfa>r-M#h{o24v5GlIPhpM+4Yr-_w(f_s5P^z+vN6J~{1&(Wi7A7(QGer~xux-~^7z z$fW;%!@f*$Si|@BHz3$zwmaa6_oMTf>*qorkx_W)zM?5QIxf!4$jAt^;G{UORZ=_n zCGVc`JRwRXi3IwC?*tfZJZk?YfnbcP>*(yPhKy(=>ly3cP7gRB&VS><`n?dv%tSm7 z$gL6q@Ao>VOo{8@Fs7mc%z5yqYI3z&%J#cf!b90lVK%RjTa81_@BEYPdyBDX7Ot?FCVN{fOQ@P;lMzE)!I#85tOQ{B}RboB20_pN_nI z`Lj|xbZ`C8vz?ug?jf8p+{!722Wc-n?3>ZH(Ak0w6?s|QzbV7Q&HMv@mD38V;Q=26Je+9QdsmR%&E0dJE0?XKT!;G@ zD3tU)DcKW^ia^&|aara&p->R%elr4)`OI~_NbXljPt|ZkHn>ProYxYOda$Fz0mMUu z8*hwfS3l4y*zFf9m!|B}5|w~k2D+N~2AQA(!W!+`dBA8g#h=SlPxUx4Efsa10QI1a zy-{>0lg5*;(Q$tQ7-i|XL(TynJw+WO9K6UJq{)R;NyN7Cf(x4Lw4HN-+N53}zZX;7 z1Og%P=1}c)vDB2?Ah2MZtn%E{{5K)*KSS{PDPyxI>j?CZzW*W{-@_`rC*nFXBURyS zIditVyQ8hOwV`AwR{A)MT)iUcGrH8`>WnyW7>cO=j?Uim?3arka z6e;9Zoqz(GUO4m;Th7+yxsIW}Klv*J@72lzZV**tF_FrvdhSo6pg{Pm(opX7krDqr z@|9EFG%2b&9v-eTw4-y?LjVNFK!B0Zg7O|@j`Hn&l5}($fv9b z8+$U-I40$)ryi|-@FkXHf*!FtTv`5HkJx{>;jpTx$ezZ=rV@XoGpo?6DY(&i>_>-H zh*yTC2W?vb(}MZ!eBYnnzvtRE8VglJ%C9zuNlSaHLH4FWR}^^igbXsO2n2$d614q$ zOGHE@qUy1Mk--sp$6MD#XQQ8IFPGO?crO`Wt&Rjz%yKz3E$vRG9F1d2g{$^_&p@)U zqsXhyE^_|-3|(=WHfoP8McQ-s54I}SXasb}NB?$5E4gRqo@tMmLeKrc33{w(wdkZG z0(JXU!#{s+Pe-4uPEBR>no`Qpf&0WQPPB#$$XXyvH_qyU4g(ZxE<{CXMTP%PWjy-m zXts{>uK&*;yZ)v{kPnbQQ@j~@og*^Sd`m=IdB4m4ffvKYX=@|;f&0D*O!fNfNahQrCcXHp(8=_ zZsSq?Wt66?w~JY_veBtrMr=a@WxMgDc$7udW$y~kqb9_7Zz7(Yu z&TqdWZ^B+_Qg1O7>;B`$W};$zJbP@yLNt5Bn-`YT2k&mRDf<1M6k z0De@+P+eNOweOZy{N9MR*GpIXYNL9F4P#Yf;o+Uy>XKfoeG_?UDaLsyTPG(;|Iu%zIFZK9@xOy*>d+NSz&{xN z$|R+gp0|IQ_3t3cbh1p`kx4RVs0IrVH34@0Wv&tgM9ycn3u=jyu9L-x>|F|_T222l z28RJpk`R_5A0u4t>@|1CZ? zb_lvrzeS8sXC|opsB8Q*WLq$Gs>t1B7N+jQ5c8jKWR+8tffkjcWe*aqaRsTK6Wbt1 zQ{_zD22m}i$x@!Brj{R=itpdADbmo`GUiQ4LBFyJ7P{v}zPV1AoKuT#l`pZ{QUz11D`+71kP0~Kn1a|@)&I}&PLVF zW95qbYOE3%`TTYa2GcPxfW=~QxN=>194n6IGK+XlTtnZA-o94AMu^5t4?cctAMomd2I}%^n1bO#jKfw?*!9aW|G_lS)DU9^yZfuZ?mw( z$EKR<>Df3tuM-F}lXbcqW>SC>&d;olkLq29*dDw0b_Fdg=k=t~iUTZgWb_+6gWC>B z?eBnuI#6H{MUMbx#8RIHOGd(J=`{#=H?RMc3J9NBSt%z>qSaHSQINLcsVNnpS%Qog zUyquZnb}>%ODr;8Kcv%(N76to+bxIz{p3-6XLeRGxGn$V)(Ehpz_@y*yxjfbMcVN2 za3Bf-3(pDS6cEsWtAyYXj*B4srMI8DE#(0q`E&9+N&o}`?>|aPOdJDKr`ic$ENu_0 z+F~!bK}FdH}T&fgaZjbyzRPLqYzT z@usJgAP=ghX7v9AnXb1+c)GdOSMymCO}PIJRmf(Q3aI{XlMmH{{<;6N#pwTTB%-4F z|7ttb>sP0nw}iMi{C9ONwYK5>eCO@(Wda+vz<+I&YwJltC47(5Gcw}*Xt28)vxcj@ zL$i_js)$DA#L#o$sO7H!sI-wN zM!#(*jaRFP`lG);zZf0ER&5g5Mv?lscf*C{v2k%BiVCI|8W>pJd3d zwE?Jicg`G+VGZ<+Aj%GYqm_S39HZL{mdD1#VAe2^;bC6?05WlTq98ybn7t(wA>q9t z@#teoW$D3Q`M!gDjxyltP|%jQgPD7L=5&YCBj^g%0RO$OfelcMyoGkX z*TxuP2Y|Kyr_Iamj1+?^qw`w_G%n&|;H;OHv$;X;u<97P|5#L1aEgsmF}E@GtuHB+xrQ-% zR+WqPk0TY82Ft(iQy3)BCn-n!GxPAmyMuyHz}ahBBjoceeT*J|VBpyqDHS`B$iTpW z|N0rlbiMkdZ_73RyRYir}=IyJpFU-&fE#BziKvYPG ze4AR@P1&#PQvR5d=bj!OeYwvs8mDse^Y?aj^mKJJ`W!e!6{jwAr{0g97!r4NGC3Ns zhR1x@)=^xTTbQ3yFrkYYEwkzk*5zp^}Y0ffm$ZMh0E}Z`~Fn-`_0z;?Bs2z*R`C>oskh z6!~u@a4Ipwx0f#~jJ(AIm)r|68wB7A|(ajdRepBMSf_TG9Y zKeqYk)&rluD#L$}S5ef3!Mxdg*JvtH>9W4y;B1qW8pFcodfZnU<63dHae4fNJh7Y&xW zqN=Q{;<2d9P}6>kZQ9a+fW~>()w!{w53<3xTZ#k1obTV;+Zb#{!_uEe>YR7oex>1W z`TfdFV&1~;@>qA%_;^Q$+n=9}U;gQf?Q@*Is!5l2SN1#@Yia+(c|}|4zG`o8ea1Xy zm+G7fo%0})_Bj`W25--^1KaVW1aSiAmmchzP&@Cs`A4bNq&VS6T6FCk@+QpMx<@TC zeA36h6>|Y$Og1z&WM*NJ5EH+&gxg8gXH-jls=yJg0d^WuQDGhePm$`K!vh{3Mp=>{ zRM#P*4LJXU*}kKd^#f&EsXX?xlga`bVAFyvU2ZOIS^`44`~?S-cz)2^mv&&rAxOWg z{;H10H*9Qz+62hw1ZlE7uyLX={Nu;alg$-#zy&9)%KeB-TJ|BIl^#m$-3GPG_ie{V zR_NthgqP~+sDD1%;Rldf)%OGZJaJXfAS1ulcERUZKQsHy+XZoQit5EC)rJBlU|Rkz z=jRDUd9IVXwB0{mF+VQ3C&}T^_2-V2;$=K^Y6-@Z305+whx}49Xw&3j039-?uQJLN zx8jxmbo4xY_^`CJYHx?g_9P?9;riQ)g>Y=;Mjxy*lPS@#m9tgaj4fO7lKrH}v-8o( z@zv0i=uUlE!SXVIBhDCAr---~CBw^GQ+vC*CW1(#Lz{oTc0KZy56RA!TZ~*&pl7#) zSR2~+fOV2886dr9;>1{?j z1APoTyx@?n9;vVS^=_8)ZE?SCa(4n~aA0?y`u;)#)MITjPZ8$m*8N>JxvOY3SW)d` zsA8_|n~pb6XlS79F1?fnXce8E?((BP_x-}gsoT~ zOT=y2v5|`HaL+rxm#zjkb8s{l%kv&U=VwGIc`006T>Hp*P~3bc&G19nAQE=8aT9E) zv8I=KTtEoqEcDE=ww~7%%?i1~iqj7i6n7H67)jfNPR#S>p^!d!{o_Njm*~hdx80hNl&8RnOim2)t=`nzsWHcAS#U4;o zo4isj!`t22t8eqWMMWQk;erlhLkKPtpp)&6pdD}Ac@UB__XC{TwjU__#R^7Kn%3B zI=!y8aQ69M|kijbUW4C((f1TL`A|!?xdC_gc8_rJ{YtN(}{rZwFQ? ztI!P&KhsYp7@=Ho06hITVYqQa;RXKFr#e?KpF(KUG)`g#{(N>PkB5M%1n1OrQ#Nu_ z#*Xd$+GyC4aZ%rA-bf?!FQj*lJm9D)w4NOg>5RO*N6!2H{ELTbphU%uOQ0fQ|CD=27qhF@^dn|yFQd8+QXtZgrZ?id#svfX+>VwsW zdMY9H#Oe#VaF1QwOOUhJ>aYqlPm!r;PFN1N_>F#2nYzFeE>V7ThQ=}N3Uya>5-5X{ zeb0}JLro9&7G*$WY=9lz_Y^ml@%sApS(?1B8)FUO4XKFjYI0T4;QvUR<2g!-5qN z2z}HiP;LxBh^{cwUq_Wmti0l}ehTHw?Aip(YR%g?y$gJp1-n{^gWu7?aS1;5uH$p}N+4*~WSkDkA<(jeXQ=d4Jo|Kp) z`3rl7_r8=OksuGxo?g?I zwk?eNehxJ%j}l(+DR*}bAF%o-vnMh#(a`YlGMj||k&dO7TR@tJ@X=bI&g0xyw@icU zsR8cbIQ_363c+T!;~7x~$Yj8jlg6f^IVIz^i>;%e&D^)G>`*wGa7|XSd`VS4p5@|1 z+PXGZB-5xml0j{Ihjx$cDfh>vfr5rC?JGkO($a^S8$riAs|QnikiW@$>y6Np=@Lgx zU>WVOr^6b#3YM9``w|coOqQ}rvWOCrSlrWJ{Ibr@jU=F>!5(Jnpf=O;+Xd|E(&F03 z_eo3B!8{RLJ6qd3i5AYzfm54Xa&2mb=o{B%>r19w0R*|xyoRj^^jJ7!eo}J}kd^WA z)nid}Eul;IKQ8rGYJ;tL9iGYj=5+i#JfZ@%0Z5*Bu&EILAimSMax>r{W2wTcTBGx;#P@$l`g4fNC9D+v~-z$IC~tph0z*F{F4l= z-WFsbmoKO#$~nUa`df79j9ucFu1H=@s$t@+LpAs0a>(#4ZKp z+`nJlEYA$J6BTY?9eBX^&^|>>i!ljRtG6PoSKR~rRQI=qN5+*J3feaS@C9b!HQ8Ol zgf5e+=6NTUj0a)`_TJ1xpX&n!#qO(~7jSIFC`-{87#eba=?hWOzhQO5;+8z(6WIco zQsu^rw&{MmG5cnRYpCA9qwVGsDN!WAYqFmJ8I*}fI!lJA3Tyx154qXZ-HjzzUz5F; zmoxl2Ov#02Z{R_x)wr$zgcMhiCotk~N#f%j1ufn8+0p+~r+aDIACx7Ojq3y#vt(w>eU z%{AUN%UeDO|}^9%k1#})uC zz~a{pLO2<$;ixNISTc@Ae*b=2f(UwTe1g_5l5i!PXuSPHOcwB~iqdz%2+E(I?(1sQ z>d@y!XKa49HP>UGtaK0C$_{&oJVQOchP8FB?D$^zZkQJ{`Mla;+R?#5%gBh~ItwyC zRin3sUxd%>{PY4L%WS%RV4w-tnLPj{B(XN8fyBSP<4<@51rWH9D-G;=_<9KG{}>n>gZ2L>l44)y_OT*WrlM?Y1ErUvHLD;|m&huOs9iFVm>z<=#Xz1YV!u|A~ zN4xn+p;w-XlA)m?%%(#~0e^!AF0I5J&3#=_OiOe54v5?rESe*Pl*CNH8#FU>2Cj8e zuPtCssC`yx@9BYB+IXNAHAvxcq-w%YwhE+{H zJ@=MU$d&CcNgtnqUx5@M`~&LAdxVKfZRos@b=c6sxpU0MV(_^CMvw4;+1rdGdK8Dv z&9$AIpJ#EPhu`N%KRe&Q`v0)^*I`kuZ}=##+pVHVs31y;NSCyP0-OU(; zw6xUFHFOOOHA;66UDDkhGv~#<_vicn?{}{A$GOhB=DGl#dDoitzUztmzMtnsz4%Dm zy|Ho6{=_{dhBUk2dJ*qgXE)G$a9bD`*}>;I#zUPE8veyf79<#@sXF z=*nyl|DT9ou|IGzVKs_1;0rbGFZ&9GTFoZsU&& zdDZC?F>!j`1#)Y4$ItP+2hcD1cphCv-S7ScEeFxC@XW`y!v}8pjZpmtwFPjf9;%c3qZJ-7NmARy?v;6vB|fCEqQ0T4{pVu$6UtBZQux*mvc zOha9dF^y{$ZQY%!<$~OH8*RW44$sOY)Ktr44gk+};*$~j*^0TFD*5|!pW?C#4p;i$ zAizGeqvfv6$r&AJn2YvR()p1v6ODU%>Qo64)UlxUNe&yCC}JCOxG$=2176e;-eLwr zAuFmTUqNaHLTmoxC2Jag{QOFy$e<3Y0es|-v`NeU7+H~Y{LX$eh zNxpd(q|uJ%Gd)k&^A(ncQ&=uzf5fx9j3JTYIG}3x7Na*nV&_nB>a|jP`+H6&PL43% zS#(RwxSS4x3F-)(&b=fFg!s3EgJAbjVI8t{ra}Usn>lnI;F-0#4eqRfKqmNc#>t0* zbW*d@sCtBMbAfZE>ftp{6FS{%5LfXxl>?m0-(}78J*_%;nqKDF+89#M_C?;+?kY z3G4M3Z}D$cCr$8#s{Xd|j#By7pTPH)%V!^vmQ`&L5fjrtr^BUAOx*gr>z>vqQIbj% zJ>Bv$iD@GBvo*gz(^;q9p=@Gi)}QA=LfwQ+d2iIaNlGE;hNwQ(kzA6?P)W_b9ZMRU z`7I}bGcJFi4PeZ2@|`|}^&&7ll?%^WVPC>_#@scI=!S;0DxEvGgre6DgbFUvQaYg) z^Xv1pJ^+}JeIutK`Vge|m#}M1X~Mb1(tz9zd*cj!&VQ zv|pg9ce1}+G=!DHBLhpbqD1*3F9)VlMzcAXmq7SzP0G<#B5Lg-yH`&}zL!w^U=v-? zza&$qT#GBOD9g;qs7#UkBg%7QXbemS8CHW$F_bgT^6-q6$2-8-(1>>Jm|`+MWCDm( zd>x4#b@;k$GYAL$ zOxX$NO;sY%ugP*w?y%t#^K{nMJQ+~PDo@?yS5U*Wx%t~Ld*CYM3Dimvi_8HuV>;mn zbTw<>TivNmz4tm$d0brJ6y45Ya*-e?*3o0jTP#_D4m`Q2Os{ss13okGs)1c!Y(T(n z-{(so!V#Mm&&&Dk@R4-fpg@q{TqVsQza&m_3nmrO;&8Cmo4%b4t@N1roxouir#-f~ zSUNCJs&gkm$9d!Otd$m|1|Tg2DTGR4ZCcugRnfd27y&OQ=ewOI%cRRJ9BSunQF6-r zK#;`^VBD@Z3ciG_m!3po;R+MZ&C_x@$u#OKGd2Ec+6DSCY@%LU;>vb*Aa@2|69Xwz z#_6xOdIcGJoBPX~Hy@euu^_Gj^zZW0B9)-q(Y_5d_iXXKm5p2G06$kY zsJjOci5F&*{TFpAKx)7xUg0&n0ogr&U)DOtapn};Pc6-0_1t(FUW#z!rlb^us#{?i zhmrc%!0MBT|Bc|W6hC10*yoU6N-emIEWq_GYivQyfA?3%_t>`(EF^6=hi`) zQPm8Vu&r*8gj4&BLQk$TRuHU2N5#a%cB%|=Id2PIfUMe=5G=9Rw~nkvKg=(zynt@e z(9yM+tv=a?o^_8aOcc!oGqSKSvBv^<9|pX8I=lC1&e5eQ|O@Cuaat zFy@*8l^UzC;h|qabc1MxsAnTA`xa{4wA0S&jlvtcFfv-7Do@&t*vw9VjDyQAEpt9~z+sU%%UjJ-GETKrhbrmm5Lj5_;4_q ztzLP5X$kc+0X-a!9BA_IujkBui}v@)uIQg4JK;h$#a^Mq9P`$DvhBIxj1R;C5tSUv zHdr;Z;h%*^*(c3GPVq4Nq{?GF!wx{k9d0clq>r3gnVzk+M4&Wkk6WHererFHsfjT~ zv+3@fEYDm5z~!>F6b+B86`EY`PD0P}r1ob=(OW0tHq!Hl!`%!q8X>@RCCVO{b^X z%S<{k7OyD&ny5XRl&V@osOfRqAusuD~lfNRjZ^U?&{en+6#cc>WZUEkogk4`fX$bDS zts2&opYHH`cZ@5hV4N`dM+QbmJN1D^{^eE(IWGgR$Zr6PM4I#@3pwcVPuC&y(qES` zr-E7!gRhB>ewUj$VtOE#W&lk@u3bYTonG*|TTdH;!G)c88R(8Q^t|f{9|-CsfYAQN z*^&83R)xJH`15$ia}^-xJKxp;{Kk(&KAQ5H zDBU~STNx~~DB6Mmt-rx>$N0FMU6v0dr~~91J?%UZa+{l0Iwxnqe?fHcN#VtD@jrz2 zhZ6gC+9FMQXM#gJ+q(;La*9jK5B6-g$B|w$dctqskaq5WNiI5;-mZ10ez`CZU7WyC zf<&GI0t&EtdN9hQ@ygASnwq${t)F1|fZjFdYiTCVTl|EeypWw81?nDd8OcVc8lLIK9y+n_a7`I z_9pbL)=7fFlpG&V;dt&_1*wVxH!#jaEJa{hS<(TA!AV7IilF*H=E0 z8=UtKw*lnjBP|i~g#Y7NhTZYoT>!7Uu67duNer+CMmzPdfQ|a(LFDyMi`Xsum6i6% zTF+dy@fH+u?iEkW(D#o_gd3vod2F!_gXo-{y%gXEefvN^9zpN3$zL?|K7^O*1u}5i|8=wd#3Up8 zzy7zM;C~*ts}aB(|G##&*Zv=GXKNU7EhiJdsP21ALf&Izqqk4DE|E4Al(480dTt`8vIXI{l`b>AzZXf$edt;7% zqC3~-6@o)HSy*nwd)zZ8JF%Vl+=D=5pzBuhD0J$_k&+x6tFw;}c1JSqL=EiQL#Hd{ zIfP0LmzmNScjKNU^d|GW&@eq?q^^02O37%NY$}h7j;@S!xXqp{QgFkseEogaj8$BV zMX1^{4n7{Ias=r2I*d}7@ISj}Tbk?H?Pk~I;}5-fhhxV~ zEZeV%A{39x%E+{tS{L#XS`Cm2`gC{8$_R8AaYKR^JxwEYvSzeNquvy7Jld@xB4hPA@$NnHKgYK%Mo>Qlc4Uin z5$P6yx=e7`2)@{wkattIGXpPkcW)Wjv!yBURgTE8Pm!NIG&=%Ah$SnN`5=_>jw4P_ z>G`ZKog-wFO20IG$s6$@&DCkwDpf?u^r9e6)GH|>vcSQr<7aR~ zbT2fgyPh~EisfXzl8eeyITzNEkNHY;J$f=N4H`;;nHddOpt*MKr*OuJvF<^Qew5K# zeVtmpza;T^!C9Qdt9dn8QUY>2=fmh^?*wN`=GstNj;R~+&1q5Z1pD~W#xoAyXH(oA zxYhYGY!iAA3&bsd<*&roiM7QHT6(?~5-8EB_rJXV_Lnf@=?K(^urx66NBNLarPZX} zZtaKK{It2t5DWaeOKRD_C#f`&gfukt3LnFU%fim?1!%{CrcA85y2}J)!o^mCE*}1S zceU-EkC0@_I$wJz>mD)1c3L=e=YH?B$$E||-?Ncq2h|D}b8~gnzsk<=RFd32j8NJX zLQ+RJiRnXG$c3k6m}r@=Chd-q*%L6f@s_osYU`+@wawhZ)7bV7&eFRL&m2ON+Hvcd z%-uciq;RxNx=3JV_KnM^y_glEi67-+EGF`Tn!cF^Y9LM6?@~XPl0pazpe^y7Z@AN5 zRO^P*#ChP?bp2rXtxOv7;5zYSV{pbmOrbRj)C8b)N;u4N1H%pORL*g62~m(om(ebN z#Clxp>{=L%s7CAu-epzju1!k3U2b-KGW#1|#<)QADDhjB^Rs6xFPDsUrQG$DQ`ML% zDvC`P*QP~NHoLrUBuwxPIvhXXX%ItiMN49DpaDiXG6O=3lfI8 z%JQ0(X2Rz}yrx>+4+=IrdyR+}nEhtrFMcLo|B9Lbc6KocOf0^}@QsOE^)HP-;F4%k zH^t?Ao5A10t9BEv%;`&yoBsj&?1jd?H!;Yq?=h!PhHSj5ko)4!zID6E@1ho}2^M)` z!x!z#TLnUH=j)WBCL^lbP`Yp#HIn4ZPM7l{U6p%EoBijpt;xf2OIX;;G;&x*t{AGB zY$QiY)vt29{2XZPjQfxc=7LcfZ%LBmoBw!v zlU2l{v)WLH+J^=86l<@ASu9R#Bg*x6)O1;E(uZ}xS7TEgsjEQq7^;|rp@II!qhP9m*ZteU(Za1o=w1#yhu=Y0?hI7C(RnZm3P-TD8kqk z2a9+Iqgz-16rhRv$5XcBTalmgJK+UaltC= zppA8qV?0RBHbS0upanuH;;W4v<2Rh8rTL4C;%$rk^gTvYf~t-*SwhUxY+icC75kG!Em>yYD z2U(>3SX#Ru>r1Hd86(9z%%c+)UJT7edsSaLBw;!!mhy~{2A(Ve+G|HeXZq+wYpr-b9X4)${ws=IZi>u492?k;l&iCZ(D zu1%=AC1t1+msf88kP_zrrMF=v}LsM4bwwlY{9V!u>7mzMrpGaB1dTchDGrQ99> zrw@1cVWeYqI&T)N>@RdaLJ60=dZLh$ z6Vy*Wl#5g;RZ6VcFJ67WeL3#XVJb&!theP&RhuRvVt9^e64hw$?AQ__h$eY7^^LLm zeq_Co%|5+}fC720ov%^usxo^mggiEdJY5UUu$DE_iF>bt*!Q5o;~`W(Rj-lmD3WTA z-FI48QA00Y6_%uGODkkGueh!GX!K~y_$Rm0L@|xqm^*T#hbACD<*mkBEpi8D!`ADq zC5@BIqR#+d;T8BquA3Lm`pY8jud}8QA|^R`!)Yv?=t<8*ZG(mGhPH$;f2H`Um+1b_ zu~`|CBF7qLDuMQI#I!zwm)_fNBb`bX1)eF#VrZ52f~J9}@(KEyvh-YetC$3rgs$#4nZZz9-(NV-?(Fx27Z)7KmxwbDBHD>DBdHC{9 z7sc7aPIo3Ixk$DU%Yz%fk;R>dzPnX!dANBVeN;Bv!J{pf`qpYYPinoyoGJz9 zs&nTTw)=0z<|VorCwsjGi*3 zEbSKZv?U&@p3W#2`yt&$S4h9}hPKn*56&NU%amyuC#h8RXCKb#4zn@~XIhFW~mZ3FAc_8+Chqeqpi~71ARi^2AMne-gBn zW4!VnQ=i9f+|o%%7@4BY6lX()M_jbW;JJ89&_;w))a3Mzp|^XcVS)FS%>i9xtJEg! zB710*)A%D*C1`C*FUx8=X=j}iO%e+STwef+wRhC*)%;K} zPNF{1qwgo3oM_WP6|hjqqo1Z35C}OOQ(yhVQMf8dKZY5OoBNz|oV2a0TnPm`QXKi68piJe5-Ur{6Q%Zk&kkcY?piD`&M8711< z@~*gRloX959`WQp&@mc8Z{5`kRlKreF4$%1kV9uj#rAfLb+7Yc-ZTMk z0mTTirfhRlBokt}NlEt7Q7xNI4_~pz+4&J&))P16ZSmLBGgVUkNltd!tEKb)Gl|@g z2~5WFqT>0{J{e0<=|s5f-gI=vZ9>UA*tMv}}T zx?I(am6el`qo}H_W%5fW;DcV#XgYFf9O|Q`;trJK6JsuplX#62@Tg|@e*Uehxm?<{ zF;t<$bxf%otaRp30YOo&SuNpEvrHh4Jb&MLXwtetll>5SDy^q%cTR^LeUeZ}ajzme zF>)ES5D9c-=_F%l;b?+--uNhV-f!Y)%Y1F_Cv}b0Pa*&bb^Y8wF-;H+T74m^7esJVMNoyl{Lb~pS!xVc@CsS z+nC9@&lPn>ZLfg*5Wg^Ss_VLZI`0AIBq$aBl4<8tXd>?X}plBuFLzESa ziJ|LH?obn5CNpcDOdr;cd2R(SIhcH;5~esl?d=GsK(Obf5fED;^RfJyR^3N8Jou82 zzctCYudma(iNzAr5TnI(DPbEzR=RG+&Ica)jE0(HtvyveFunOsN$;Jcogaf4$~}Xv zWD{c@dwZRnrGbXli;#P{3{1*1vXUE3ldXg0r<=DI;tN`JGec_Ts{YnC2oI+PiJl`! z^eD47qVy{2iV&2rz)u}}^U@ud^N#d9{Z1;%x~XT`m67`+WlOMcIAQWi55<{%#pvj;xLDYaoJfFjlDX6W)hXyMDW9AmMz?qzmuHFR96bbOEfl!bufk*vg;^Fz)mdzg7}$acf@)^5D^Cfb(3f-KBIm*yQsHT#j|F1 zhh)l;MduJTeG?LWlUNQW1y^*l^3YShkz{^#Myngbbv}fjVS4f~-M;O*5oYs|&lyR+ zi^mSI7knJV;zB%*KAbRWpOuCA38JzF-Z8A5TQ*l24J=u-b@P}l%6djCf2+h)NT zDrj#z`1kn8baLl%gmN;qFuUfVer1i|6ZfNSTC(1x!t%BgwkcwR+VO;e^w<__2ochd zg-2x=Bir3heI-`xJBc=#aq=*mTS+;ynT$mlQ;BGwwiM|TG}l0=C0e##@9j;ve2Tfw zT8D8tZ*Lpw9^3b&-(&ZWq07EQ%@inK8V0kA=Z5d$Ap!!nC)i2(${vcp1QDr+0K;~} zW`ZX={(8k^;GD+v@swns(q87HB{l*@HN{=;O#2B1Iz@bZL%&C@v>7&2d5={INe#Xa z^-MZw74FH*U9NEfKuH{ji-2#N4@Qu@|7FgpjPRM^vRL=;zb{p~ip@)vS?}nb6ZMsG zN)M;jbQBzS;Jf>!3~huLk+vx=`L66msW7g$2i)r{$k zht5PiZQVf}d{Hz2bNd@UWV7Soyty>8chS9GA=`c*wAA?WL>wo~*yV7OBUh(auoBLc zRMED$caio%GCarblzS3?DC2p~T7O#fvv`&E`LF(re}U*9fHdGIh%MOIYPnSCwHK_cUFU&5mgu67>InLn_S%W4gY9@aHkbr8`7HGV1s&RpG0VCqVWf; zs-3tI6^y;zH4u>^$O!q{T=qqN7Z_iNRe=Q(!K9cso~VkbtCqp8(+)MA7T$)Qf3D*( z_W$0=d^d*G`BoFA|q#V=oj;wn-&3 zGso$`{Hg>jv`Qz#HMp-r3u!q;g#D;|fuS>+16U9cr3O+mMySW!RWs|%nGOXJXI#8T zw*A*^!^LECCF$&Ux6~#nwmb`ry(jhvXj8Xqu`cnsmC&9NFxJDeIx+VUm-(7cp+cba>+pgjqBqClt7!LIx&{ z?Ov%~BCh-nG9E58QrYT{&EU%|vscqt*L8bs-E;Q0nz`-l+Ei`AOF^a^OmGkMjydE# z_mcf)FXO{(K^v~9aAG`dx8tvEgomwy&sH99WL_1Yq0JR&|C_YxmItma|M*qgVh@y|Tunt#=Nu*t8o&PUME>6;s*+F3MRu{x}5 zIAP)YTbXa>qzl{Vn@8r<9V*H+ z>lYLIDq07!VQ6GqOiEdY0iWQxjj5*T?=>y>RdB2GLNqN}7B zzuOp_@Tkt(Bv(^7dfWy=1z<;6EUmn*yL)EtZk$UCaMsMswA__8S#P@fPO8pC14$RM zf(kA}qCPg;iW@H;PwhTg#nbV#u$Ayrzw-1UNfH(^-aS95MYpDMSHwOxB2^)tIt=U2 zg-2^pX%9)fUQSHsD#PN6s8cx8UT3?E6scg&w*zRD2&ghG?-XpE)FAUPWRWpz(;J0H z6%KcCqbNXrnrMl{{rlRrs@tPS8xbexcNbAlY6$ohC{$PTc9?#=9F5I1V({C%b+G3) ze_A~%d&5$)GjC@j-9_Ljx1Hq@a!0lkQ9#UY6^YSHKzP@DCTJ?r);`;mejq805ARUf z5{i&v?XuA2Pgm}XnB|fEEJu7la2r0BQvbjxjxj)~x^8@^LkZdu3OaajXYL`QBsFmi z(kx%)OWjuYfV2K+x8GlT7Q23$$T_sw0Aj-JMoj==-5g1in|mPtAR^GZ<&mT$bCyOF z*uzP5{9@Oi7`p^R=6oB?MI5VtJ)G?McJUXoLpl@g_BS`>(A!^Lc|}mN#hvitygPAE z1W1%OM-9z;MPA$I!GxM_iRt)I`41sX$SM@MS{z>+f0w7VfAK2<7!Sf)Jl5#1O4763-bSTTy$6eg1 zXmUlxiUWKSxp>Y`bLaXF)5 zk?;J5U2?HH6dKZYs@k87f>KKyPL*6oPib_hzq?%ul7=h~`sM_EH%SnQtA_;1sOkA? zMn$XdzuFRdNDT0v(E9yVSboZj<1Sx8hg-LXAXy_zgyu=nXvOQfxdChVvCp%o8Sll4 zSNKY)ogXoThG<&3ncAym!#{(N(@iukyyp)i(Eg2C$0iT<9jg z6w*Z1kY#&=j}}%FWkLG~b}-yz%galD#fQ?u6NviSU*jHXek)+vloi3g~5cVF& zD|Hf%Fe(o35N_pyvLYXQrRUdPNXhM8hip#M%yk$jNBO|a89B|6ib{ui1ZntE)ft!p^*l+!Px!#iqRKOgZ=JQ*i z*ffa41txojux`BhF6{938~~rdZ)uakLX|s z<+IamRWfaZ{%$^zZhmhB3zrUPt`QTi?npy--nM!|g$;CFTin}-2br>q4u^`_b;f12 zxBZ8vZf4qcy5f`BqK_b9!=-IW(#XAyNdO2h#MgfxymIx;htc<*tDe16*vf^;FKu@yCxR^C;{INZS;G;pVGtZ^2$TxFjR7 ztx6cr3A$#~$bfhOI7y+%Kp@{J$=BeH>*;)TTE?sY&RQa2JsbY*V=VMD&_V9 zNvV9ztsWqCMb?*XqOeTyE_q_s2)pCN{%q{{AUzS*k9HEC{=A@aZ|t$$7mRzw9>KwZ zN@jhgBry8rLgjJb83nj?Cgt-w_4!De&-<%z(hYn1<4xO|f@6SVM``Jh`@pHBoC7 z`89bHiL0`JmblP~_(6WY8YbQe@$Nn@?m>JQ$xhf4yd21g$|0HVN47zx+S?ZUx4kki zislQ6)!8kUww44e%=b`a*4Yl07l)GDxxL1sRbf`WY)-NB75qN$;M#Xb@irhAGtWvd zz9iUqWjVZFnx@20zh$x|oW?S6_OVcG0fH0Yx@QD7As`6W4AC8)O)*b)0dLA;Qv@uJ z_&0zKmma)%NPoML(i^Xb@rHgM9D0?Tb=MF^xKL*VqQ{T6vbyvEsna(!tywXgu14sd zZKyV}mGpNXgEEK{%9f=<&v!n@Wwo5$!cV5)KtzTmgPr?KNYElVhb6WLK5nnK)eMDP zc|QKqN43K@XuCe?6pg2GJe+~;y&}HU>U#2G;hbMdgY*>e8~WYx7G}m8Bxa+(I!$D1 z@cp_KsGG96jE_}w=Te5oT9qo2ay&9UNp%5ohM}bL9C7&w^0kLG>Vjh?1dsE{r}GD1 zhrvO92lN@Xj@g9*=aP=4CF)kF!lM@Ft)t&xxtK$`*ce`tde@veJDeNS2-uR~0F;)! z#B_7yY+1?D!{uSMHG9aWRe|8tAzsD)jfa!Vy)i$246RZQ_iH3(hgF&u{5I<;M_QEK zkJs6%FkQ76ELfT$Kne*VPbZ%|*i5Db1k)Pe&8Cc9Ht|zq%)kHDquLr@xucOhm?gc; zgx_?z%1^5EvGl;kU_@z=L*YYPy2E4PrgCK7nJY+nAjXQhmtts=Ht4HH{ozCs5i;>-m%($`P*y$6qnL<`E6T`v5 zHzC1ZiHWVwTZs2fEBeRVxm#H?$rxK>Z;YQqHW7BK%GEnOcMO0veMdbb#b6p8Xj&E4 zhSz5vN{8y>1}Vz7voj%H@EqFOZcKGRHLzk7wVvE;S3b*7&;{bPcGADoXA!VF=6e(1 zy_EQ&!a4}pcv%>rmaIbOwzHG5cH2Dw5B)Hx8cp$IgpQ9`u*~^)XJ;s-p2O0~;lgv8 z-12CP{eKALQ5_4-Nr%NUwxRszD`xL{MW`&_#j0whoVdt^P4ld2FTLP5-jq zKjrETew=Q{t0{O6g?Upm@!QGwjd3Z5^QlIb;o^357{wT8^&p%<%mJi6XE)0?(H*G3 zYY}>=coiCEE-qky*y>s7j#n}{K2&bb@f(S5@sD~)&K*rqCDxXE<-mAO#o#@OV^TR&F1V?BdA1?fG~?i zWO`1!aac_q0RUD$Qk7XwF3;m>55b#avQa^Yu?tj*WvjN zx(jlf>zINW8T~Xv#ev=BocfZy6)(&UYq4l4HLP(jh5yIi+d z9eL|LE8cu)rm0akKK#fHu*`2eIuru^8cM?wongE^#c?`i4jr>_C#0?0`!DMLyf3}&3~&M26iE8P_owR#1tbLznaHeSvXs5sG*Vt1r<1opZJkWrLm1mG;;lGl68%-q z(CC#RO*o&U^k3ILoudKH&hL8Yj5p(D!?LvkiEK^P)MVB1z9|0S+Pl8(G^}^#;m-ItJe>5+-Zg?NP_HOQl{nF@6ba6TicE!-$-2CVvV_VSRiuK`!G`ccd8qtSb@ zx9Pd}h-6Myi-IBkM2!XY=RXHfH#5bRfK1D}_AHe!4VZu)g2zwO(;wBV5)rfdyeKWl zIoY=`6x4a}={{pYX}k)NqL;iY)%(G-m!Odm&>G~{c|}l?A^pn zeaypV*1)PNv7D&Rr425rOX|GwFe0{yM0^2;#}*pOb-bTdyFzo^b(7jBw&Fjeg$LPxT%R<4qCHu0;D%0^qM8l*&jjxCSB)olIMo^3A7^MS1R`t?mAa`GfA%ZFuLn#lr~ z&iX09mg;L)Y#h@kD<{nijaF$gW9thh4t5D}LXoGd!yy@e9t<$pZ2fak3Y75|{AhE_ zD%ob@m#EPo2pK+SN)3}uuob|W6dx8E5JSmgd?7z#u21H`1 z!CB>pJG!VHbr0q=#(z(K4M!|e|E<%;yHTKb6aTj3bK+biR3y?!r6ep^Z$Yq#0$L2O zX)gsQKaj;=$+GUDUK%&7Kg`8| z{v5NEfam@1jo1GFd?c+q&>m16sGs+Y+ogle%FPH^s&1LI_ zhtoC*wMy^DaO90=KiMX zLKa~zhnfEJ5J8ZV(p{Xy9+UOvu`ht===@^rImRpCsxa=)F)OmbZ4^8h#S!>>ZuDf0 z&APv%^6;1n9?Ie5;^eU*mK$dV5+r*?SS+ZV9@C2KQMe6uJpK2>DeUJ4msge8fddQN zD(SMRSN#UAj!%B{?6JgB(`_hn4O{;5rx(?=`ENL;4Tj$L`YH`Ky31M98gGaIwOc+5 zPhDDmS5yD5t|>G^$>kg)9AarRA}#}|gd~gbUqCOrCR-zg>hxtZDw&yNSXpI4=)k6+ z|D7ckxh>P61m){;U)WnPS|8-y_^zqO&>Bsb8AX=~QnPk~`nmmKaZVh$vDV(Z)Nc|g zx+fv%>?K&;505i(ZR41shAZH&5USML|N40rUB-mhr~i#k$ND?g=!qb~V@sz80r!bX zw)8U@&2sa2aj7RMXs)bbgAd-$tY4lMEfOKHG*+(Evub1Ge(Zg*Q7h1b-Bk&QNlaxO;P)*h=IV$#255&1}I zl^^Y&uT-oZfgN8bGgsUl-K;4Y-HK?gXy>aVC4fmmK@)!&hTp;Ot#N7tq+LKfs(l`I zT3^!!_ad{hXcMuo77~=^u$)_X&zHiN>z)iKolx)PpfANTK{55LBHmMN9i`*VEd1&? z1|JO-K1{Fw&t<=QNXyKih7TGUks7zLe>U|M^WVKoMr^7xl>A1o!{(@IhOMEj4X zflh3yZ5&>00hCQ({C7A81=3I<(*@u!|GUr#u<;qj0^Kv?@H-O3@Tb%UcF9;>Wi=Kb zknnum;@{Ytl{jqP|5|j#^!fpk8_?R{GmZ;^R5sIU+-Pfl6}xT{$l;R4Sg!|8m%heC z=Nrx!iPgIb-;69o0bCJKVrOP=zVvzii6Cea!}9KQ+;Lmt+06R?Y>{bNu+NxFY?pSZKBZqGY~6bMci$>_t`6L z?Py)Gp>VcEIZ4qg(hGu|Co`gpNC~ zw0tuK5d{3$a#we9;@A~hYiN1yI0&_)4%ez6C{9a{Iw$?t!f$C+U4+T?6RDT1n&HM` zwMiyeBioHH4?@hr#?QL$;|JR!z07RSzFxcL;!PoP&iHYICJ@zCCvA>+K*KD5F6I@5 zrDKI?KS)+nR*Cmud+E??u;@BLnZ(S>(An85;HrCCG&oeUfMYw8{*U9B?0pjw;!$pj zW>2xgesZRN)=)oDR~HzkYF`S1aRw~ph35fodaJz8!o<`>4Ebcf}dcYsIEOR6*A ztNXo?3+UcJ13}zQ{bbi_m%L9^!R26^4s{7_K z%bTaA#HDmfe9=(6ZiX2Ro-88)ERguOFMZ#>xfQ9s>nl~r6TJL*L2r1R8W^F{TnucV z(JOg-!m4c$pI$wZTP8p@R(u&Y+IJ69 zmU429yi7>t^_pFyUEE3k>W7cFVQOYP+%mO)Y3A~b!9tOOospSF>1t9Zy9%3Ii^%xO z24A8}6_{sSu$sAoc5X-(!*lONRF9g{_4u~MFVFv&1O`{`n!#7K5|fupGHua~ux@g^ z`qidz4`0~@LJ3n9AX`__yxfV{i zs@h3^%wcaBIgs`_A0rwf9yAL9JgTg+l=CDD2>qiQ?N(Pievck-qsPSO=Ya+JdD($5 zi8Xh;#^!Up-s^_K=x&arzyE0Vd#~?#tUui!ofYMhp z>11Pg8V1Jbd-|{S+A=jZx;(yECb>M1O@jr(Ig+Zqxz2a1b70SWeh8Y50^t@(bj)39 z`@6W5m|cqy&^me0|0?Jpz4hOtG5W+awRNH>M;b^DYS{HRPLG&9*!1&*Hmms@w9bag zVvLQ~BHx5;d*Yf}7|2or&y^zfPetUl$wvt=#B7m1u3Vrm;iIf9d6*XAX?M~6nTx&s z$LF^ z`o9I5vj_^3-q@H}W zr$`Bm=Z_C3kx#;eL|m%~KY$XWWUC1+wRcQD3{0#^*yW-%+;?$tVe11^vk8yHGZsD; z3Ar7_Ue@pwZpPg=JiA?fWiqezz5UlQ+v@BaA=A{x_1BYY zEb)6&iS&rSlcpR_KUlh7St#G!xY)1rKM|iDZr$5;a;Cm&3eZ>l+#Z>z_9~snbwQ%} zXSmmd%vI?uI8p*;o1c`Ix=_Q8% zi#qY@~Fy1^4FKR>X_{!2KYN;dmtqn2qn+b|BJHk0Bfr0){UZqB1I7qK@dR! z5m0&$f{1_^k=~{EBE5y6B78^{=`9EX(xrEzs30};7CMC9ODKWlPW=A=lzYxS|G7TE zs{}9XST(mFA^~yx<5t_gUeBXPUzOTw~vD}6A;?|2V^-Z^FhFJIOs_2 zD)C$n0ouB%l1=UP2E3;$(%x*E)5vgpq;5lOYO`i9JX~}F?n|t%cfIq}gG8@H2g#m= z1Qgw3=Uod_%d`q-LX1|=E)?qd@30fCvo>@1{4k_%0tV{4zgC9yy;JG89@q*VsvO%( z^s!KrQVn@@i&cDQj3)IJ?l(aJD?G5G0_2#L<)uX;M%5)XOQB^O?{m1dJ0*5d{Ju^U z)4nug`@!C@VA_nHMTU)@4(#kPeE$m+>SSBAU3zs5Fd6z9O5&;{hwS% zCF9NgUVK!N4@O^?Y~)pJ&?(fbC6*lz9BkXnHn;S~(8~GPF5ApT zL~i)l{>opcfP7P{nv2e-Nysj5qc6F+aAq1=>ytodKG>U3WKcaOW zG-w6{l7@%_DE+-;M8>ozOrDz{Jv+oo!pR-|?2e0Qn@~o&yO9c%Gx`6plLj=pZsiDu zl(ICipT*_1dA)|;0iSh+@W*RvRg;=l@vTd{S2Jguc%fTD9`L789FFjrl7sCQvbvJ+41q+{%eN|+-<9S0^XzXYrM9g74)b)afTdTQ z3^%2u#N|)=OvYxCQ@5=TtQ;$;7bm$}`r%bpj6->MJJ)VZ>KLhyZx3%TOXr{XEY$D= zyI1-273%VxVFgOxw`J{co1djg|um4QISe-aeYeW}BZT!{I z!M+Bp&33cUc3i=;YXTjGryycYdP}CI=qudhJT?OKQ!mflUCsJb@=9&1)=InB&vEbl zyA`8+yD!N5=8JU1leM?8#d-PowUibX-Fj=orNMD3s(0cw>OqmmQ%hEXaoa1LEbs-@ zNj!LQsl!Aby*t%jA;UzeYm4_J{)mS47!y4dn!mh!Jt;|<$3wc&ZA%zcd)U4|Q7CXK zfIYLm_6W@{#bn$tT^~xVwOxyGV)XhEY(^v;toCiKdF$%NcS>Z?)A1ud=LD=GB?$a>}9HJE?SN)_^1%{7aVv{$GV(i2uR4}KK&dRe)H9G5UUP)3-H-{-z>Hdo@5 zEV9#y?&i*g*3Z0Ut;gn+I6mMtHhRi&`S3e-=yTTR+MK7=vzD<@=5dC;naYqwBRy7> z3Y0#5ZTGne|4^#$G#L`V96zzOE*Oy<72A`)*A&TX$SikojN!u%k1WvKW&w?o8ZvY6gnPs9_JhU;P!$*rIxHAkGkW{rTJ= zSb-djuP@wyuYCR-!75zYVz6DrP?Oxnu z#Y)tEidUQOGpVfbcZi)HCV%j)$K5R{xY8WT->@~Kg7IonBYWS7I23kd29OH&@;AT4 zI)cv^0sOQojkNPZAIyGS(+IrO-L$WnY5idDy(&}7MP|PN8>=t3K$k!`dj#? zEO}6Q(xz*5e6Vb{&wae3$a=JXf>846%4x)<4hv~Gl=$nnIwAf1*4r%fJ=Vj57N?qI zzTV?#$dJx}aZV%ayAFt-o?`Rbp)OgPhZl_l4?TI(|P;Ee`c3-t14#Hf6_Et6= zLA+O*dgHQF5T^Mnx49~G`zT&XMEH}WpBN9Kwkt^(r^#)`|Gp%?qzqU zFAnJ{-Y>V_B(or#iEDLt^Y)4S`Syz5`du|9vx70U*80ja^=>_?k;KeTxeppG)&_b1 z;GMA?8N_!JU{)PlHz&7*4!CqtXM+SzKe|G)|mIn0dD$MLghkg6{rCfI>r(7UTr_`y{>cx-I4c$TUq;uHt|K;oPg1(NJYx~B z;R9D;$W_lTag<_AVr{>dK08%g8|++2yAaGpO9 z<;-G5bVLE5=Kz6CS(HdA$ z=yv2V?zleXgxG#G|0TcC`8x2K{Rl)fqWhRqdrHTqKEcm%7j~IBq-QJ_2 z>&eO6Wu>z=%43pNX7^@6uVZQWe`_TzAy@_!rYJU0rXw~c5d!Y!>>fR}?AZwyX!JSpW~kWuX5nQ27k}g; zG#y(seR8zHUAeV4h%Rd=6=;+qB(MQQ2kHJ*A*~ql?$(XphM&1*S9YI&di#u7`VLNE zVy|k_WxqZL9L1*w90~w)=#%NLVG0peu|uJXGtT!B`kQ|CUI3A$H_2=yyfS5?88fQj zqyT$?9cy@?lAMhOTDm=LN+7_Vz`s*&4FCXLDGAKiO?PcM_A-saXmZ38^?*RPn=kpn z(57tfHCRR)S8j#B7WEUW^qc(^?|*n>!+WRjWAssS*PzUb<`j+#^ z#>KT|x~Z4{0nE8*%K?9~h28CM@Q| zFygDnp1=qc7@9V{+J)}Nr|Bge+crPI5Nma${Dw20mz$5L(8;=^y(>-3d>D9TESwW9 za6|gr_YT7v`@?Ke({}L&fnu4pf=brp*bkSC9j$HcYw|z+>d+i%@VT+M3Gle_UMOqv zXeoE4PfyI!WZm&~Pr#rr+EPDI9zkrJf^jLI8dXSdGJ1^TAN6V97&U6-LQNqi_8N)a zLwJoFvY3V6RT=?6oB zK9fEPU$|($8c7>CFCVn#I`t#e_y{eDpksZo3hhE20NjCiv3|sAV-CzqWZF|o@j>i*gR^e=c8)L-y(l*hSZHa8y zq$}sM2wqWPQN$k1b5hvznY}PS&k9I7x5H-dRD-IsEJF`YNFnAlv*Bra$l_lu@qCT* z+M*k%6X32VcEJ1Rw)bV6_bRMXhg>!`kco7xZYv-c%L%_Wus%8VAHfkHFFylVy8h0G z!ZzGaU16oBJI!D<%YFG$2q(gbPQ<=SyVJbU!-0~HTIZE4M=C_~&Vb>(oCO&V1=&PI z6v#j%C$AUs1UjF5ugfW42QVMuHV};R+ZYEUiCpM+l?aLwLr3A6tAIWt#nzECQn@P7g*pYu*Z0{B#7PP-X@ zIH<4gKAYcHcVA+8=ykSU=(-J1QXC3w|4X`0=NQ&3n?vl_RAzYYaDA zDKu^#3JVE07}Prgi`&uSHLiAafSUmhJhrUsU~6Eg(q(G2P~V@}k>Z`NWj|WdP*h^_ z{NXgnAdQ2mGmSx z566K?@OL)m8wNkh?Ad@G3>nMdoYh@j{mej=pK-`?jpI+PGaYe29?Y543R%aMm(=3` zx7K|Yb1&(x>}9$y5c17f-wnd}80P?)TSZ)}jG_%`91L^5XS_SjA^rLS2P&+*obGXW zny}+B$i0zOwuhaO$C-QE|B8*k-8YnQ1drA}NeuA~ZdPhCtIz>tUgm^e&;3(A6r`3x zh&*|(4vXaF*QvT=a>xoje(84zc!mu?f$0BUs9DV_7h?r1VK0}Qh+2^%4OP+hVmI(e= zKhq#M04VKl?nm5^LNJ5AHY1U9b3L<`P*G2uE4pB;{=s!B`6HaPXVcsYy3QK(e%hZA zlYbY6pVyukHoAKL!>Pl=KE<%wafqBf*9%e=ueo2fmmFt$`Sn59VV{RKKC;5 z{WD2m(Ess?AYHAGy(RR@g=@gc`-;4x$d-aO|*_}XQ-T)v*7a6_!aRXJ9mruz?VjKW) zg2H<+k|)cDx+5eWsa{FFnVQ+}DN?4+))tU)Td63bx5{Cc&TGuVF&@R|t?2I&AxT<* zH?p-a$dXfN1T zKh+;oan&9KPri4sv3Jy~UatUd(jx~nm~J*Yz@y-P>gRdTX4t|kn6--uu-lZQALq2AQ_8@1S=Hme{xokL1;@FRQ&9cptI0tV`Tz$1{ulos*aK z@sBfy^G$s92SFd|vzX3V4-!K_F^#L&rh2$5iR)y$eAViv-$>030zSS{^cHSN!6$3% zAAV~^d$q;!1iNFWRXO%TO<%vnW)|d#q)uRSQV1XIioze=ns^=M_d?^;*U0}i?T$ET zdD9;g12f*mG4VI{Q)T!j!Y7-kFK$*K%5q|ZqE7t;6MCXg>+n>a#K@TW-M$j?2uv#m zoDHymJZwK+(74WVTUsN4Ea6{g+sZ0R(##C0GN%2r#)tFjJVtQGy*7ZW0JWzvI5@w% zEL=|vP;U;aa~WiuJ$GYvu-lN?=XXz0PlM%T)pICNVWl^yh7?`RB2g>%ZFdV^&BB;G zkLxwto&4um9i*K9_LoX6d`G#IlQ{5QNyGta9-JSN7Q%~^G})eNk}hkdzQ=X&Q=Gv- z$l_?z{{*-@aQ|ZqLK)O=)2Xw^y5JjosId>RwXB|r?T{3a0MoTRsP8{Ck?CV)IqtQU z<*;;Vn*?IPxAS6r#G_jugQ$=~_6Mh+EwY}k%8x>S7V&6->0-&(SLA;1AL-#vHNV>y zvGfS|*5Sc2)q-)q%Lea|(WHyJvk4My;I@?SR8IhOsHy(l725_jHf|7Z)9KvqDTZKK zQxm=~<^jMh{rvJHZo|%vc^Tcw;cNP7VAu;vX6aZiJCsQIOuclcP*MV=nc%tS88HW? zPf0EeqNdydsMNm3!&uSEcm+9gS%1RrrA$8P-fmelY;|~qMEWKHCVB5@htD?4S>9}R z&=5=NW3aJ*nQnaShpiA-zRmf-&%iYYM2PJ&TOilgBurMYR7B9e5-g_o1>9^F1ZLCr z;&=z&27Am2<$|%u&SxazfNm*wEsX-_M42|JrFKc5MPqp3hTt=)mnHlYu0sHVm7$wI zwuIqvzQxSK>A&@_L`MTuF`DC=83Vt-A?1B6b-sOt^w>&DF};t0k2j7n-t62r>X{&O z`X}v+TKn+yvEFF63|vsKw?T!C1X9vfsnsJ~zHO7&51xW$*~^t=*ZsQGS5<-pqDP%3LLyc4z=?@pvlN?N1~I6i`#>WHF1u z>e-;(CDEJw2yKT>L7sTT!UiHDzV#{ET0Fkhy=rp%Thn0)Ho1uK8;pv*cN6Fs6t!vZ zw->a-Co$!Y0MHIypy$68!w6O$tH$$$_(V-D%^I}(PFX2AJ#I@R=+EGo|MddrZ4yKT z%6P{-PDSoWenY>`V#kU9N~iJ%cK^3-7~{3~hh}Y*sr2hzEa`h3Ch@YQHYlhi?w-GW z>x|a3y)9S%Cot7L%RhHaUr;V(wjeU({+SDQhsYOi9yDH7QdK%cQ}Wf2C_kM>5TLpM z{atK?h+5}9p9RWMlK5k^(J^%ugk<{xXR!Bf0q7S+dMc@<;gSvMrI-g~Zm^^?%j@u=(F)!yU{qq7LCEHgCar-w{ppm4^_26un}Vh-B*%LKP?+A)@1ks9F&b; z%%Do$n?c{@tMljy^14eWcg;Xow7PUX#U0~_lsR6ybfR~a<#_`>A6>wgH#+WO_RvuN zq2Z68z3){9WBm5RO0vX#O%s@my&R4!+i+UhB>ya-g6X>gvT*G?&rhZMs=WEsknK3W z7xM1c2eLI~rwF5}uU^`gM>DSvx59_`xh6POZER%s)>jy=B33$H1sjzJkh=OP<4S<} zEzm%nM6H^#nXZy1x?DO=F2$ExtRG45K1a-B`b z1BxK(qC<3C?NB_yyN+qLU3kg&#Hh%zWVBA9#2C%IRkJY2qOC;76o{+xG;y4mn_~X5afaY@|uJ**RLa=h-#nzXx`BGY}A+yTC<&%o>1nzvKUgJrWp$HXeJa<8bXx zaq*H}i`JHTTTv?m>KJoEf>|mJXVLP zEHE;gWkenlLuO|4tEl(AA-cf8A8=m;te`RTU@@Y-p~rd`Pea&?kCYzkuU}xd_`k`3 z&rCjN^!UJvbVPG2D4Rd+5R(f$)BwzJ)Fkr)Sj$A%LTkAT-;n~T9Y^&1jNlil3B3~Cpc6?&SRlVXcCc@5PX%)L9`S9I>} zC1wVNpZwY%^-WvG=2wmHKks}mtSVfwbb_BlN|}AbZanSm6yI%f;|IoDt82%>C*Igy zycjF;b-5Fp64V@5=d!Y2poR0<(Z&TVk0Vo1d=28)<%Uy}wv!XtjTjSHER)IzCTc zU_yFQYOeoQ!!i3B>b2Es;;k3GeJ*#x+7qQ@JZD}B=MOnFtf1GVl#OTv9hctdGLU#HBk zvl%(Ag>uIPu^yZ5s!lFCo7A#{HEP(v9X;j{eOtkM0fn1Sr=nBg=}pQkNz?#8>6G_{ z65^?Uhw@$-6fVb>zpFSOx+ZL_;vwJn-O|XWebe|Mh_3;U0K#=TK3M}f7sYh2w%_8f zp~3R~i!E1hHd`i+#gVQ(U#%{du)T94oZYp%tsh@+#H_!LjQ0e^kQz$G^pjqAGOyB( zw%t$OI^5&f?hQkRecvGE=lH&lcX-SnZ`6((>H4jDU zm6@tyRaAV?31cNqe_VL2qdLd=QI1#0G}L;GsK?JZ;4w|Re*2eTBrSvB*v-19(K5$<@?fT(gcFK3+ zwD`obU&0^ax(dRQcKhqcNh4LPQ@h}5uIjUlD+n7jMvIp%_Cy|dr+nI-y3FU3zVs8K;d8KOZufHLl1grh(}gCLq*hV$J-?!4h{d{-A|Li zRg-F>e(tjo(g^1rP~kr@>v?mq4D!%H$8nd9ot^bsLLX-&+3l5{Zrb*v!FGvrZle_R6)gHz6;auQM5PyG*Qq;phDt)CN|tir6No`3F1 zxO})4b}4jXIqkCC@Q2u~b~?qu9F|?tgZ1(l;}o$xDZvKb_=v|%>}Kx?GXBpo#tthV zT37K`D`r1k?mSzZ0i%u_&+I(fR^g|K}XXu1>!Su8oinC!UsTO6n{uf-mJC4wm!QMU1N% z>I_Ap>OP5I&V2?Ok4~=1GhVMZfs0}-D)$GLL8{)(2R#{r#yR=02Rg6+Y~2vwijfx| zkmpD3F4g%nArXyPiblFZK!yO=fmsICL>5f+hsq(=k(7VQsg7skM9k;7y6J>8wcgS8 zZtvhGW8;P-18E&>ZHo#+Z*#5PeJXkzjKG%)*;;oy^Pco$4~uh9Nt=H3v+D!OuM2Au zJ)bcRHkYmaZmZrn_`ZJwDrV9e)F1PUY=?oSV`KNaHC&5caDWW*^JIBT5gb02m`L=* z2Gd}rnA$4Gom+ST=gbxeG+)WZhl?Y0cNFD!Csm9OSI4V{L`Ad)2f*vwi;;)T`=El$ zAjf6jqklVavvA)~+-EACw(0O=wvq5k2T2ZQPZE2%M!w-XD;6~$6a?`18e}pE+QR_@q!DMh4PS*DI0lV=| zlnqvjXi$DLE<2D`#<^{2BgOlL{h>5xngCFX0`VT5+!wnq2(K*_AmlNjFgbo8ug4-& z;DkEZ1_T&@kb89%7UjkPWXeh~8n#E1N2sbSYq-2U8%tby{{8V7kd?9z9d}<*k|E{N zVy2brcb!y9qO0C_{!{5Ciu&aX_tiyKWjl?K=wbio4_O@;Rj*A2yl@4ZTEpw^;wA@l z)WBIIpR=>gEj-fy(P|H_n?lKv)as(a(71m)8RbGtoKx))JD|G-1Mh!^O?thCN)tP{ zGh5Q6OB7*FIJ1L0HUy<+i9db08u zaa%9t$B$j%@PQeN!?I`AHTla}aA5aHOZd8uNiFy5YmhZ-ub+g9l`la5$%ccS#hocs zyi>NJKYWzJ1A&yA0$HW8|6FayZrEG;-#}ETJ!GUEV@x|K0K}$fNvT2WegC$L+R$JedrTVJFkA*m0| zGJH!lB=LVm3J+xTM4d9N9P>a|qNG#A{eBPY`_oS9xc#cTn}*2sIb0qhZp6K2^>gw; zHUU5@N?5J5c6G(Q%Jc;ReET9;^`2KkDu1`7=Zmkr@%(u;&1d!dI$oV5XsZ%vy;p~8 zX4(Gefo1;nabT~4C4Uu}yPLwcD_o8^hbZWECy{ zRabJ1TlY|k6^$NZb>&7Bh3)Idts+eaN<^V7y8x~Kb`*e zsZv4+Dt43PLAnjE3xi50&ykUfZ9m5+8t;vvFP0KbzbnPt)Ui zyN9d4c#W6kYs*ym{AzM99uV$_T@%<1Arf$>D2Y?wQ4`0%q14qYf7in>TB_G_B)7`pf?lR{o#@8SJo{0wu!GgIZi>TM{tiET}N9BUPSvmQPiyoa9RjDI5 zNSTG$X$NLj?o*48fW&NI#R6kWi5(c=FVNjiGO4mOcFAcv=L>)Bdgk|RQ8fuz4X?zRj+SGXn>nn?^qkGd;BcL7pDju|ZFkf3({5U1X2`g6ZkM#{J?AIUJ}@4q`C6F& zujH!`SrtVx@_@`I>2eE96U9Uzll9cl^q3KEiMT!*Za2s6^xnKrVIoQzZp{zVdLg4r zL;Msg>QSKJ03nB6m0pyOX}J(dzpHWXU|W(BRhDjd`Aheq(WKE;Ij-Mbn)KcDdJIX7 zZu}*pwmeQQB%`<9u%3lDYfxw`niKu^VzosIXxLH&EK^1UG6#yfo73x($85F#1lZlp zJCTuZAk6eYsACOkUnKT@(sd>mh>NcobaP7cx&y^9R2sSgxKawXr`JV}?diUJ>#^3h zHj5I{LuR%$=XMCNqhY<~!Jo+sOq6Lk8HwvK64w!Z-V zfrGYIKWrIuKLLu@OMB;opG3&fSOKmyokudidUaHRsW&-ui}x+h$~o=9)G|lRT3;|! zI>Q)9yVHI4wUN4p8WnZe<~OxL;0t8e@dez6>x^b9uDJJBx7hr^Ye(XNFzWKS>rzJE zxPoRKXS~K!zTAUW?iP$|_`ep2n!J2l`g)!k=-9Mqu{?cuQ0|wE{Q9O|o&%k{@h6acIDLI&ddm4oLyR zXr;`SJfX!$E?2Cvszrb>?^gKyIFr?|oGmSz1YB`H+1LWxjmO=(jvUwmpHP4+)6Jor z!W9x;Bue$5wC!M7-Yj$->iC>u+K{KxV0%!^@IUaAR*rm;Iov(8EKJfn>y8HcTjJUU z`<##>_1w2X(q}gMl~KQKXXUya#fj~5j{SHAP&x1{gv5sBD#1=>=YXEQo^aoLQGBJ! zLPVq`^cPvIXYnvIbPh%rEag1*?Vz0t{nU7bzbaJQ5R-X@bq%uk=FHu0b+u?uKtf>=wbNLzeImL+UzV3>?(itolw9L~!(e*tV3J>9@@HC# z7UO-jUX6_WvXJ)}cr3@&-lZdTpZKXnL9&#Zxp#MB+_^8a7+0Jd8T33Yv9S`?)yICs z%iShM5shc*Wa;y6w7z_EG+a6B=ckq;C{f(NCN6oh27QXf&}SKJ zO(eD#HT2*w1Qfy2Pb(Fb-BG!CBZPiK>QqX!?9>1Wnny@>+=uSDLn86n4jNdhx zZOf~5!iK(>`b|n|oC7s?m6;B-RBq>(6u)X9k&bs(?~ptl=R$F)d)wr{A{^uTcmlgi ztCT6G*K_4X_q#5#-MbzG1sy*4eag&3qosY!^LQ#QxIZVmgK5xH#c|#<&EOIy`g(Z8 zh}uB2S~{JrZ_+L0QzYTtzxUZbcWsBY(duyC?C!E01)6HD46Jd_iyCuW&U0Y55ZLM= za>YCnZs&gh&rZCTH&Lc%5H)_}{cN;aQ0C{>*rk&+)=|z%DJPxD^I5T0^x%a^0!c}* zVFSWuLI!OU)1fXe4S(lEUp9|J&)7<=^_LQNxQkvy(O6uK(Pf|??4|>Ypw)fmZ--FX z-+V};7&lZ89~h*SvnVdF%cfrOsNhw8oeA&;PzxT-8OL}M?6OL*ARO3q_6$G03x1x@ z@;!H6N8)Dj-p-ZEYuYsxrWHtBuVe8bp%>-Wfq*q5~PYLa>S zU7ET*D^ZO%NPTeqkA1jPjy?{~?d(}$D1e0nnI{S?CS@Y_KnbiskNMPBY^$4=nI@`6`sMg=1V0TU#NEo6)TC%q=Hb=vAm zeBoj|6YI9!;0k88Zsr##^B|c<7dmE2fx@@A{?$s@CD8Pp7_GF1#)4n%Jb&>r*JU2D z^P?KQo)bz54eh0ZZGwHS6G`&7jjCmYE!VxY1zO8C*P z<;Yl>FB16G>m>n@7Ig}H?_Z?b^=h7di6G1OFfOk&f1#v+96fqMfhiM+yNZ<-sQrSy z|JC6`Pro1Ts1%fOg0=cV9C=JCL|_u)E1aFbr+hfk^FO%AHn--OQtLiMB1v)!cnchU z+ggvN#+6D-nmHg-ynh_=ZA{`{YBL(>dQQ|j&?)nP;F2S%=xLr6Key|%mC$r3e*v?{ zAmy9@YLK2!nFxngLq7>m)&O+j-3glfvNDqppl)o@ZT+TL9?PLoCKLBUx zNEG!AAfX17Y(m+O))PehdeVRP)o=W6JHUppLl0+;;^ZmI%O!cS44e^n z(PM&v+TNB&p6F}%EN{CD)ItWsNiyQlB;h;PZTcd+nM01v#=fBHsx0jaR@`Pt&EB+5(Li6TN&s#006l0CVs>O4t!OPMO8ut+@;ZHV(Vj- z7Bi|a!^r{eZ)jGKsRPRB^d~-OfX2*>22E_Kl++ZqOt%-gi>1?&2>rtO_>wCzIJlj z(f-!k%N~3oL0=nh`U0Astc6O-lCPGj`Ui;5mDd+` zDN+S=O|2LFLQ*A(Ch@j{KM!Z@{IL&RHVYL=sdJq4p4Z-8f$#T!;)RCd;C9xfv`wI- zH~Ry;)(~WAR7C}nuJR+QIT1!Z#Z{OKhM#nS20on}Knhq7H_453#1C`tuy&;hnd3gQ z8Rm4-F0^Yc4g%(8mzfuTRWy55xXn&Ibl~73S7HADv%HS-0uZ!8UA8Cgg zFZNgTZpi)4bvhbzwS)Rje{M6vws~WT%SW(?NXOwV0pZJ|B7SWZ*nI12x|huQY($mY6EhV;Y1X zeJusc_<4;TD&Ga^|AH6okB(A~vRwzAe5a}(uHR#hZT{AKFX?s7cK+5AYty<9aM^Go7VedCOeq7+P)prHtq6af%ihAzPVqQ~TTwvVd^ zJFcfP%IOGkAwU5SWXK^7tYw*_4)53}#d>d^4gfWKA$lRE^k5*;d*aA+p|qt@c*Iw2 zU_Gwxecpb-RR(iTsR1P>&ur8{uEqLa{%D#tI{h@!f{no5wzBu&h6V`LlN|w9N6--kZdLRVPw`KZDIhqY=r5-$ zXAGbPX2y~GKh!W=#{aE`37zZP8A{!5ju_r96nqSFaDJnEM)Ut)Rm}%R)pw-beYGC@ z9oBq;cirO)Ei~F`V*b-Y5KnqCj(}tgPzl7+4Jo9ENNFsj;ABz581pLZ`sk!L~WFSLP#gwiL-xzr3 zPIXwvl*ImFdAFapCNcjA+XYe~^@&q_h6Yh5#thHutsRyCUVNLk>?16!ZSlE~|F^$~ z4sszFXQKkUo=#TC5zY>=rhI*&TV0PL>4K~;sk0ydOI-ch@Q=6(Z=reSmG*B%HKz@g zLRHua{@@|r&FB=hy1Wve0x}BR0|$cgq57%C`y^Jw@N+|7+E`g<63$K%tJSBGeB9I=ty}N@Zl)MLj7IE$Mg{X_+7sAg> zQFH_C?cyye(&m-03ZGH`bgjz}W(jbegO_yu_n+$*99Th!XZAi#f0J?avL3~^52Vt@ z)M90TPz+@13)uURN?byjAx9OsWkq}Arcorq@ta!4r!$aXuv=;|U28k?Bu8m=*jtDu zfbSWQGfdsD$ip2{^c*QtJKTnd34qzP@?V@p>F^-uK_i#{B)kbSFIUTMn`CDcf_A|E zDQhGy`zBx!tI;h!;uoU)NOZoN7H0V+l2MOX|LNBmh++V^TOjQCeB^t0HA5CaC8T?G z>%S;O=!xlMXY_zy0f(Y29*fB4Fy|`)K^kDM0)UT+DgZr*XIDN|3y$j}lV#v!0QGf{ z5HZ_+G(#4jrR$3Ey6RUb$lsR zWA0Y)o@k8n;w^_Ft+srjKY#L#ZG_k^16PNPlESphF|-mL=K;SN6UtnoFVDcx{^YU@ z*h&czdkRy_li7j1Zp-%8(0 z(ttSklXmPy$N&K3c*LsBEnczA zBacX+p5rPcGPr#u*7-dLyC_K-$KXdFuT41%s!Ka3F#A}=y!|u0e+1Hy^n4Jw7w@IY zWDF6A#Bvz}Hc-SeCe7wLkQJr>pkC?Va(MdO4II>#UI(`5LEMJZ42(|=lO0TZE;I>DF~+i;^gP6(V2GY6ax zWHAn;In>5hF@uy>Vr`L-ys~gfd$J}0oMPCvf(GewjOD5@Qd7}+j4;L^W?N)ncxKogqC!AOzR_U z`o-*81uJl;#pyqvJO-a!b$bjHt?LTlIig443Rp;VO3__sjtZGkBg}ax7k1aSsCV-D? zdegWhw+!AuNzVQVJQs@!*hKAr@-q{huE6hLtJG-ZH%-$x;?iE!+{`do;A7I6=Zo~6 zham}mVCUS?nSdfCZF*}4b|0|k_#xQ74tD50yDp9Ef6$UD;4#+zYo(F+&6q|@;C8ft z#ldfI$cQ+6Yzu7sz@R)Q=C^NvW1{A@Bc)5h_6Ghj*P%jIJk8#Dh@qB#ZQj9rJj*lE z*r$Uz^Q~DQqb!!(ZyMfi>rB?;7PlGRR74$z$LWI&-$%=mlVIEU)(>(buDhMGJwKn) z#qGtZ0KUS#qPs!F4D#ilU0fx@E=@N?^Fgig2{ovDjo|+edv6^VW!tukqCP4Hhyh58 zfG8>545HE{DLEqDl0%O{C`gwyNaxTEqaa<9Lxa*ebi?dx2A}VJ*ZS7}e(yhf?eE=t z-9OZExaXcL&p6KWJT8Tp9q@s{{;`_m+>(1HsaL@VfG)w0k4pJL4W{ZIC;(b8%nFgR zq~+t1wZapFCFfifQObaU>Bj^0x84f7-(=Z+h(Iw1tpDx((sOv@AETERX|c-#E9}Kh zTn*fYgJ?YOd98bwLrGk6uyGdMADf{s;o%oTv%rc0bN}`!qf&C8YX6|#R!%Ia-~dEE zO`v`(UmPzfwhHVFZ4hqzgO0yK`4MIkK%MO41~w5DsiMjQuxFyW-8mpalJR(A zMS`IXa1dtj=otx>W&qI7G99gDciT&?@V%CWuc2dlRQ#h;ZB<)?Rm|R*p>2Pmr`%d^2D2_8Tokn?>tkju zctQ5XAqbToPm!kB97-(vX6!B7f4dQ^CL4)3+S~)FHQw_Hb(jfHbj_P}M^xr9s2WTD z@Y}3y-@xY>q*$}Ggro!MD8#9&)^%1#u{~$%-VS<%jlt%QIvs8QQVd!^?W#N%-~W~q z3;c2(hN^kX{}~QkU0<%*@J2P+HhyNzdHe%7%fD$1fkEPhLcJ847nd0=n}Kx)Fo9FI5hDLnpW%z@|!sXxs$7f`vSk zT_?pO#!|3J4S*mo!~H~-wVY*izv&x==7&M|YWo1l=YF6@H8iFga>zr|5!PX`7r( z^$?l&DuaqjkT6Vq;?_+yfucyS#qKsS6SI5-A`WARpNy$y|CIKKBf7zNTU6g1Eg9mDc(jnvF(IhDN3hItqP{b@RKgF+9VO%L@hK>C2JMcGG&tLs!1`|1P-&#BDpQv=n=um1E z2@vzH{k#G#!ICJY%`~c{mlx9G1-lahovc#6vZSFxqs`VWhx8a(PY;*E{}pJ~U~KuQ z8)1P@yPcDjSmcNT+;msEch`txr`zE%{PYP?pkO7+BptC56}x^N6it}a)``>|{v4{k z6_uFCojtp>yYl%ch=RMOW^N`x$Us}`5-(*iR-;m?SMymB(54d4q|F(ki*r!H0IWRh zWB{v4N=Bwyp?bOyu;h4~B$7)Dt(PwoZ;`42%H)Q?_(z(WV5VjgonBA+cLf@zR?}f4 zuXtjY9oIl7WSD-pgW{CHv`4gF>ESIPgrVqRAZh7nC;hx$%t*;-yd|Eea8@^Z5ksln z|1NRu7$q~UQhA5eklms|c#Bo2%W^G#mdZ{{5Gd$3hX2cC$We&3S>qZXFDloS)l12F zeWPxv+}a=cc1c4+9a&}9Os5!l+!ZBP5&9R_sBq9>bW$VJ)4L62Yh4_$(~w?E7`Vym zN7YfWsO2zeXWsCt)@|yw{;}*kUQX zDJ4f4jfk4@fQ{Lh?OM@v_41di;{b8o_OMte_yf|t`A;AnIw@Yk7_0!^mY4YYAO~RU z=^a)#q6)@!OA= zXVu+KkKcJO{1jc6I%__d`AZX}8zy8VzESR}Q|zic`!%iBM9v9{D`>%spZaN%{imE| zWe27$&#KNv5qrDtOPr6$R#a*ok0na);y;ChQSTr(pf&0Ab;pRF3o*(ghcH8eM*M6Sz}j@U35JW-t4Wvm2yLF5d7btdIDoonWW!e};+me>SS534q`LDiV6%yU zZEolG&{?4J5qee1NxkD(D{+C+itLn`vL;8yV<~~?z7_gTs?kH`s7U@C(;*?u%K^T< z!Du>APbW8|e$aZb{JO7kzMn6?*mD%%<0F`L5Rzzc8faqQ`;mESh4A{Hz{1$kr`EMM zYr(g?f6dkXclzavw!hOaM|kKwENc)oVHp{7rRWsTrlfPF7N}-r%0GjZ)u5}!==TqxCJ3oll6nU8#8kxa#h?5 z2RI5UHG_IZ{C`=BQt?lvD6Jj{N3brFnA<6Ngc;~kD(mXZH;Rrm8CM1joeu_x#6gPT zsApNzreB-4|5JPqLQ$%d@BKEcKqBmRIjU3%VWn++t^wD8&BGSzqd~(>pro|dn9gFt z@#`0BLJ|k;O1a&-<95?n)2q>rNj(h!GL|2%VlEq_{3OgNxG}bgo!FJn84}w$3lY2C zCa5M}!f~}&?a2zolCZz9`zEPcYB8ec$m>ycg09l;p5&%tJXs=9opnP~+eOfrX?iSt4ZtlR{jO&sdMWD8Y&=HG%+lqx*>f#> zZr(5%Eqe#?#e^3@pk71Pd2ztQoA(wS{_DSLA>=@``Md02n|84&FV50{3O zPfx}1HEIs$__Y9+(8_0^t)_k9uBTNh6GJz(>l~Xtf+bQr94f8o0Wvcg=nHl6usMB|I-$^v^C)_V$*HhsvEV28}adG)t5zN>K zgmvattwEI2rLi}_O-p4l3&f49c2==I(%UsQ`~HIOn2lt4*4eFVuYg^;E)rwRnS#6X)0B5h8e<)Xe;_ zq0Zy2?)*Y<4U( z2iW&YMFlPkyGW=w97b*etW?|O92O{*S&Up|z|N!80}plqKKr3yE4T;FQXZ+=uJb;~ z=(zjlKxmXFMLO3_rfb8%jf!jil~GYMmD>u&(+X8{Fwm}nii+ex^{3Cx83$5oy&3uB zJs(dktq5Ze1qc>M8qN>k8esniunLx4;`x9_*!mG@|5rB|Y$Ofo)-=T4sjZ@fD!@$q z0ZbjxHFTu{uHP+2f9GH*g$fKHmfW{ad8!uB)0Z7nDCmL1c<aB(Z;PQPO-W*$_V)$3QP$LJSYEhlo$?!AuHfm3&o{_W(-3sj2szOVDSs z0v^6%T@4ntmQ%zvM!k|U2N1M+qUKl|6O+T)J4k8>5z6fb6<#ZoeY4>KLWli>&5`AE zyRK%)bKigeYcCql2Efzs%=PFQ~%q`=zlBS2i=qJ7xYsW&q*o;y3@eCc8q7D5h) ze?A8a$v_@SS-7Q?yr%GDILNnKbH0Z2?iK*z3DR#>tBtcWRIqvq4@lZ~f^1<2m?-#yJKc2Ezz55^LY z|LnQZzmS3wAF)pbU=Dcq&_T_N4(DN0vI3+6uj;C{ct^z#7@gk z)>2#bh%$@zj$MguFa`^SxV9Krp?!KuF1M@V9)Ppw_ME+<+$n-OM7(>gR4nqJ&{l5g_p!rs)})^}2M2&2;E|W1`}{hR1&o`wZz!gUXKp zGsD9%)!_xl;m85OuT8zB7ti5IX%&8+d1c>^R=Ez|(*mI?(7V!GIUsWy+=4AG#1nbH z0)WQ1PxT2w)4}x?XQpCb;btBC0taqkU*Ot+eIbRN66_U>l{nbz-Z|{Uz;!kT;KhH( z2lc9+qf%GX*c%lH5-BnLRb|CM>-7SV*b4nhUGA!UEYf*|>%`mC-~Kj&rtMu+=#xcE zoq7>Z-RwXB0$H`Oa~Z0&j_UTzf*D6L;f4U4Nx6`;jNab?g_GCp?CK5@n7a}lKVoH= z1!x^|cOJs#Qo9d0+8x_VQ91WFWmhS<= z_=;tp?$l@Z0S61xy!}Bbm>)2%fXVIWO@M?v@OqqC&@FKLLHU>{0NJD)Bmkq@5ow2p zEbiI#1OVwM#8Dun7IdFh2R>Q>UUPwUfYv{n`|)^Xg{ZzMUTAl+)NDSGs2&vZ0YY8u zCH*u>TvCq4))HSIbYGE8d&KStB(q1=>47>GMAnNxUlPErCUzO*n*((3x-KAX!BlLa zCf&P<=jFb{MaFCbrdrI_JVXLe90UyV^h>7nNYVFdYUlvmYCr*pTd}MgzH7b;{$jl5 z3S4Qbo1dOppA5lk=I3Z4SC}xMgwNlmTdunT0Dt+MFEHi;U>87#y0eb6%`Y&?{u|QVNIuti;l`0`w|3qXzm9^y-=-NdkbY zsF|_<3TrwSP2q!ddt1N&?r}0svwv6(`Dr&Dbm{^J49EpHmvG0^y-n3ub`|Uk%x-Ij zdN6psY3(O7zkqGe+f-Tr>47Og953FerXZKtrg`%#e31z^Y2Z_UX?Ou-*1xPZ+5%LV zMjhXtEM}R>P7ba2!B4hYjbs7cD;$udE@owAD6{vF(RigDh|{P~1NWVr@RB(e@vXXj~~FT-xIt=Iz&epDi?2e73vowb{u4vZF{ zOFsle;_&tDVk}(_xTL#Nw{b82UAMUyR%C+fo+hw^Ygg-h&DC?wva{sTlZ==dvI}f* z6LV_??ZENFBV_o*(YJFjf`g9cLmg-JdA&e3tilABnBB;K1s<-hMNQ-51U3+KUjG_M zN73DfjwdCoTZ^G{5~yOa$BKLGr)7*MX@F@?4KW@fwQjkq8Xg)FBIH>A(Z-!EG*tPl zNV;lg78};WeS$uA?(neffZdnq5P1|z^yx>r2_a4IkyjS27bqQem~6KNwI8@UrFb0? zR;Kx&AZ|t>zMwmQ%rQF)tP zQUk7%g`0Z3P&u$Z@RMyd8c(aqnVFvAhxGS{=XMar37}^df%oG>lpU2Xd_F2=8Eepi zE7R~ne)|C5s$_-%GU+=l7Q_7K~ARP|doBs?4x>p?L{?WOWaiDimx5_aa( zIe9D+J1x;X9~LxhH!1>bBG(-PXrN7&X07$OMqN2T)RqU46RMg6I@%iLr#qd~8~i0%S*IOoPjXQvv1 z1>tIRaIQ<(z2zgqgZJ6}co%^x;XhVEYZjVe-k6Z0Xcci|AopdTs&>>w`D08tZlks@ zDiS_$8!K!+MWdWLW1+R~jvi*3_VR%5-FmJi`C>-=OXrEjknQcRT1wr-GlS!8Y6O?9 zj)H%y0-^DrGQ7Q!JHP=MBGd_7Yr&r3fW7`dXv6#w5dZIY%`E%v)9sle2Dtm~V$@)5 zSxS{T;MQBDlkfm)MvuX{3jfsPVp64IR!qLZ_w}VbkqXj^*`4r$_ zacyuRb^J}NfpB|T+UMx}^gTAfH&u8Z{MGtCp^1{o;Td2LB@$P$-vTBXAjS7|+AO$< z<Grqycw@4MuKX^2=Fh?etPrQVzRj8XEJ{NvNv*x zbw?}YI{R~xx;#8gS8*K$;rwJeKtd;!ne^DJ0GyBH;bb*&Fc%33@Q86kA!Y|m+b7sc zdVGDrZp+QXS3Y)nWRh@|_VWVdm6%@LAJbI;YiVPIF`*ODn(chGww(yiqU{|5z+4Uv zYr&qGgl#@iJzVe2r@w7;?7S}lB9rB2%rqVH^Z;=4iB?*gwddD@?qhh0i* zE7Oc32aw6rVQ}Min-O4dv*YB-Af!0>&ea9jKdwjohw=4B3d;j~@PS4Rq;G(Wp6`#M z|Mz#1{@aKbtSb$!|2=MGa5`?S;z_9kc+I97og71K5o2x+BFL=}m!`Mvlh|`9_ghR$ zoUCjc{n<5GHD4f0OKj)!!#nh%>3m4yTl7r5PQbbW!eRchspv^O4}qMr1#C#v0db-33X(Ck&4y0l)p z-E5}Wxyr2~et(9QlrA8wlToA9|38W3;`|~4o-WO!%?}I3Erxm%@ zwl+#pdQ8dfBRvNvdb)8cX&k}>zCTpn@_kszam@)VZFYx{_E~I9jK%V9khYcvv?y;o z0qIM0N$(^flJ3*$qxC`lSMm{R1Bm=wU>XNXCQu&4v#0H`I{XQ1fTy*lS0>YX5lIVu zgH3Im-tRf|3GNHD=T$@9!60K}vMn}E;+33F$&Oa!|0P>_>+QMg;~)PX->|cw8t)pC zS!!aNSWAws?%*I? zkIYK=@jn^U|J>gPyj$)DF3(jQJ4LIVY{md(6kt?nB3lHdwmU^?ng$Zsn5lAc@yL!0 z=OqGsn>2nLQzdrhL*?f(e?1~2qxH&eY~LML+u6*a^#XQo`K&OmTf4}Cc6B2VQ~nP(La~P!?U$OWdKQdPMwW7GO)8Z- zYi{t_`0?_Y(D1n?KKfk_2sc7y^YuI$qNcTXQyGEmh(?(VMM0w1*TJY{MIkKReU(&6C8Ht%@IXFQGFf-*x;_j3pe2|vG1x&UA8*4O?_`sSV zNHhZ%4*ZJ-gpPG7SZ;IMmdAg;|9FNp{vLV2T4ughh4m_yzL=Z*j#8wAGzY z*O?x=AC*Mk@Yt{bw9Hl`g(rN9$FilkuL;)|T8J|^@zxzE(?>d&wLdWXQ7hOMiF4+MNA_Aws z?I(mO#B)0g0j|Hej*g#@RGaGsuT7r!bYBWBBSd?w__sn*Bb3eFmzV^U|)(P?<-=4bPrMTuU24OOwF4Fk(Pf zTuG6@{H8+%EI{sYiA--#2O^Ik)^hUFng!OYakH{&H?b?}&7FvjI>^XaoQP*f0sB}m zN9IEYIy%^r*Uia&9eRc|kUQHJ73r`>MtEZ9V2i~T@{_@NnFvtm4?+J9m!hxp7)vtf ziU(f(`nP)00bKazO$lI}#9(&UgxUNyZ%pl9Xz`%A`X2x4gtPzNyFHKC7^CRuG);Ag zd>pA)J|4OtU3at+UA~w& z8P{!%$D_MC!elkPiFhOlZZ&r4lk*R7INC6HIWNEq&(ABJ{!%)5q4e&7byL&K%={JF zDqica+*y(ae6&9zTHXwvMVRp3nlLcluX@O?h_Bz99oC)wrhzzE5`~ zpInToh>CmYRlvc+#mB82GxOSn(d(m(0vij{>*r_N5YPF6-SlnH43-Xu&k^1FOQEI~)uk{^cI_h_@;}4uMldtR2 zK15%ADm7cBc{=30iHhp$zpljj#o*eZ@j(rbU}kRq8tt}ucoAk{d8rKwdKFDZuHOHV z`ou@1dD+69eK(r4W2^RQ-Q$le_pd!8`N&d#)r&afBJOo=26J0Cskmu`6QPPM8g2ij zsK#Aihr|9M=1t~lh4{soU=8`rt5;% z{xNY{;AVI7aYORq^i^Wh`g)L07ZS6PI+NPp!6^Qbd_=J6@Nkx<1^pwLG$~TgK7;+} zx|rd)XCz`T&)qA+yMIk7YVy-la!g*}r$VByQ$5q6VOOakH^ogC7fdP0Rr@L1Ae7mV zt4u$={PW_8?$5SPOAc$wbf*PL$=wm>M`fp_=yISS5%WplL?-MtHA@e_M^E;Ps@8i$sXEzXH|c1+QV9p zRQ$TdXu?@g6f{M%XAl%a**x-+8ivc=5%P_`O|?fq6+=}2)vXz~q8jWAR0^*qMF(9b zK=lmPQQE?q2IYGm-LTu*>h|)H8(a6#fAM0}m8~|0$K*^uZT>==z3 zn`(X-YU+75m1@86S8wz4`%GRPQ)cy|X6qyorbDxR7;k3Jlkh<+enYr7lsG5iocZt^ z++N3Ul+h5}HP^T7X#2Av z);lzBELka)cYUapEViCx?da+$E#pS_mksX!5=?&(!Qs1k!*}U3^BGu)okaTe>t_}d z_CHtd{L?R#{wO=}ez4e+Gp7%{7CV3O-~R*R1nTG3dt(GF^wxJ`8?_xA9rJQo#$$M? z1>8Ss_ceu{5L;U2?yU?l!Q-JyZ-ls4wAMQ6vEavC!hP8Z%v(I>#EzXTbHYrI)`5pE(>VAu$*H0+BuBvKPq8V`a(Y=pqs5 zMkjLeYp`>9NkAxT$G2T0e`j0pk=M(|a3=)yc-w-D(01UwE!qNQlb)JRLQJfwsl^Mk z7(pm@J6UN(%=0Um>o7a^4G!{haO6X~E2AXKzh5|9T3EP)X~F(&reQ^P$|MZf>(o|QV%*&ga&q(e(kc){44zAGe^h=i~kBB3uwzX+5qbj%d6?HjI zm3k2S*m}Fad@>hMB#J($<)UK}G#<}R>l}}{rE{++o4-i)I&;!!{$Dip+A#zJo^y6GGYH}D{@j54OLRASgGcjsPpuT^z!-ve!}W9_EyJ? zjf`5I&V{5|8{SkGvlXbU^gNxPK?J!R2(Omdcs}{Q#6vM-;!C{7_+VU7PSAE!bv|$v zf?0t}`1nMW$31`kJctHTN#Z`OwH^>Ani0jWX_|y!Pu#x8U^JKOa^=b=zb-SgMN+nl zIu&;wzg|cR5_JyByMEul<$Y>%P+~_y`D2k2+MM@O=RSa-^O5i#q-vFG6&Qsw^PU>DYvrT*Fwcpq+Lep@Yezev zd#@idDeP>Bv zabe-dj~}ko5SN98jV)4k_~zpLvO4PkxTg4%WXg|^kN3Tfs&^^*=NvZ=JGDnKLpxnF zd@~-N`x{YSUe82CYGD~Q!zmUfDk^;y`}tlM6`C@3g+@uK$= zjSoCuTA5RP5Za~NtI~*4t3KroinNT%TMn!HS7=yS^AIE*@!T6~mybrOc+6jFec4;- zaqNUmYH4W|ybv?YqR{jrPMQI?(*9UazA##5tw|pi&+n3Lm?&~|qqY0MbMsdTyJ+m} zVD6{FkdSwEo(-4sX*|{^G_K8=TUhw|j7J!avw;&OT?hx31Ts*Vi`EmX@#;{NzN?=?&@>-KfTA?Q>jcq1duou#&qnQo4F+dzHs?B~uyM zz{tP=C8$fzOB-lUsgBMv6Pp?^V03@*nMS2XQBQ9oFy76JsMgJihS%^_YqZ1a*hWN8 z18j8yi3y(1nU>K0605P)w$fYK(Acg)4LUYAx467aOoF|x(vtI7D%Hf6l{qzBVhefE zqZW(+D_Jx16r4(}c{n~96csCKU-s2wX@<{>$WJ?int#P*svyH%`-mEnG)VOB$@fI| z!v~Cv<@@qFMx4WsFvZO^?rUQNKrV6faaj!4TOAL*AkfuS{jAcM`uVe*+H0l8n>p9N zKL=kDL8ZxKzCZ*P+LI?2-xh&wo0^(>m6mdONMlXl{n!}uJCqmEg~MIr(hn{7bJve$ zWMwgiU8FC1IQ$}8G8e!PkTe>sP)ieS*({8{BEsWx;KNf*dgf4c?KM*NFw7TSh12z9 zu!)#O&h1o`qt`?(`x$Z`Knz|_^V%G+@`~zrVMJMGS>Bc-e2 zirj_Bn8-CCUALpR>`l5?M%$l8@T)Oj8lWm%x(P%wKJ!$g#FLBPkepn__2}oPKy=mD z*I&xt(JnQAk=5YsJITz%1jrP5EPl_fR_}caS@`mWoX5PPs-LDGsfZL6aT&URTf5#i zkqAG(Kn@;`XbvlxC5dAIWps@o{ImwJp2f_?HyTKb5P zTEKaKXft9thSz%L6;zv3VZiT+qh#*+zeq!#8>(Ry0B$8a53dfD!kY7I8?~9<;0_+BxNVjSvjn86Ft-n=XOrK~RF}c%T@9eigs#E(5f^KfrM*gIO znaL@ppFaET3cASPbHKYWuDc6NHYIs#eP!yM3RGhe5|%^X90!UJv-9r*t`nHl@G;Cd zy#(T>UE_VY!qKJ43I@2VNqK(!46BEm+x}tvhKR@9heh3d>c9q|EpG@74UKSZqlp=>OsOvHnY$77EHwKhFT$WLBGw2wLArKW2E9Y>&=@X8yi$zbv z=N;l!bx%x#z9PH!sMaK_=FI-&`D9G=)^kqHjKH-(hO53Fu5OgO+gIysWwUAx3q$~gU=KqJF9%N2zxvG+qc7jm0RdG`CZeVlm%Lp>DWGJ zw9U?VaO1AOpP$Z>5W9Ab*y*F@!oUVbt_%}nql383$!ZLdi6_qJnS1?u+MB2mu95jO zGkH1f?YA#q_|u#+ssd$LfkF}g7@GVdyUKCtVd+cFmn{Z)&ISq!fr%K+jc(mfGeG{h zI-y;s3g&^N@awDF+^XDkH6+6th9jdPs zZuQ|$iY2!$mD1)s5i9Dr zzbC5iDvNaaw0}$7(9k$i1atiU{GUN6tj8*yY*B3IwUU$0C*7eM8>QUXuDVSf5*B9g z>XnjoO26arQO5nU(r~0WbDb*0CG>kVln5Ah*<1%fKl^&ABjdtjBJ&rNcn^a55)BPi zr{1@sWwz<*QqV~+7&1s&4fQ56^x!+hw9ovFcb;B&`XX6lT{iph<`uB|y@>G`Msm;NF+B3j*;weAZlh=%EQ<^XC?&f8Nb`Ur z)$ejxOms|vfEd=0?^+TR#HX!a^NEVPL++%3|DxQQsx`U=UgFHUZ~UB9?X|Mqy5pxx z7nh?pXo=Z)ef^jYexU&>A;WQ^(gY3~oRFYb&PJ8vOkgrUgHK!HrEVrk98GkeogGn7 z5Ud*l^wUQcVXIit&iqx#Q9QruF(i%gIV#(%*m$;lch_m2T~DvT^iwaQe;H+eOI)Yg zv$$=*yKN-J0ji;9y5CQnI8iM8+e2E@KjJaP$` z=FxJ-1_rCkj!KKnwE$BZEw{WQ{-ddJvc^%HIsy?$1&-kmn1wI26o%Rhxt;!AtG(<^ z1axX>;=O#CK*C+|kBnqw8t?NT$qUZH)baI!fwq_te-HHe{&Jept0eZ*%1K@COf!Sx z-D(EH-BNa)tNXweIygA+lLof-U~li$N-8kZ8a(LyB}w`U?400*1HfV(0j%KCUh72ub~DR05Z zDRoxzSdFEtU8DW+)h5dA=+Ma6Xm)01KT!Z;^BpDr(T6}Mg1SVB33x>(N85MA-D_v6 zq2-w9;NX`^N?(re&DN*AJ%F-BVyK~xoY8Wp6bI)#+#+0;FEk4hcE<3Mh*JvvqO)-^ z>iD?3X5~ecGia?-<1pK)U0z&N3?wbuOVS;=)_f7l%F4uKK9>1Cy!?Us3Ay#!VE@E~ z0{+H8IWwV~tB90zYiFnbXFt6{*V=MKBGGFxJUKlZEqpUd7c@r;jOBBUNt#4mK89f5kD&CHnH8>YDMFyztFv#D+Rr^{#F#H6oj$$1V{+kLi!vWpmDal0 ziyqAMbpw;TPO~c8xjNn|&pQ#G{d+tPa`;B;ut5^2R`~cS_Bg0gh;CO|ryc#`hr3E=LL5A|AbaCCV7!*}_kUF-%9W zOKH75JqLCZ8O4yJ&5dbBW+di`nRzvDo*o6nCq5Xr?OJx0S8WeFDpfMN69hffCpaB9 zDQ>sb@>u2OsU!C1riZuK_3Abc>TO8b;}|DxOzI#!TwGiIi^afR8>ush>V=kCPUO~4 zzB=2Eu5-XvJ2!9@pk|m}0NL>H*k;YI^whQ|ojWn_|C#mv$@pmL!EAFWd}Yj@VI6eY zf$nLF1c9zE0ol(yS;jJaoN0cM!Xih0Sz~n={<9n7?(Y6xuEl(G*lD@fW$epM04p$c z{Sbq~TYq1y%Qy~7T(1POI6lJ4z(5z7&^Pt&gq+Vhb8xeTCT4XoSDNbbQO3^VVvp5$ zQzBP!(Q$V?8n_hr$vHVk!z)2PH^`i~D`3Fhe3WIzsTmTHmZn|&;N)nR=>Y#{Hn8!R z2NMU;x@)8ot zc>Q$q2wit#I#?rn?wubOvZ|^I+J$idYZs_x-!VeB?e&o4x}c3GhC1NTkAeZvgy^`q99F-iZwAadr_P&zhx{h1hWH(;4^LQu*3&) zt|k(y4jol$va!`XYA1+DgwFD!qcVQstJ4)dk*bK5$NIhf0C*#?Q0KPlmD-WU6S#KI zB%~P6j%2-NK37KO^U37(DxiJi_t(`kt5JtF1NBg{4;?6lc60xrPCg)~Y z-o`zzKd&VHFcXj6|H`z4qQ{Xu?K6FSePd&FIm@w#Bn#glYDIZjXod6eHjFWdO5}a% zUk^Q)PhaP(cBvYRuCK0(eyN;-_4!c zzyvI|d;9x*zK*<%XHr^PsrfLf;nIzQ;=Ht+91A0(;ffa@G$aV1TjS;Nf)yIW^^}lb zbY6Ew9bSzNvichq-DXrQd?(g*9{gax)Zgo35|gd>)pcbq4vy0$|8Bj+V?EcyhDprW z$xc)vaUxS#tHME|$lR>`ytT8jae5_4XC<+oZVohGSW>PTZLzgU&47$mx#$7_nWlQz zYmBhHt1Vx5)LLv(58S0(wPgl=lP_}Aoq%+}czz}NYAjEo5#{>f(zX>U9k}0-RZlNO zFtLd~!ThRcv(3rN8t{1r7$gFD!D_DO$pKB6P00u(l`IC#z!(G#gC+23kn;aH^KXx)Xp0p){$_o;{-$brP3d zo`jlY3UMn*Yn5BFVFCr=zxY%h$XkwOKG{gzkJ#?qR0FtJ`hy2t>GF%cSswxeyMQOg z1F@8;NUV6XQ50Z-WIpO{pK|Qgz^BE0wJ8j-{ zdv7mI8}vsbBPCc;EkEDv_}KuPq=h1WNzn87d&$@OuMG)$M>nXWIacF;bD8~tA~f>I z$;i8$ZDfFH8a%2Wh4|}{D8sYojrg<=6z-x*?Im6OKX_PyF&wXURPT`dko->O_lhd4 zqb1pGmxl2ARq(^h%WK~MtyFs2Cr^vlrZ3yOH|&;;x7Vdh7ja(E>_mP*O;3s&>8KV@ zh6g`0_sUvY`#UP3lD=0Q?`_6pq@?=$`j!Vv!)nrOY;J%Nsmthge|$m(fb^>_#o1nu zu@debb_}??G-HeiTB#Deyh-^@M9^2jnACtM3k{yWq3J_-NJKm<{1<%AJw$FWE2a8( zUzvE~1oh?(8ktqZXXk3x!SoG#fgJY>MW1MQ(sTlwf zEKd(tU@{_Yx#(A{_e-Z@0R{lJ{v;dpgr3e|`(yH^HAmrw_{XLI^60g6ui zYtnhKF=XOHxW1VgEj%P7T5~sXhV;(EQs5#*PV6L_Ag%yxnLk%+cPBj&UQ2e1+?HMW zG$hR`HqxescVacMI^6s*x6;KKn8Mwkh;pL)!oMeer!l8|HGof4d91&!+nj&chJ388 zsTqE(4Xm}_lbN#Qe71hQH^R%T^)$MNrSX5k=G>i}0s$y_v}@%LTj(2|27#BPobNY4 z9`(pjIELW=z8h)@UTP9u*+pJprRhn5K<&qGIp4+cFL(ff(X~keyjj{GUbZP2btlG+ zfdgZCi-+i_jMVh;8pk8!mQ~rP3+$Kw(!Eks=Gw>y;*&Z3$?yDrFDb(ZCAo~W`mLv6PQDI_(DdI&-Z_z$keGauBBQ_>+FDbiSbMs}Yc+8Ja*2@of;(%C(tsZvDkoPx@w1K@A zw5DrZ!w16Q@<%tQPPP!Wo85;IBHT)}OZ|mW$d;pdnboR5Lh+9U)xQ{oKuCvJygyHS z-pqcq^kiOn^mbV2hqTw<+ebAvG@-JlB6-?mVZfFVxHbbOTIu4S7PW;HBt#bX%j1k2 z=Z7NeMesI^-A3>Bwl7d{x1m%yrs$r$;>bw#met2LG{`nRpE+xJZ=%-L8`++wVg{sa zrzh)AFCQTx8!*#J1$+?ZWqJIt!nwOZ=oeU&-bC>u)j2|K8@EL{qdS_KMo`O1z!jFk zSWOnaz`qFs1b{Hk1%Baz2kkRsb@f)w2#zaQqYGycsRe@pds`)tBS;ULjK`pZwM?#F zyT-*;{QCFr3m@{S1nuNJ<5#x#(z)N{Ar_{AOk4lvDJfhFvH4``nOKTu|GK7u|BEDeC)gK$jmAk!>NG0>4RhR4dCnSOid z3Qf7OycdxpB_(hXU8wh*vgR$#%#zhs5N%)n!mcCiu)Jkz(WGY)8;?_?x&>zP?JewD zdJRx=A*WT=VQVw9jg8NHRj}pUe$lOytmx?I428t;Is?=$RdslJ8-h|p13+qs$>Vhzq})MtCca)llnHrUT3dbi0^yLxr-h3fWL?@p+cVSmqqm1|WGD`xP|K*%>FsEH@YN!Yhq*Nk z{jD}~cOeq^FPiB8Lkp_Pciq=Nw=|b)l2Fj5AP1}i24EdHFM@z%%v}m#$?`cknxV=Z zQEXI>7X1%PN29hNR6h}WJ<}5a9i|p>H#9XhH87Z6QbtOmy4F^;EFN#yA0#FSzPI^cC;Rt1)@5Zn&5)exZZc^#%_46~FGfoFl zW7zVT{ws_qaJeuCgXUJHW+&MoMtOLU7!;%!d_~hrXcUO+APTPfA3t7>_slBFI9Lp( z27Wl{JKWx0D6^jKj@HyXo$uOPo$X1p_pL032)} za*HI%y32L$O3uoUXFybmYjH+@KFjjO`fN&R+y6u?*ie*J$w@P9_>Us)PnmaQMxE=AEZ^6mX-Rf)Rr zeobP`ZF|A?#fuQ{N=+a)SI0*IGNx~8y54J@vXn@b&G~LK5D)L0{X1-amY5odQGN0Y z6nQhWwF-l53(R(3&2r@uO_=Vn8R{`$d`bMxe;)fidFkOIS)u!hwd+e1=v=MhUn>Xx z($cL00s<~BF2G0cN=ZPvha{_Unpnx37<&t*0HthRRz2^NDkp{<6zyn`Tq>u57N z(lw~A2t@-(?bppcw#$D}?d@uAHKePvj)N4Z!d0+;u{oh2OHB<#>2uNpI=`KItgKi< z5=0>E?A@qt_%US!Q^NPSzIa>Nq=sgbys z(m#GUG~8-C{F~j%$|7RCGOi&aJ2P_uq#1nrzuG&~a46WmjqC17T7)~xjUuVUB*~Vo zDQ?_|$xdiwH@57;kU}C$)*^diY?JK!T|&08n}m^lUlPVT^ZuvjdEeuCzrCNIJ$q%}J`FnH-g8y3 z&?C4iBYC7tGxEb17H|Iu{SGuhz@7zhz@Eq}?*r|(r%b&?^)y;SOd`J3GWEWIe3Y+> zQVT0PYjtIR@{I=$EnCZKDak_Nrh)ul5o62oxZCk6-HA&&kI0lTRAqp~b_lci(%g8n zvT{-_zlV7h02ISs`ySHgfX2!wE_TUJaB{8_{fH~F*qJS0Es^yfE%4EfmLuI<&e+Ja zIodh+ctJfkI&S}#E5!;gJ}^dn{J_G^)m5?(J&-#d5E=GdwRKWJlSSdpjSng)6n;Bn zy{lmhR9s0>!76VqE(wYT-{+UufTUM{B>Jsl>i83P2@R`Xu@c+6V83<>cgR*&q=>ut zX@FY6ZoKJn3TlV|tu1{Ys^s#UR8zFqXrYK*SRqD^AFE^gePMaA z!o2l-lTnoh^1uh^tO>X9ut>PU(&NiDFm)3&J=%F>c5b)5*$F?Nm1zvn)OP!$>{N_k z5u6PnSy>xOV))Y{Q-jjCZ{N{Spr1(c)?1>0<{$aC0KN_ujkyC^lX+1@gm%p`9(XA# zG$#1hrGHA)*<}22fqya2yVUtM+D**O$waQHWT{dHhG5L3;a(N+i|U_FB6J@toG zo)pnjE1jle*z92{&zXCKxZqCWK#S~~JAu(7ZoCx6rL`Gx+3TRWaDu-r%u-)JVYJ45 z3%verExZx0Act-(_5j?9rRC*sWsaRJhpjFf2@00oIq$5yV4^A!518bXNVWSy`pQ@y zcROE&qJckE(ON%YKaGi3e*Gts5;A6xe)QTz0h z=vUR*qIl#f+kHFa^H~FGikP~r@7z*}{uP%*R#&~^Vq$zYzCIV5y;^X;{%f|sd?b>N zuGWFp6dns*jBSOn=eOVRP86$TW(5O-3Y#dWb=3xHl`p7X^z`%=Dom@RB{hYPg)d%o zz}`jElz`scD?<{}_MX?~Jjb;HaV1~vRg8DEv2fhnR#gV!L3o$~%F&MJ&_XA-equ2l zl(-pxiXND_Y4_!4I+AeibxOE4v5`K%21-l1JrTyUOB#M2w;JxV&&+^wg>*hR0AmQ8 zmQh?$*2Kucprh>6Kpco%$`NO@yM}P*8qSSV4nWS#&PM8bN$+iK%?%G*G`F}pk*ekQ zCOM%5AgUbloI?lfB}t48Ie0vo6jveV<1Qh2z1B_2ym3YtfErz|;3cGcKdxx96z2>! zR_^=vYg4#BZYk%8ut=;|Gu3=@q9uY;)bR?Y#F@oq&=_cjZ_zqp|IiG8~w0>!T zqb5^X7t7yv_Quk@8QOQZzZFCiDM`v+?ef}#r+%n3>3LtDZ~YZDs<&VNRj#C7DZ_3=bSEEe&Qi1G2TH639!)CHQ89g4f`I4HElmxcso2 z)x3SZy#f1=?0>|R(7Rogu+~Ez&pYGX9(z{Y@UWV$i9*>7wKV?S$35+)HVNcN_zTHy zVu164G0$ktADmnvDt|id+u0A<{Z;7YzP2U`NFLKW4HsvkvbwG5vFd72-k|kC8FpWs z@o2)wYXw{TLW$44`RkzMF+aTcC2Q>3G}Czr_nQY(he7)ZW}au_J)q2@6+-bG9eN|9 zUv8lr&}pfRCX1*P%8yo^oSfRt>V1+4h@V=1H$ki)t?|T=3R;tONDZfXTtLKatr2K7 zJ22icNeKF3L3(7L88(ZP;90Som^iKZMh-F(UTw)1DpAo=)%|#Kr$Jg5#o@Nu z-L(YAhIW*2HZUjq_IPIViJ)R<)b8w=++TF##iNDM)4Etia9GpUk%*w}?Z$anv|`9G zrCUPpL#N3JU%aSS=wbfAsorT*TTDl12Z`pCkXzE+Rvqi>5ruE&<2VuIvc3hr5;ASC zmD?GGx}z2u83|U@+ZA(z{?I(Z; z&H@X>;l0gqx+mA;8f3INIdF3R>whvlz+kjpdy_Vkw|Hgl)mni*?Pe43*5yiaa`M4@ z>#E?Ys{ORlYWFsg%i9adY=MEq^%KWg_1qX|OfVh#%G%xMWLa}2xVm56`zhj)ZMRi_ zq3Lt;PFmf$pl!>U4bY3)xRWLAN?qp-Kw?){^S{ft`j@e~J#<#n1zCiQ>>y5TXOwB; z^os9ZDvw;chzcJa-F+e2BydYlteaR-Rygn>csO@lk(E4n!;50>@e}Xsd*@UVM7xqP!j3wQ+S`8#cV#PvdYBS-|%Kt!L{-_ zAjRimTY*GVKDF0{{aIF4N=mxHckS{GAsI0DjDgwG3Rd{ zlUbYbk)lIj(Hjd5Qw7`k%BGz>f5@qObxxBRSuNolx~F*MuLiBnk(x$EUOa7XPtPpk z(#jUsUO&&)+Jl(aLQG5lFMSGd!fHuQR^xf!~n0vzn{Yh!0ur)$X&^8)K zv3*@fM%z1uJBii3;=X91f+7SB5~#UZCm&X^I6oi3D~kcehHX8v+H?7B?z=D2veH{i zJpub%Y4aL?x_u)wdTugJhG zuYzmp{ba(RRidwP7>mtTWoMs(h9pFx9~cd2LuM{;YxS%wne4{0?QYIj?Ib#~kk2bR zJ(B>xjK+pdPkJdci2BW)i{$aPmtqAq9pZKGMvDX#yZnUDm^&p`m^{&1+|=~B=@=g{ zL1)gM={J>R3_%#{TPMJ)e&6z@E}T7kmY+WhxY;kf3CS;CL$U_3TrM47`UX15Qh@_x zqOz@8Pi5DaeUM+4-0-o|jj`lfL9$(^{V_iFe#-2x;^N}kSnPA7W@J5!Y@*G>DW`U zi^MxgiEpp>Mti)b6>Uo^Jxo#wyMm>a&FO#pT|a*L@)tu-jSU$HZ#~bhWWJ1bF{8$i zlV`p*R46WgQ&v_6B$$B@Sb?@-BEG-Dhgv5HjyL`vrhDIGr4fn3`XiV-G1iT$wMG{F ze1AfVZdDBq4mL6}g2VT-b($Jr@(o}6&qETT9HJ1~zp)V$tA@27u%zPNv*?jR^T5&T z22>1{a#L)wP7tj1XHh=K_`nta^72p@4mA39^qabeb|6&`@FgHHg!q@NLi2xXM`+i| ztS;K2q!3rG=*lx@>zi^wsNu6r61FTS7Oe{XF^>^TDT9C{5lN+0feg{g4gtOXxy*7kh5Y6S;IPR2; zx9JmmZZoj3a6_%TV$4UHK8Ywi_h|; zW;?$DJ+T?&QU!(_b0py|wm9E91wIc*O@X#<>7Ia|&_6Jsp`kujKZ*a~J2Op7*)hhY zxw;6RyCz;x@N=8S2w+7kyt-EYH4;`RDCikoi+F}m%sI6J>l*xIGOg*<{dH-0YfqCe zNXJwK+#rR}q7<3A!Q3BfJy~G5ng+JpnVFpny1Ypdg0l~>YcTUM@@drWHk)%-XXo$^ zk%FnA>aq?y4zOp0vx{^Q z2#;opO|;zfr3B+#Y73tn4j1ab_eto6SzFpMUial{?JiT_cro>sEpTHXk^%=cykd5C zc6Hw#tt7lZRKfv!KzPOH&&??SnlSWKJ*~;q+H8VP>Vd^W=vc)*)Y5?`!0&uMW>Eno zEO9xYxk=z8@AdAc>{O=+F3{*zTbYW$r1horLYn9^GU{2KyY$1oNK@$2qGDooX|LRS z$?lw9r3T$SJ+G1EfRK<75Rt91<==<#^|l9HSi97Ba*gs47H6N`tn=~S+jiW2WbE&M zn_6Gj(b;KkK2g5ErE=@L9^dCFDS^xS?Ue?~6w2(1W5enlH8omVszhfLA}2w6WZ=}_ zBqiKj%pyw|YPQI19WrnN5VVS$ix>`iSS@1bCGc5oM3TAGkcWb~Ws-Nf@lb{T{vj5d4=FXWU zX}9rl!)vf$%w6!@TBl&2O4J|UiX81_Ywm^6(?j}ZoP3^!!La)&hEhka;MUEnGG_{$ zv@cJ38y|sE9BGZYLOZ^)UYG#cB?8=c`{ALjxj`c|8hQS@E2NDf{*S`6&s5ni)Kxi}BsV_J`&MKsTjns%?sqT)Lj8|+e86QHxSWIK zB{dHP)0ebL%Sx@W8})&KlvM1qnGQ5mg*18zMf0nnkCaq#G2W_rz(Iwpx(%7kS0JJ z-%E(VI5O~k7U0#4ick-&J=%JNcQy!t@`c(f5%Z!Be63hgJE?^{7YEgJDP##>t2R$_K0 zEba@`VfR;*lx=1$lapUpN!ZKyOhtodPbs1M_O0ZmPXk2u8dajkYT0(z#uY6rCck-> zr^r?@ado3gropuT7}IpZbD6};!}9v&%M5q&BQ}WFeCp`fc($5&utU5FHuUr7s<6;9 za#guuii>Eo|Us#P^RxGYOR(=NQVgTqLH>Lf5dV7eq0+<~-$qhSfnxB-_$Rn6bb z)~x$|VEZerM!?lphP?iQ;RY6~H|nH3+D!hGCQ6kpO7SB79UeH5KGrQKpGHF@L3#4O z%iwf)&WjF4x?7ur4(;Rtn+HxBlu+T{bYlQirFI^C8X0luiY$aaQNWz5^i&-sRFm|Z z9O6EyUR~c04Xd@Bl&{}j-D{ZUI69I?+q1ayS?TD${J2YdmOK64*SghoW^*%m1WSkR zmIdggHCsy@3Z;&tPiQZz2(kYv!E~v`Z$?J4S~N{+{ghNxY(BrgUBGO6%fIS*7!mi- z@%a6d(1jBH$Ok|pnwl;Q4dy{!!E4cTZg9*K_SV6^K5kKmYxxY&@~-^yIQ|xkO-Q(j z*-s7&0J@`}P|luzgbT4F7q7kdK6A}Hy44cYVh~XU%NFP6Iw;=CXOH^ogMOWB_$#0L z*yfJ0dcJjG66}CJlVu=q#_4b-=G|5Mc%r$j&9x~;380N2l|OHKM{D0Tv~{!^%Cq_j zP-b$K(7AcrL`7herobdnv diff --git a/tests/unit/pipelines/wan_video/test_sol_quant_example.py b/tests/unit/pipelines/wan_video/test_sol_quant_example.py index 2e1a8db..9fa8a01 100644 --- a/tests/unit/pipelines/wan_video/test_sol_quant_example.py +++ b/tests/unit/pipelines/wan_video/test_sol_quant_example.py @@ -11,7 +11,6 @@ ) from telefuser.core.config import AttnImplType, QuantConfig, QuantKernelBackend, QuantType from telefuser.models.wan_video_dit import WanModel -from telefuser.ops.fp8_gemm import FP8Linear def test_wan_sol_quant_example_builds_compatible_configs() -> None: @@ -50,10 +49,23 @@ def test_wan_sol_quant_example_rejects_unknown_quantization() -> None: make_quant_config("int8") -def test_wan_model_enables_tf_kernel_fp8_on_transformer_blocks() -> None: +def test_wan_model_enables_tf_kernel_fp8_on_transformer_blocks(monkeypatch: pytest.MonkeyPatch) -> None: model = WanModel.__new__(WanModel) torch.nn.Module.__init__(model) model.blocks = torch.nn.ModuleList([torch.nn.Sequential(torch.nn.Linear(8, 16), torch.nn.Linear(16, 8))]) + calls = [] + + def fake_count(module, *, module_filter=None): + calls.append(("count", module, module_filter)) + return 2 + + def fake_enable(module, *, options, module_filter=None): + calls.append(("enable", module, options, module_filter)) + return module + + monkeypatch.setattr("telefuser.ops.fp8_gemm.tf_kernel", None) + monkeypatch.setattr("telefuser.ops.fp8_gemm.count_linear_layers", fake_count) + monkeypatch.setattr("telefuser.ops.fp8_gemm.enable_fp8_gemm", fake_enable) model.enable_quant( QuantConfig( @@ -64,10 +76,18 @@ def test_wan_model_enables_tf_kernel_fp8_on_transformer_blocks() -> None: ) assert model.tf_kernel_fp8_replaced_linear == 2 - assert all(isinstance(module, FP8Linear) for module in model.blocks[0]) - assert not model.blocks[0][0].options.cast_output_back - output = model.blocks[0](torch.randn(2, 8)) - assert output.shape == (2, 8) + assert model.quant_type is QuantType.FP8 + assert calls[0][0] == "count" + assert calls[0][1] is model + assert calls[1][0] == "enable" + assert calls[1][1] is model + options = calls[1][2] + assert not options.cast_output_back + assert options.fp16_weight_storage == "discard" + assert options.materialize_fp8_on_wrap + module_filter = calls[1][3] + assert module_filter("blocks.0.0", model.blocks[0][0]) + assert not module_filter("head", model.blocks[0][0]) def test_dense_fp8_example_selects_dense_attention(monkeypatch: pytest.MonkeyPatch) -> None: From 478b714bbb8f748c817b18519ed013b7eff2897c Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Wed, 12 Aug 2026 08:58:20 +0000 Subject: [PATCH 03/15] Consolidate Wan optimization example Replace the separate dense FP8 and Sol-Attn FP8 scripts with one Wan2.1 example that independently selects attention and quantization from CLI configuration. Document Dense/SOL and BF16/tf-kernel FP8/TorchAO FP8/NF4 combinations, and rename the focused tests around the consolidated interface. Verified the direct --help entry point, 18 focused tests including the H100 SOL kernel, pre-commit hooks, PR-scope diff checks, and stale-reference audit. --- examples/wan_video/README.md | 90 +++++++------- .../wan21_1_3b_text_to_video_fp8_h100.py | 112 ------------------ ...an21_1_3b_text_to_video_optimized_h100.py} | 38 +++--- ...t_example.py => test_optimized_example.py} | 62 +++------- 4 files changed, 86 insertions(+), 216 deletions(-) delete mode 100644 examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py rename examples/wan_video/{wan21_1_3b_text_to_video_sol_fp8_h100.py => wan21_1_3b_text_to_video_optimized_h100.py} (87%) rename tests/unit/pipelines/wan_video/{test_sol_quant_example.py => test_optimized_example.py} (70%) diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 3be130c..8bfa355 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -89,36 +89,6 @@ python examples/wan_video/wan21_1_3b_text_to_video_h100.py --resolution 480p --a **Features:** - Video Frame Interpolation (VFI) with RIFE model for 30fps output - CFG parallel when cfg_scale > 1 - -#### wan21_1_3b_text_to_video_fp8_h100.py - -Wan2.1 1.3B T2V with dense attention and online FP8 Linear layers. -The VAE and text encoder remain BF16; only DiT transformer-block Linear layers are -quantized. `tf-kernel-fp8` selects TeleFuser's dynamic W8A8 FP8 GEMM path; -`torchao-fp8` selects TorchAO's dynamic-activation/FP8-weight implementation. -Both paths keep BF16 outputs compatible with the rest of the Wan pipeline. - -```bash -python examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py \ - --model-root /path/to/Wan2.1-T2V-1.3B \ - --quantization tf-kernel-fp8 \ - --resolution 480p \ - --output wan21_fp8.mp4 -``` - -To use TorchAO instead: - -```bash -python examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py \ - --model-root /path/to/Wan2.1-T2V-1.3B \ - --quantization torchao-fp8 \ - --output wan21_torchao_fp8.mp4 -``` - -Use `--quantization none` with the same example to run the BF16 dense baseline. -The final log reports generation time, generated frames per second, and peak CUDA -allocated/reserved memory. - #### wan21_1_3b_text_to_video_hf.py T2V with HuggingFace format loading. @@ -202,32 +172,59 @@ pipe_config.dit_config.attention_config = AttentionConfig.sol_attention() Sol-Attn is built into TeleFuser. Eligible BF16 self-attention calls use the sparse kernel; unsupported calls automatically use the existing dense fallback. The defaults follow the official Wan2.1 profile: Morton3D token ordering, dense layer 0, and 10 dense warm-up steps for the standard 50-step schedule. +#### wan21_1_3b_text_to_video_optimized_h100.py + +Provides one entry point for independently enabling attention and quantization +optimizations. The defaults are `--attention dense --quantization none`, which +run the BF16 baseline. + +Attention choices: -#### wan21_1_3b_text_to_video_sol_fp8_h100.py +- `dense`: PyTorch SDPA +- `sol`: Sol-Attn with dense warm-up and fallback calls -Combines Sol-Attn with online DiT quantization. The default `tf-kernel-fp8` mode -quantizes transformer Linear layers while preserving BF16 Q/K/V tensors required -by Sol-Attn. Dense warm-up calls and unsupported cross-attention calls use the -normal dense attention fallback. +Quantization choices: + +- `none`: BF16 DiT +- `tf-kernel-fp8`: TeleFuser dynamic W8A8 FP8 GEMM +- `torchao-fp8`: TorchAO dynamic-activation/FP8-weight +- `bnb-nf4`: bitsandbytes NF4 + +Only DiT transformer-block Linear layers are quantized; the VAE and text encoder +remain BF16. Select the two optimization axes independently: ```bash -python examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py \ +# Dense + BF16 baseline +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --attention dense --quantization none + +# Sol-Attn + BF16 +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --attention sol --quantization none + +# Dense + tf-kernel FP8 +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ + --attention dense --quantization tf-kernel-fp8 + +# Sol-Attn + tf-kernel FP8 +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --attention sol \ --quantization tf-kernel-fp8 \ --dense-timesteps 10 \ --dense-layers 1 \ --tau 1.0 \ --threshold-type diag \ - --kv-splits auto \ - --output wan21_sol_fp8.mp4 + --kv-splits auto ``` -Replace `--quantization tf-kernel-fp8` with `--quantization torchao-fp8` to use -TorchAO FP8 with the same Sol-Attn configuration. - -For ablations, use `--quantization none` for BF16 + Sol-Attn and use the standalone -FP8 example for FP8 + dense attention. Compare only runs with the same prompt, -seed, resolution, frame count, inference steps, and kernel warm-up policy. +Replace `tf-kernel-fp8` with `torchao-fp8` or `bnb-nf4` without changing +the attention mode. The SOL tuning options apply only when `--attention sol`. +The final log reports generation time, frames per second, and peak allocated and +reserved CUDA memory. ##### H100 benchmark @@ -252,12 +249,13 @@ The benchmark prompt is: > Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely > on a spotlighted stage. -Example command (replace the quantization and example script for each ablation): +Example command (change `--attention` and `--quantization` for each ablation): ```bash -python examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py \ +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ + --attention sol \ --quantization tf-kernel-fp8 \ --width 832 --height 480 \ --num-frames 81 --num-inference-steps 50 \ diff --git a/examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py deleted file mode 100644 index 343880b..0000000 --- a/examples/wan_video/wan21_1_3b_text_to_video_fp8_h100.py +++ /dev/null @@ -1,112 +0,0 @@ -"""Wan2.1 1.3B T2V with tf-kernel FP8 Linear layers and dense attention.""" - -from __future__ import annotations - -import time - -import click -import torch - -if __package__: - from examples.wan_video.wan21_1_3b_text_to_video_sol_fp8_h100 import ( - PPL_CONFIG, - configure_attention_backends, - run, - ) - from examples.wan_video.wan21_1_3b_text_to_video_sol_fp8_h100 import ( - get_pipeline as get_quantized_pipeline, - ) -else: - from wan21_1_3b_text_to_video_sol_fp8_h100 import ( - PPL_CONFIG, - configure_attention_backends, - run, - ) - from wan21_1_3b_text_to_video_sol_fp8_h100 import ( - get_pipeline as get_quantized_pipeline, - ) - -from telefuser.core.config import AttentionConfig, AttnImplType -from telefuser.utils.utils import get_example_name -from telefuser.utils.video import save_video - - -def get_pipeline( - *, - model_root: str = PPL_CONFIG["model_root"], - quantization: str = "tf-kernel-fp8", - sample_solver: str = "euler", -): - """Load Wan2.1 with online FP8 quantization and dense SDPA attention.""" - return get_quantized_pipeline( - model_root=model_root, - quantization=quantization, - attention_config=AttentionConfig.dense_attention(AttnImplType.TORCH_SDPA), - sample_solver=sample_solver, - ) - - -@click.command() -@click.option("--prompt", default="A small paper boat floating down a sunlit stream.") -@click.option("--seed", default=42, type=int) -@click.option("--resolution", default="480p", type=click.Choice(["480p", "720p"])) -@click.option("--width", type=int) -@click.option("--height", type=int) -@click.option("--num-inference-steps", default=PPL_CONFIG["num_inference_steps"], type=int) -@click.option("--num-frames", default=PPL_CONFIG["num_frames"], type=int) -@click.option("--cfg-scale", default=PPL_CONFIG["cfg_scale"], type=float) -@click.option("--sigma-shift", default=PPL_CONFIG["sigma_shift"], type=float) -@click.option("--sample-solver", default="euler", type=click.Choice(["euler", "unipc"])) -@click.option("--model-root", default=PPL_CONFIG["model_root"]) -@click.option( - "--quantization", - default="tf-kernel-fp8", - type=click.Choice(["tf-kernel-fp8", "torchao-fp8", "none"]), -) -@click.option("--output", default=get_example_name(__file__, "mp4")) -def main( - prompt: str, - seed: int, - resolution: str, - width: int | None, - height: int | None, - num_inference_steps: int, - num_frames: int, - cfg_scale: float, - sigma_shift: float, - sample_solver: str, - model_root: str, - quantization: str, - output: str, -) -> None: - """Run dense Wan2.1 with tf-kernel/TorchAO FP8 or a BF16 baseline.""" - configure_attention_backends() - pipeline = get_pipeline(model_root=model_root, quantization=quantization, sample_solver=sample_solver) - torch.cuda.reset_peak_memory_stats() - start = time.perf_counter() - video = run( - pipeline, - prompt, - seed=seed, - resolution=resolution, - width=width, - height=height, - num_inference_steps=num_inference_steps, - num_frames=num_frames, - cfg_scale=cfg_scale, - sigma_shift=sigma_shift, - ) - torch.cuda.synchronize() - elapsed = time.perf_counter() - start - save_video(video, output, fps=16, quality=6) - peak_allocated = torch.cuda.max_memory_allocated() / 2**30 - peak_reserved = torch.cuda.max_memory_reserved() / 2**30 - click.echo( - f"quantization={quantization} sol_attention=false elapsed_s={elapsed:.2f} " - f"throughput_fps={num_frames / elapsed:.4f} " - f"peak_allocated_gib={peak_allocated:.3f} peak_reserved_gib={peak_reserved:.3f} output={output}" - ) - - -if __name__ == "__main__": - main() diff --git a/examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py similarity index 87% rename from examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py rename to examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py index 1c61b5a..89bb2e6 100644 --- a/examples/wan_video/wan21_1_3b_text_to_video_sol_fp8_h100.py +++ b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py @@ -1,10 +1,8 @@ -"""Wan2.1 1.3B T2V with Sol-Attn and online weight quantization. +"""Wan2.1 1.3B T2V with optional attention and quantization optimizations. -The DiT is quantized with tf-kernel FP8, TorchAO FP8, or bitsandbytes NF4 while attention -continues to consume contiguous BF16 Q/K/V tensors required by Sol-Attn. -This example is intended for H100 validation and keeps the quantized DiT on -CUDA so the quantized Linear modules are not repeatedly reconstructed during -CPU offload. +Attention can use dense SDPA or Sol-Attn. DiT Linear layers can remain BF16 or +use tf-kernel FP8, TorchAO FP8, or bitsandbytes NF4. The example keeps the DiT +on CUDA so quantized Linear modules are not repeatedly reconstructed. """ from __future__ import annotations @@ -17,6 +15,7 @@ from telefuser.core.config import ( AttentionConfig, + AttnImplType, QuantConfig, QuantKernelBackend, QuantType, @@ -43,7 +42,7 @@ def configure_attention_backends() -> None: - """Avoid cuDNN plans being selected for dense SOL guard/fallback calls.""" + """Configure dense attention backends used directly or by SOL fallbacks.""" if hasattr(torch.backends.cuda, "enable_cudnn_sdp"): torch.backends.cuda.enable_cudnn_sdp(False) if hasattr(torch.backends.cuda, "enable_flash_sdp"): @@ -80,6 +79,7 @@ def make_quant_config(quantization: str) -> QuantConfig: def make_attention_config( + attention: str, *, dense_timesteps: int = 10, dense_layers: int = 1, @@ -87,7 +87,11 @@ def make_attention_config( threshold_type: str = "diag", kv_splits: int | str = "auto", ) -> AttentionConfig: - """Build the official Wan2.1 Sol-Attn profile.""" + """Build the selected dense or Sol-Attn configuration.""" + if attention == "dense": + return AttentionConfig.dense_attention(AttnImplType.TORCH_SDPA) + if attention != "sol": + raise ValueError("attention must be 'dense' or 'sol'") return AttentionConfig.sol_attention( dense_timesteps=dense_timesteps, dense_layers=dense_layers, @@ -100,16 +104,16 @@ def make_attention_config( def get_pipeline( *, model_root: str = PPL_CONFIG["model_root"], - quantization: str = "tf-kernel-fp8", + attention: str = "dense", + quantization: str = "none", dense_timesteps: int = 10, dense_layers: int = 1, tau: float = 1.0, threshold_type: str = "diag", kv_splits: int | str = "auto", - attention_config: AttentionConfig | None = None, sample_solver: str = "euler", ) -> Wan21VideoPipeline: - """Load a Wan2.1 pipeline with SOL and optional online quantization.""" + """Load Wan2.1 with independently selectable attention and quantization.""" quant_config = make_quant_config(quantization) module_manager = ModuleManager(torch_dtype=torch.bfloat16, device="cpu") module_manager.load_model(f"{model_root}/Wan2.1_VAE.pth", device="cpu", torch_dtype=torch.bfloat16) @@ -123,7 +127,8 @@ def get_pipeline( pipeline = Wan21VideoPipeline(device="cuda", torch_dtype=torch.bfloat16) config = Wan21VideoPipelineConfig() - config.dit_config.attention_config = attention_config or make_attention_config( + config.dit_config.attention_config = make_attention_config( + attention, dense_timesteps=dense_timesteps, dense_layers=dense_layers, tau=tau, @@ -184,9 +189,10 @@ def run( @click.option("--sigma-shift", default=PPL_CONFIG["sigma_shift"], type=float) @click.option("--sample-solver", default="euler", type=click.Choice(["euler", "unipc"])) @click.option("--model-root", default=PPL_CONFIG["model_root"]) +@click.option("--attention", default="dense", type=click.Choice(["dense", "sol"])) @click.option( "--quantization", - default="tf-kernel-fp8", + default="none", type=click.Choice(["none", "tf-kernel-fp8", "torchao-fp8", "bnb-nf4"]), ) @click.option("--dense-timesteps", default=10, type=int) @@ -207,6 +213,7 @@ def main( sigma_shift: float, sample_solver: str, model_root: str, + attention: str, quantization: str, dense_timesteps: int, dense_layers: int, @@ -215,10 +222,11 @@ def main( kv_splits: str, output: str, ) -> None: - """Run Wan2.1 SOL + quantization validation.""" + """Run Wan2.1 with optional attention and quantization optimizations.""" configure_attention_backends() pipeline = get_pipeline( model_root=model_root, + attention=attention, quantization=quantization, dense_timesteps=dense_timesteps, dense_layers=dense_layers, @@ -247,7 +255,7 @@ def main( peak_allocated = torch.cuda.max_memory_allocated() / 2**30 peak_reserved = torch.cuda.max_memory_reserved() / 2**30 click.echo( - f"quantization={quantization} sol_attention=true elapsed_s={elapsed:.2f} " + f"attention={attention} quantization={quantization} elapsed_s={elapsed:.2f} " f"throughput_fps={num_frames / elapsed:.4f} " f"peak_allocated_gib={peak_allocated:.3f} peak_reserved_gib={peak_reserved:.3f} output={output}" ) diff --git a/tests/unit/pipelines/wan_video/test_sol_quant_example.py b/tests/unit/pipelines/wan_video/test_optimized_example.py similarity index 70% rename from tests/unit/pipelines/wan_video/test_sol_quant_example.py rename to tests/unit/pipelines/wan_video/test_optimized_example.py index 9fa8a01..c4858f6 100644 --- a/tests/unit/pipelines/wan_video/test_sol_quant_example.py +++ b/tests/unit/pipelines/wan_video/test_optimized_example.py @@ -3,8 +3,7 @@ import pytest import torch -from examples.wan_video.wan21_1_3b_text_to_video_fp8_h100 import get_pipeline as get_fp8_pipeline -from examples.wan_video.wan21_1_3b_text_to_video_sol_fp8_h100 import ( +from examples.wan_video.wan21_1_3b_text_to_video_optimized_h100 import ( make_attention_config, make_quant_config, run, @@ -13,8 +12,8 @@ from telefuser.models.wan_video_dit import WanModel -def test_wan_sol_quant_example_builds_compatible_configs() -> None: - attention = make_attention_config() +def test_wan_optimized_example_builds_compatible_configs() -> None: + attention = make_attention_config("sol") quant = make_quant_config("torchao-fp8") assert attention.attn_impl is AttnImplType.SOL_ATTN @@ -25,6 +24,18 @@ def test_wan_sol_quant_example_builds_compatible_configs() -> None: assert quant.kernel_backend is QuantKernelBackend.TORCHAO +def test_wan_optimized_example_builds_dense_attention_config() -> None: + attention = make_attention_config("dense") + + assert attention.attn_impl is AttnImplType.TORCH_SDPA + assert attention.sparse_config is None + + +def test_wan_optimized_example_rejects_unknown_attention() -> None: + with pytest.raises(ValueError, match="attention must be"): + make_attention_config("radial") + + @pytest.mark.parametrize( ("name", "quant_type", "backend"), [ @@ -34,7 +45,7 @@ def test_wan_sol_quant_example_builds_compatible_configs() -> None: ("bnb-nf4", QuantType.BNB_NF4, QuantKernelBackend.BITSANDBYTES), ], ) -def test_wan_sol_quant_example_quantization_choices(name, quant_type, backend) -> None: +def test_wan_optimized_example_quantization_choices(name, quant_type, backend) -> None: config = make_quant_config(name) if name == "none": assert not config.enabled @@ -44,7 +55,7 @@ def test_wan_sol_quant_example_quantization_choices(name, quant_type, backend) - assert config.kernel_backend is backend -def test_wan_sol_quant_example_rejects_unknown_quantization() -> None: +def test_wan_optimized_example_rejects_unknown_quantization() -> None: with pytest.raises(ValueError, match="quantization must be"): make_quant_config("int8") @@ -90,42 +101,7 @@ def fake_enable(module, *, options, module_filter=None): assert not module_filter("head", model.blocks[0][0]) -def test_dense_fp8_example_selects_dense_attention(monkeypatch: pytest.MonkeyPatch) -> None: - captured = {} - - def fake_pipeline(**kwargs): - captured.update(kwargs) - return object() - - monkeypatch.setattr( - "examples.wan_video.wan21_1_3b_text_to_video_fp8_h100.get_quantized_pipeline", - fake_pipeline, - ) - - get_fp8_pipeline(model_root="model", quantization="tf-kernel-fp8") - - assert captured["quantization"] == "tf-kernel-fp8" - assert captured["attention_config"].attn_impl is AttnImplType.TORCH_SDPA - - -def test_dense_fp8_example_forwards_torchao_choice(monkeypatch: pytest.MonkeyPatch) -> None: - captured = {} - - def fake_pipeline(**kwargs): - captured.update(kwargs) - return object() - - monkeypatch.setattr( - "examples.wan_video.wan21_1_3b_text_to_video_fp8_h100.get_quantized_pipeline", - fake_pipeline, - ) - - get_fp8_pipeline(model_root="model", quantization="torchao-fp8") - - assert captured["quantization"] == "torchao-fp8" - - -def test_wan_sol_quant_run_forwards_explicit_benchmark_parameters() -> None: +def test_wan_optimized_run_forwards_explicit_benchmark_parameters() -> None: captured = {} def fake_pipeline(**kwargs): @@ -154,6 +130,6 @@ def fake_pipeline(**kwargs): assert captured["sigma_shift"] == 5.0 -def test_wan_sol_quant_run_requires_width_and_height_together() -> None: +def test_wan_optimized_run_requires_width_and_height_together() -> None: with pytest.raises(ValueError, match="width and height must be provided together"): run(lambda **_kwargs: object(), "prompt", width=832) From a482ef4b48434ce7ca0cf3b0b1d3f9028fe8f2b4 Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Thu, 13 Aug 2026 08:51:05 +0000 Subject: [PATCH 04/15] feat(wan): run Sol-Attn with FP8 QKV GEMMs --- examples/wan_video/README.md | 7 +- ...wan21_1_3b_text_to_video_optimized_h100.py | 7 +- telefuser/core/config.py | 3 + telefuser/kernel/sol_attn/interface.py | 41 +++- telefuser/kernel/sol_attn/triton_ref/fwd.py | 184 ++++++++---------- .../kernel/sol_attn/triton_ref/preprocess.py | 38 +++- telefuser/models/wan_video_dit.py | 21 +- telefuser/ops/attention/attention_impl.py | 22 ++- telefuser/ops/fp8_attention.py | 41 ++++ .../models/test_wan_video_sol_attention.py | 28 ++- .../wan_video/test_optimized_example.py | 8 + 11 files changed, 280 insertions(+), 120 deletions(-) create mode 100644 telefuser/ops/fp8_attention.py diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 8bfa355..875fdda 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -182,6 +182,7 @@ Attention choices: - `dense`: PyTorch SDPA - `sol`: Sol-Attn with dense warm-up and fallback calls +- `sol-fp8`: FP8 QKV inputs with FP8 Sol-Attn QK/PV GEMMs Quantization choices: @@ -212,7 +213,7 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ # Sol-Attn + tf-kernel FP8 python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ - --attention sol \ + --attention sol-fp8 \ --quantization tf-kernel-fp8 \ --dense-timesteps 10 \ --dense-layers 1 \ @@ -222,7 +223,9 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ ``` Replace `tf-kernel-fp8` with `torchao-fp8` or `bnb-nf4` without changing -the attention mode. The SOL tuning options apply only when `--attention sol`. +the attention mode. `sol-fp8` quantizes post-RoPE Q/K/V per 64-token block and +runs exact Sol-Attn QK/PV GEMMs with FP8 inputs and FP32 accumulation; routing +summaries remain BF16/FP32. The SOL tuning options apply to both `sol` modes. The final log reports generation time, frames per second, and peak allocated and reserved CUDA memory. diff --git a/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py index 89bb2e6..68b653b 100644 --- a/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py +++ b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py @@ -90,14 +90,15 @@ def make_attention_config( """Build the selected dense or Sol-Attn configuration.""" if attention == "dense": return AttentionConfig.dense_attention(AttnImplType.TORCH_SDPA) - if attention != "sol": - raise ValueError("attention must be 'dense' or 'sol'") + if attention not in ("sol", "sol-fp8"): + raise ValueError("attention must be 'dense', 'sol', or 'sol-fp8'") return AttentionConfig.sol_attention( dense_timesteps=dense_timesteps, dense_layers=dense_layers, tau=tau, threshold_type=threshold_type, kv_splits=kv_splits, + sol_fp8=attention == "sol-fp8", ) @@ -189,7 +190,7 @@ def run( @click.option("--sigma-shift", default=PPL_CONFIG["sigma_shift"], type=float) @click.option("--sample-solver", default="euler", type=click.Choice(["euler", "unipc"])) @click.option("--model-root", default=PPL_CONFIG["model_root"]) -@click.option("--attention", default="dense", type=click.Choice(["dense", "sol"])) +@click.option("--attention", default="dense", type=click.Choice(["dense", "sol", "sol-fp8"])) @click.option( "--quantization", default="none", diff --git a/telefuser/core/config.py b/telefuser/core/config.py index 505b481..e6d4316 100644 --- a/telefuser/core/config.py +++ b/telefuser/core/config.py @@ -151,6 +151,7 @@ class SparseAttentionConfig: sol_tau: float = 1.0 # Sol-Attn routing threshold multiplier sol_threshold_type: str = "diag" # Sol-Attn threshold estimator: "diag" or "exact" sol_kv_splits: int | str = "auto" # Auto selects split 4 for long SM90 sequences + sol_fp8: bool = False # Quantize post-RoPE Q/K/V activations for FP8 Sol-Attn def __post_init__(self) -> None: if self.sparse_impl != "sol": @@ -229,6 +230,7 @@ def sol_attention( tau: float = 1.0, threshold_type: str = "diag", kv_splits: int | str = "auto", + sol_fp8: bool = False, **kwargs: any, ) -> AttentionConfig: """Create a Sol-Attn config for dynamic sparse video self-attention.""" @@ -241,6 +243,7 @@ def sol_attention( sol_tau=tau, sol_threshold_type=threshold_type, sol_kv_splits=kv_splits, + sol_fp8=sol_fp8, ), **kwargs, ) diff --git a/telefuser/kernel/sol_attn/interface.py b/telefuser/kernel/sol_attn/interface.py index 933513a..61aa569 100644 --- a/telefuser/kernel/sol_attn/interface.py +++ b/telefuser/kernel/sol_attn/interface.py @@ -27,8 +27,8 @@ def _validate_inputs( raise ValueError("q, k, and v must share shape [B, T, H, 128]") if q.shape[1] == 0 or q.shape[3] != 128: raise ValueError("Sol-Attn requires T > 0 and head dimension 128") - if any(x.dtype != torch.bfloat16 for x in (q, k, v)): - raise TypeError("q, k, and v must use torch.bfloat16") + if any(x.dtype not in (torch.bfloat16, torch.float8_e4m3fn) for x in (q, k, v)): + raise TypeError("q, k, and v must use torch.bfloat16 or torch.float8_e4m3fn") if q.device.type != "cuda" or k.device != q.device or v.device != q.device: raise ValueError("q, k, and v must be on the same CUDA device") if not (q.is_contiguous() and k.is_contiguous() and v.is_contiguous()): @@ -343,14 +343,49 @@ def sol_attn( kv_splits: int = 1, sink_tokens: int = 0, sink_start: int | None = None, + q_scale: torch.Tensor | None = None, + k_scale: torch.Tensor | None = None, + v_scale: torch.Tensor | None = None, ) -> torch.Tensor: - """Compute noncausal Sol-Attn for contiguous BF16 BTHD tensors. + """Compute noncausal Sol-Attn for contiguous BF16 or FP8 BTHD tensors. ``sink_start`` and ``sink_tokens`` keep every KV block overlapping the corresponding contiguous token range exact for all queries. Omitting ``sink_start`` places the range at the token suffix. """ + fp8_inputs = any(x.dtype == torch.float8_e4m3fn for x in (q, k, v)) + if fp8_inputs: + if not all(x.dtype == torch.float8_e4m3fn for x in (q, k, v)): + raise TypeError("q, k, and v must all use the same dtype") + if any(scale is None for scale in (q_scale, k_scale, v_scale)): + raise ValueError("FP8 Sol-Attn requires q_scale, k_scale, and v_scale") + _validate_inputs(q, k, v, thresh_type, sink_tokens, sink_start) + if kv_splits != 1: + raise ValueError("FP8 Sol-Attn currently supports kv_splits=1") + blocks = (q.shape[1] + BLOCK_SIZE - 1) // BLOCK_SIZE + expected_scale_shape = (q.shape[0], blocks, q.shape[2]) + for name, tensor in (("q_scale", q_scale), ("k_scale", k_scale), ("v_scale", v_scale)): + if tensor.shape != expected_scale_shape or tensor.device != q.device: + raise ValueError(f"{name} must have shape {expected_scale_shape} on the Q/K/V device") + if not tensor.is_contiguous(): + raise ValueError(f"{name} must be contiguous") + from .triton_ref import sol_attn as triton_sol_attn + + return triton_sol_attn( + q, + k, + v, + scale=scale, + tau=tau, + thresh_type=thresh_type, + sink_tokens=sink_tokens, + sink_start=sink_start, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, + ) + arch = _validate_inputs( q, k, diff --git a/telefuser/kernel/sol_attn/triton_ref/fwd.py b/telefuser/kernel/sol_attn/triton_ref/fwd.py index 67edcc5..5327284 100644 --- a/telefuser/kernel/sol_attn/triton_ref/fwd.py +++ b/telefuser/kernel/sol_attn/triton_ref/fwd.py @@ -16,7 +16,6 @@ from .preprocess import prepare as prepare_ptr - BLOCK = 64 GROUP = 32 @@ -29,11 +28,7 @@ def _use_tma(device) -> bool: @triton.autotune( - configs=[ - triton.Config({}, num_warps=warps, num_stages=stages) - for warps in (4, 8) - for stages in (1, 2, 3, 4) - ], + configs=[triton.Config({}, num_warps=warps, num_stages=stages) for warps in (4, 8) for stages in (1, 2, 3, 4)], key=["T"], ) @triton.jit @@ -80,39 +75,23 @@ def _forward_tma( row_max = tl.full((BLOCK_SIZE,), -float("inf"), tl.float32) scale_log2 = scale * 1.4426950408889634 tail_length = T - (NT - 1) * BLOCK_SIZE - route_threshold = tl.load( - threshold + (batch * NT + q_block) * H + head - ) + route_threshold = tl.load(threshold + (batch * NT + q_block) * H + head) for group_start in range(0, NT, GROUP_SIZE): block_indices = group_start + group_offsets valid = block_indices < NT - kc = kc_desc.load( - [batch, group_start, head, 0] - ).reshape([GROUP_SIZE, D]) - vc = vc_desc.load( - [batch, group_start, head, v_tile * BV] - ).reshape([GROUP_SIZE, BV]) + kc = kc_desc.load([batch, group_start, head, 0]).reshape([GROUP_SIZE, D]) + vc = vc_desc.load([batch, group_start, head, v_tile * BV]).reshape([GROUP_SIZE, BV]) scores = tl.dot(q, kc.T).to(tl.float32) * scale_log2 - exact = ( - (tl.sum(scores, axis=0) / q_len > route_threshold) - | (tl.abs(q_block - block_indices) <= 1) - ) + exact = (tl.sum(scores, axis=0) / q_len > route_threshold) | (tl.abs(q_block - block_indices) <= 1) if HAS_SINK: - exact = exact | ( - (block_indices >= sink_start_block) - & (block_indices < sink_end_block) - ) + exact = exact | ((block_indices >= sink_start_block) & (block_indices < sink_end_block)) exact = exact & valid approximate = valid & ~exact - approximate_scores = tl.where( - approximate[None, :], scores, -float("inf") - ) + approximate_scores = tl.where(approximate[None, :], scores, -float("inf")) new_max = tl.maximum(row_max, tl.max(approximate_scores, axis=1)) - alpha = tl.math.exp2( - tl.where(row_max == new_max, 0.0, row_max - new_max) - ) + alpha = tl.math.exp2(tl.where(row_max == new_max, 0.0, row_max - new_max)) approximate_probability = tl.where( approximate[None, :], tl.math.exp2(approximate_scores - new_max[:, None]), @@ -122,9 +101,7 @@ def _forward_tma( approximate_probability.to(vc.dtype), vc, ) - lengths = tl.where( - block_indices == NT - 1, tail_length, BLOCK_SIZE - ).to(tl.float32) + lengths = tl.where(block_indices == NT - 1, tail_length, BLOCK_SIZE).to(tl.float32) row_sum = row_sum * alpha + tl.sum( approximate_probability * lengths[None, :], axis=1, @@ -141,9 +118,7 @@ def _forward_tma( exact_offsets, ) kv_start = block * BLOCK_SIZE - k = k_desc.load( - [batch, kv_start, head, 0] - ).reshape([BLOCK_SIZE, D]) + k = k_desc.load([batch, kv_start, head, 0]).reshape([BLOCK_SIZE, D]) exact_scores = tl.dot(q, k.T).to(tl.float32) * scale_log2 exact_scores += tl.where( (kv_start + token_offsets)[None, :] < T, @@ -152,16 +127,12 @@ def _forward_tma( ) new_max = tl.maximum(row_max, tl.max(exact_scores, axis=1)) alpha = tl.math.exp2(row_max - new_max) - exact_probability = tl.math.exp2( - exact_scores - new_max[:, None] - ) + exact_probability = tl.math.exp2(exact_scores - new_max[:, None]) row_sum = row_sum * alpha + tl.sum( exact_probability, axis=1, ) - v = v_desc.load( - [batch, kv_start, head, v_tile * BV] - ).reshape([BLOCK_SIZE, BV]) + v = v_desc.load([batch, kv_start, head, v_tile * BV]).reshape([BLOCK_SIZE, BV]) output = output * alpha[:, None] + tl.dot( exact_probability.to(v.dtype), v, @@ -188,6 +159,9 @@ def _forward_ptr( q_ptr, k_ptr, v_ptr, + q_scale_ptr, + k_scale_ptr, + v_scale_ptr, kc_ptr, vc_ptr, threshold_ptr, @@ -204,6 +178,7 @@ def _forward_ptr( BV: tl.constexpr, BLOCK_SIZE: tl.constexpr, GROUP_SIZE: tl.constexpr, + FP8: tl.constexpr, ): v_tile, q_block, batch_head = ( tl.program_id(0), @@ -223,44 +198,35 @@ def _forward_ptr( value_dims = v_tile * BV + tl.arange(0, BV) q_tokens = q_block * BLOCK_SIZE + token_offsets q_valid = q_tokens < T - q_offsets = ( - ((batch * T + q_tokens[:, None]).to(tl.int64) * H + head) * D - + dims[None, :] - ) + q_offsets = ((batch * T + q_tokens[:, None]).to(tl.int64) * H + head) * D + dims[None, :] q = tl.load(q_ptr + q_offsets, mask=q_valid[:, None], other=0.0) + if FP8: + q_scale = tl.load(q_scale_ptr + (batch * NT + q_block) * H + head) + q_route = q.to(tl.float32) * q_scale + else: + q_route = q q_len = tl.minimum(BLOCK_SIZE, T - q_block * BLOCK_SIZE).to(tl.float32) output = tl.zeros([BLOCK_SIZE, BV], dtype=tl.float32) row_sum = tl.zeros((BLOCK_SIZE,), dtype=tl.float32) row_max = tl.full((BLOCK_SIZE,), -float("inf"), tl.float32) scale_log2 = scale * 1.4426950408889634 - route_threshold = tl.load( - threshold_ptr + (batch * NT + q_block) * H + head - ) + route_threshold = tl.load(threshold_ptr + (batch * NT + q_block) * H + head) for group_start in range(0, NT, GROUP_SIZE): block_indices = group_start + group_offsets valid = block_indices < NT - kc_offsets = ( - ((batch * NPAD + block_indices[:, None]) * H + head) * D - + dims[None, :] - ) - vc_offsets = ( - ((batch * NPAD + block_indices[:, None]) * H + head) * D - + value_dims[None, :] - ) + kc_offsets = ((batch * NPAD + block_indices[:, None]) * H + head) * D + dims[None, :] + vc_offsets = ((batch * NPAD + block_indices[:, None]) * H + head) * D + value_dims[None, :] kc = tl.load(kc_ptr + kc_offsets) vc = tl.load(vc_ptr + vc_offsets) - scores = tl.dot(q, kc.T).to(tl.float32) * scale_log2 - exact = ( - (tl.sum(scores, axis=0) / q_len > route_threshold) - | (tl.abs(q_block - block_indices) <= 1) - ) + if FP8: + kc = kc.to(tl.float32) + vc = vc.to(tl.float32) + scores = tl.dot(q_route, kc.T).to(tl.float32) * scale_log2 + exact = (tl.sum(scores, axis=0) / q_len > route_threshold) | (tl.abs(q_block - block_indices) <= 1) if HAS_SINK: - exact = exact | ( - (block_indices >= sink_start_block) - & (block_indices < sink_end_block) - ) + exact = exact | ((block_indices >= sink_start_block) & (block_indices < sink_end_block)) exact = exact & valid approximate = valid & ~exact @@ -273,13 +239,8 @@ def _forward_ptr( safe_scores = tl.where(has_approximate, approximate_scores, 0.0) candidate_max = tl.maximum(row_max, tl.max(safe_scores, axis=1)) new_max = tl.where(has_approximate, candidate_max, row_max) - alpha = tl.math.exp2( - tl.where(has_approximate, row_max - new_max, 0.0) - ) - probability = tl.math.exp2( - safe_scores - - tl.where(has_approximate, new_max, 0.0)[:, None] - ) + alpha = tl.math.exp2(tl.where(has_approximate, row_max - new_max, 0.0)) + probability = tl.math.exp2(safe_scores - tl.where(has_approximate, new_max, 0.0)[:, None]) probability = tl.where( has_approximate & approximate[None, :], probability, @@ -311,17 +272,17 @@ def _forward_ptr( ) kv_tokens = block * BLOCK_SIZE + token_offsets kv_valid = kv_tokens < T - k_offsets = ( - ((batch * T + kv_tokens[:, None]).to(tl.int64) * H + head) - * D - + dims[None, :] - ) + k_offsets = ((batch * T + kv_tokens[:, None]).to(tl.int64) * H + head) * D + dims[None, :] k = tl.load( k_ptr + k_offsets, mask=kv_valid[:, None], other=0.0, ) - exact_scores = tl.dot(q, k.T).to(tl.float32) * scale_log2 + if FP8: + k_scale = tl.load(k_scale_ptr + (batch * NT + block) * H + head) + exact_scores = tl.dot(q, k.T, out_dtype=tl.float32) * (q_scale * k_scale * scale_log2) + else: + exact_scores = tl.dot(q, k.T).to(tl.float32) * scale_log2 exact_scores += tl.where( kv_valid[None, :], 0.0, @@ -329,33 +290,38 @@ def _forward_ptr( ) new_max = tl.maximum(row_max, tl.max(exact_scores, axis=1)) alpha = tl.math.exp2(row_max - new_max) - exact_probability = tl.math.exp2( - exact_scores - new_max[:, None] - ) + exact_probability = tl.math.exp2(exact_scores - new_max[:, None]) row_sum = row_sum * alpha + tl.sum( exact_probability, axis=1, ) - v_offsets = ( - ((batch * T + kv_tokens[:, None]).to(tl.int64) * H + head) - * D - + value_dims[None, :] - ) + v_offsets = ((batch * T + kv_tokens[:, None]).to(tl.int64) * H + head) * D + value_dims[None, :] v = tl.load( v_ptr + v_offsets, mask=kv_valid[:, None], other=0.0, ) - output = output * alpha[:, None] + tl.dot( - exact_probability.to(v.dtype), - v, - ) + if FP8: + v_scale = tl.load(v_scale_ptr + (batch * NT + block) * H + head) + probability_max = tl.maximum( + 1.0e-6, + tl.minimum(1.0, tl.max(exact_probability, axis=1)), + ) + probability_scale = probability_max / 448.0 + probability_fp8 = (exact_probability / probability_scale[:, None]).to(tl.float8e4nv) + output = output * alpha[:, None] + tl.dot( + probability_fp8, + v, + out_dtype=tl.float32, + ) * (probability_scale[:, None] * v_scale) + else: + output = output * alpha[:, None] + tl.dot( + exact_probability.to(v.dtype), + v, + ) row_max = new_max - output_offsets = ( - ((batch * T + q_tokens[:, None]).to(tl.int64) * H + head) * D - + value_dims[None, :] - ) + output_offsets = ((batch * T + q_tokens[:, None]).to(tl.int64) * H + head) * D + value_dims[None, :] tl.store( o_ptr + output_offsets, (output / row_sum[:, None]).to(tl.bfloat16), @@ -373,9 +339,18 @@ def sol_attn( thresh_type: str = "diag", sink_tokens: int = 0, sink_start: int | None = None, + q_scale: torch.Tensor | None = None, + k_scale: torch.Tensor | None = None, + v_scale: torch.Tensor | None = None, ) -> torch.Tensor: - """Run Triton Sol-Attn on contiguous BF16 BTHD inputs.""" + """Run Triton Sol-Attn on contiguous BF16 or block-scaled FP8 BTHD inputs.""" + fp8_inputs = q.dtype == torch.float8_e4m3fn + if fp8_inputs: + if k.dtype != q.dtype or v.dtype != q.dtype: + raise TypeError("FP8 Sol-Attn requires q, k, and v to share dtype") + if any(scale is None for scale in (q_scale, k_scale, v_scale)): + raise ValueError("FP8 Sol-Attn requires q_scale, k_scale, and v_scale") arch = _validate_inputs( q, k, @@ -386,8 +361,7 @@ def sol_attn( ) if arch[0] < 8: raise RuntimeError( - "Triton Sol-Attn requires an NVIDIA GPU with compute " - f"capability >= 8.0; got SM{arch[0]}{arch[1]}" + f"Triton Sol-Attn requires an NVIDIA GPU with compute capability >= 8.0; got SM{arch[0]}{arch[1]}" ) scale = q.shape[-1] ** -0.5 if scale is None else float(scale) tau = float(tau) @@ -400,7 +374,7 @@ def sol_attn( ) use_tma = _use_tma(q.device) - if use_tma: + if use_tma and not fp8_inputs: # Keep the original descriptor-backed preprocessing on TMA devices. # The pointer preprocessing below exists only for older architectures. from ..preprocess import prepare as prepare_tma @@ -446,13 +420,22 @@ def sol_attn( tau=tau, thresh_type=thresh_type, tokens=tokens, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, ) - output = torch.empty_like(v) - grid = lambda meta: (head_dim // meta["BV"], blocks, batch * heads) + output = torch.empty(v.shape, device=v.device, dtype=torch.bfloat16) + + def grid(meta): + return (head_dim // meta["BV"], blocks, batch * heads) + _forward_ptr[grid]( q, k, v, + q_scale if fp8_inputs else q.new_ones((1,), dtype=torch.float32), + k_scale if fp8_inputs else q.new_ones((1,), dtype=torch.float32), + v_scale if fp8_inputs else q.new_ones((1,), dtype=torch.float32), kc, vc, threshold, @@ -468,6 +451,7 @@ def sol_attn( NT=blocks, BLOCK_SIZE=BLOCK, GROUP_SIZE=GROUP, + FP8=fp8_inputs, ) return output diff --git a/telefuser/kernel/sol_attn/triton_ref/preprocess.py b/telefuser/kernel/sol_attn/triton_ref/preprocess.py index 6341736..06f3a8b 100644 --- a/telefuser/kernel/sol_attn/triton_ref/preprocess.py +++ b/telefuser/kernel/sol_attn/triton_ref/preprocess.py @@ -6,7 +6,6 @@ import triton import triton.language as tl - BLOCK_SIZE = 64 HEAD_DIM = 128 THRESHOLD_GROUP_SIZE = 64 @@ -25,14 +24,18 @@ def _reduce_kv_kernel( k, v, + k_scale, + v_scale, kc, vc, T, TP, NPAD, H: tl.constexpr, + N: tl.constexpr, D: tl.constexpr, BLOCK: tl.constexpr, + FP8: tl.constexpr, ): block, batch_head = tl.program_id(0), tl.program_id(1) batch, head = batch_head // H, batch_head % H @@ -45,6 +48,10 @@ def _reduce_kv_kernel( ) k_values = tl.load(k + offsets, mask=valid[:, None], other=0.0) v_values = tl.load(v + offsets, mask=valid[:, None], other=0.0) + if FP8: + scale_offset = (batch * N + block) * H + head + k_values = k_values.to(tl.float32) * tl.load(k_scale + scale_offset) + v_values = v_values.to(tl.float32) * tl.load(v_scale + scale_offset) block_len = tl.minimum(BLOCK, T - block * BLOCK).to(tl.float32) summary_offsets = ( ((batch * NPAD + block) * H + head) * D + dims @@ -95,6 +102,7 @@ def _reduce_kc_stats_kernel( @triton.jit def _diag_threshold_kernel( q, + q_scale, kc_mean, kc_var_diag, threshold, @@ -106,6 +114,7 @@ def _diag_threshold_kernel( D: tl.constexpr, BLOCK: tl.constexpr, TAU: tl.constexpr, + FP8: tl.constexpr, ): q_block, batch_head = tl.program_id(0), tl.program_id(1) batch, head = batch_head // H, batch_head % H @@ -117,6 +126,8 @@ def _diag_threshold_kernel( + dims[None, :] ) q_values = tl.load(q + offsets, mask=valid[:, None], other=0.0) + if FP8: + q_values = q_values.to(tl.float32) * tl.load(q_scale + (batch * N + q_block) * H + head) q_len = tl.minimum(BLOCK, T - q_block * BLOCK).to(tl.float32) q_centroid = tl.sum(q_values.to(tl.float32), axis=0) / q_len mean_kc = tl.load(kc_mean + batch_head * D + dims) @@ -137,6 +148,7 @@ def _diag_threshold_kernel( @triton.jit def _pool_query_kernel( q, + q_scale, q_bar, T, TP, @@ -144,6 +156,7 @@ def _pool_query_kernel( N: tl.constexpr, D: tl.constexpr, BLOCK: tl.constexpr, + FP8: tl.constexpr, ): q_block, batch_head = tl.program_id(0), tl.program_id(1) batch, head = batch_head // H, batch_head % H @@ -155,6 +168,8 @@ def _pool_query_kernel( + dims[None, :] ) values = tl.load(q + offsets, mask=valid[:, None], other=0.0) + if FP8: + values = values.to(tl.float32) * tl.load(q_scale + (batch * N + q_block) * H + head) q_len = tl.minimum(BLOCK, T - q_block * BLOCK).to(tl.float32) centroid = tl.sum(values.to(tl.float32), axis=0) / q_len tl.store(q_bar + (batch_head * N + q_block) * D + dims, centroid) @@ -215,6 +230,8 @@ def _reduce_kv( v: torch.Tensor, *, tokens: int | None = None, + k_scale: torch.Tensor | None = None, + v_scale: torch.Tensor | None = None, ) -> tuple[torch.Tensor, torch.Tensor]: batch, padded_tokens, heads, head_dim = k.shape tokens = padded_tokens if tokens is None else int(tokens) @@ -226,17 +243,23 @@ def _reduce_kv( dtype=torch.bfloat16, ) vc = torch.zeros_like(kc) + fp8_inputs = k.dtype == torch.float8_e4m3fn + dummy_scale = torch.ones((1,), device=k.device, dtype=torch.float32) _reduce_kv_kernel[(blocks, batch * heads)]( k, v, + k_scale if fp8_inputs else dummy_scale, + v_scale if fp8_inputs else dummy_scale, kc, vc, tokens, padded_tokens, padded_blocks, heads, + blocks, head_dim, BLOCK_SIZE, + FP8=fp8_inputs, ) return kc, vc @@ -248,6 +271,7 @@ def _compute_diag_threshold( tau: float, scale: float, tokens: int | None = None, + q_scale: torch.Tensor | None = None, ) -> torch.Tensor: batch, padded_tokens, heads, head_dim = q.shape tokens = padded_tokens if tokens is None else int(tokens) @@ -278,6 +302,7 @@ def _compute_diag_threshold( ) _diag_threshold_kernel[(blocks, batch_heads)]( q, + q_scale if q_scale is not None else torch.ones((1,), device=q.device, dtype=torch.float32), kc_mean, kc_var_diag, threshold, @@ -289,6 +314,7 @@ def _compute_diag_threshold( head_dim, BLOCK_SIZE, tau, + FP8=q.dtype == torch.float8_e4m3fn, num_warps=4, num_stages=2, ) @@ -302,6 +328,7 @@ def _compute_exact_threshold( tau: float, scale: float, tokens: int | None = None, + q_scale: torch.Tensor | None = None, ) -> torch.Tensor: batch, padded_tokens, heads, head_dim = q.shape tokens = padded_tokens if tokens is None else int(tokens) @@ -326,6 +353,7 @@ def _compute_exact_threshold( ) _pool_query_kernel[(blocks, batch_heads)]( q, + q_scale if q_scale is not None else torch.ones((1,), device=q.device, dtype=torch.float32), q_bar, tokens, padded_tokens, @@ -333,6 +361,7 @@ def _compute_exact_threshold( blocks, head_dim, BLOCK_SIZE, + FP8=q.dtype == torch.float8_e4m3fn, num_warps=4, num_stages=1, ) @@ -365,8 +394,11 @@ def prepare( scale: float, thresh_type: str = "diag", tokens: int | None = None, + q_scale: torch.Tensor | None = None, + k_scale: torch.Tensor | None = None, + v_scale: torch.Tensor | None = None, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: - kc, vc = _reduce_kv(k, v, tokens=tokens) + kc, vc = _reduce_kv(k, v, tokens=tokens, k_scale=k_scale, v_scale=v_scale) if thresh_type == "exact": threshold = _compute_exact_threshold( q, @@ -374,6 +406,7 @@ def prepare( tau=tau, scale=scale, tokens=tokens, + q_scale=q_scale, ) else: threshold = _compute_diag_threshold( @@ -382,6 +415,7 @@ def prepare( tau=tau, scale=scale, tokens=tokens, + q_scale=q_scale, ) return kc, vc, threshold diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index be6c3bd..3031fa4 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -37,6 +37,7 @@ from telefuser.offload.async_offload import AsyncOffloadManager from telefuser.ops.attention import MaskMap, SparseAttentionState from telefuser.ops.attention import attention as attn_func +from telefuser.ops.fp8_attention import quantize_fp8_per_block from telefuser.ops.normalization import LayerNorm, RMSNorm, fused_scale_shift, modulate from telefuser.ops.rotary import apply_rotary_emb from telefuser.utils.logging import logger @@ -146,10 +147,19 @@ def _prepare_sol_qkv( k: torch.Tensor, v: torch.Tensor, sparse_state: SparseAttentionState | None, - ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None]: + if ( + self._is_sol_active(sparse_state) + and sparse_state is not None + and sparse_state.config.sol_fp8 + ): + q, q_scale = quantize_fp8_per_block(q) + k, k_scale = quantize_fp8_per_block(k) + v, v_scale = quantize_fp8_per_block(v) + return q, k, v, (q_scale, k_scale, v_scale) if self._is_sol_active(sparse_state) and q.dtype != torch.bfloat16: - return q.to(torch.bfloat16), k.to(torch.bfloat16), v.to(torch.bfloat16) - return q, k, v + return q.to(torch.bfloat16), k.to(torch.bfloat16), v.to(torch.bfloat16), None + return q, k, v, None def async_usp_forward( self, @@ -226,7 +236,7 @@ def default_forward( q = rearrange(q, "b s (n d) -> b s n d", n=self.num_heads) k = rearrange(k, "b s (n d) -> b s n d", n=self.num_heads) v = rearrange(v, "b s (n d) -> b s n d", n=self.num_heads) - q, k, v = self._prepare_sol_qkv(q, k, v, sparse_state) + q, k, v, scales = self._prepare_sol_qkv(q, k, v, sparse_state) if sparse_state is not None and sparse_state.config.sparse_impl == "radial": seqlen = q.shape[2] q = rearrange(q, "b s n d -> (b s) n d", s=seqlen, n=self.num_heads) @@ -240,6 +250,9 @@ def default_forward( sparse_state=sparse_state, input_layout="BSND", output_layout="BSND", + q_scale=None if scales is None else scales[0], + k_scale=None if scales is None else scales[1], + v_scale=None if scales is None else scales[2], ) x = rearrange(x, "b s n d -> b s (n d)", n=self.num_heads) if projection_dtype != input_dtype: diff --git a/telefuser/ops/attention/attention_impl.py b/telefuser/ops/attention/attention_impl.py index ca322c4..d295b8e 100755 --- a/telefuser/ops/attention/attention_impl.py +++ b/telefuser/ops/attention/attention_impl.py @@ -187,6 +187,9 @@ def attention( return_lse: bool = False, sequence_lengths: list[int] | None = None, cu_seqlens: Tensor | None = None, + q_scale: Tensor | None = None, + k_scale: Tensor | None = None, + v_scale: Tensor | None = None, **kwargs: Any, ) -> Tensor | tuple[Tensor, Tensor]: """Unified attention function. @@ -403,6 +406,9 @@ def attention( # Sol-Attn elif attn_impl == AttnImplType.SOL_ATTN and SOL_ATTN_AVAILABLE and sol_attn is not None: + sparse_config = attention_config.sparse_config + if sparse_config is None: + raise RuntimeError("Sol-Attn requires sparse attention configuration") eligible = ( attn_mask is None and not is_causal @@ -411,13 +417,12 @@ def attention( and q.shape == k.shape == v.shape and q.ndim == 4 and q.shape[-1] == 128 - and q.dtype == torch.bfloat16 + and (q.dtype == torch.bfloat16 or (sparse_config.sol_fp8 and q.dtype == torch.float8_e4m3fn)) and q.is_cuda ) if eligible: - sparse_config = attention_config.sparse_config - if sparse_config is None: - raise RuntimeError("Sol-Attn requires sparse attention configuration") + if q.dtype == torch.float8_e4m3fn and any(scale is None for scale in (q_scale, k_scale, v_scale)): + raise ValueError("FP8 Sol-Attn requires q_scale, k_scale, and v_scale") try: output = sol_attn( q.contiguous(), @@ -427,12 +432,21 @@ def attention( tau=sparse_config.sol_tau, thresh_type=sparse_config.sol_threshold_type, kv_splits=_resolve_sol_kv_splits(q, sparse_config.sol_kv_splits), + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, ) except (RuntimeError, TypeError, ValueError) as error: msg = "Sol-Attn execution failed, falling back to TORCH_SDPA" if msg not in _warned_attn_fallback: _warned_attn_fallback.add(msg) logger.warning("%s: %s", msg, error) + if q.dtype == torch.float8_e4m3fn: + from telefuser.ops.fp8_attention import dequantize_fp8_per_block + + q = dequantize_fp8_per_block(q, q_scale, torch.bfloat16) + k = dequantize_fp8_per_block(k, k_scale, torch.bfloat16) + v = dequantize_fp8_per_block(v, v_scale, torch.bfloat16) # Fallback to SDPA if output is None: diff --git a/telefuser/ops/fp8_attention.py b/telefuser/ops/fp8_attention.py new file mode 100644 index 0000000..6777dfc --- /dev/null +++ b/telefuser/ops/fp8_attention.py @@ -0,0 +1,41 @@ +"""Block-scaled FP8 activation helpers for attention boundaries.""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + +FP8_ATTENTION_BLOCK_SIZE = 64 + + +def quantize_fp8_per_block( + x: torch.Tensor, + block_size: int = FP8_ATTENTION_BLOCK_SIZE, +) -> tuple[torch.Tensor, torch.Tensor]: + """Quantize a [B, T, H, D] tensor with one E4M3 scale per block/head.""" + if x.ndim != 4 or not x.is_floating_point(): + raise ValueError("FP8 attention quantization expects a floating-point [B, T, H, D] tensor") + batch, tokens, heads, head_dim = x.shape + blocks = (tokens + block_size - 1) // block_size + padded_tokens = blocks * block_size + padded = F.pad(x, (0, 0, 0, 0, 0, padded_tokens - tokens)) + blocked = padded.reshape(batch, blocks, block_size, heads, head_dim) + scale = blocked.detach().abs().amax(dim=(2, 4)).float().clamp_min(1e-6) / 448.0 + quantized = (blocked / scale.to(x.dtype)[:, :, None, :, None]).to(torch.float8_e4m3fn) + return quantized.reshape(batch, padded_tokens, heads, head_dim)[:, :tokens].contiguous(), scale + + +def dequantize_fp8_per_block( + x: torch.Tensor, + scale: torch.Tensor, + dtype: torch.dtype, + block_size: int = FP8_ATTENTION_BLOCK_SIZE, +) -> torch.Tensor: + """Restore block-scaled FP8 activations to ``dtype``.""" + if x.dtype != torch.float8_e4m3fn: + raise TypeError("expected torch.float8_e4m3fn activations") + token_scale = scale.repeat_interleave(block_size, dim=1)[:, : x.shape[1]] + return x.to(dtype) * token_scale.to(dtype).unsqueeze(-1) + + +__all__ = ["FP8_ATTENTION_BLOCK_SIZE", "dequantize_fp8_per_block", "quantize_fp8_per_block"] diff --git a/tests/unit/models/test_wan_video_sol_attention.py b/tests/unit/models/test_wan_video_sol_attention.py index 0510163..6e5a700 100644 --- a/tests/unit/models/test_wan_video_sol_attention.py +++ b/tests/unit/models/test_wan_video_sol_attention.py @@ -6,6 +6,7 @@ from telefuser.core.config import AttentionConfig, AttnImplType, SparseAttentionConfig from telefuser.models.wan_video_dit import SelfAttention, WanModel, precompute_freqs_cis_3d from telefuser.ops.attention import SparseAttentionState, attention_impl +from telefuser.ops.fp8_attention import dequantize_fp8_per_block, quantize_fp8_per_block def test_wan_model_enables_sol_attention_state() -> None: @@ -106,11 +107,13 @@ def test_wan_self_attention_casts_fp32_qkv_only_for_active_sol() -> None: q = torch.randn(1, 4, 1, 128) dense_qkv = module._prepare_sol_qkv(q, q, q, state) - assert all(tensor.dtype is torch.float32 for tensor in dense_qkv) + assert all(tensor.dtype is torch.float32 for tensor in dense_qkv[:3]) + assert dense_qkv[3] is None state.update(numeral_timestep=1) sol_qkv = module._prepare_sol_qkv(q, q, q, state) - assert all(tensor.dtype is torch.bfloat16 for tensor in sol_qkv) + assert all(tensor.dtype is torch.bfloat16 for tensor in sol_qkv[:3]) + assert sol_qkv[3] is None def test_wan_self_attention_casts_projection_input_once_under_autocast() -> None: @@ -128,6 +131,27 @@ def test_wan_self_attention_casts_projection_input_once_under_autocast() -> None assert sol_x.dtype is torch.bfloat16 +def test_wan_self_attention_quantizes_qkv_for_fp8_sol() -> None: + module = SelfAttention(dim=128, num_heads=1).to(torch.bfloat16) + config = SparseAttentionConfig(sparse_impl="sol", dense_timesteps=0, sol_fp8=True) + state = SparseAttentionState(config, mask_map=None) + captured = {} + + def fake_attention(q, k, v, **kwargs): + captured.update({"q": q, "k": k, "v": v, **kwargs}) + return q.to(torch.bfloat16) + + x = torch.randn(1, 65, 128, dtype=torch.bfloat16) + freqs = torch.zeros(65, 64, dtype=torch.bfloat16) + with patch("telefuser.models.wan_video_dit.attn_func", side_effect=fake_attention): + module.default_forward(x, freqs, freqs, sparse_state=state) + + assert captured["q"].dtype is torch.float8_e4m3fn + assert captured["q_scale"].shape == (1, 2, 1) + restored = dequantize_fp8_per_block(captured["q"], captured["q_scale"], torch.bfloat16) + assert torch.isfinite(restored).all() + + @pytest.mark.gpu def test_wan_self_attention_executes_sol_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): diff --git a/tests/unit/pipelines/wan_video/test_optimized_example.py b/tests/unit/pipelines/wan_video/test_optimized_example.py index c4858f6..0e531da 100644 --- a/tests/unit/pipelines/wan_video/test_optimized_example.py +++ b/tests/unit/pipelines/wan_video/test_optimized_example.py @@ -19,6 +19,7 @@ def test_wan_optimized_example_builds_compatible_configs() -> None: assert attention.attn_impl is AttnImplType.SOL_ATTN assert attention.sparse_config is not None assert attention.sparse_config.sol_tau == 1.0 + assert not attention.sparse_config.sol_fp8 assert quant.enabled assert quant.quant_type is QuantType.TORCHAO_FP8 assert quant.kernel_backend is QuantKernelBackend.TORCHAO @@ -31,6 +32,13 @@ def test_wan_optimized_example_builds_dense_attention_config() -> None: assert attention.sparse_config is None +def test_wan_optimized_example_builds_fp8_sol_config() -> None: + attention = make_attention_config("sol-fp8") + assert attention.attn_impl is AttnImplType.SOL_ATTN + assert attention.sparse_config is not None + assert attention.sparse_config.sol_fp8 + + def test_wan_optimized_example_rejects_unknown_attention() -> None: with pytest.raises(ValueError, match="attention must be"): make_attention_config("radial") From d3b8cdace9e1bc9b77960e6d907b5b70a9af063e Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Thu, 13 Aug 2026 09:12:23 +0000 Subject: [PATCH 05/15] bench(wan): update tf-kernel FP8 Sol-Attn results --- examples/wan_video/README.md | 14 ++++++-------- .../assets/wan21_fp8_sol_h100_benchmark.png | Bin 0 -> 50062 bytes telefuser/models/wan_video_dit.py | 6 +----- 3 files changed, 7 insertions(+), 13 deletions(-) create mode 100644 examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 875fdda..ced15de 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -188,8 +188,6 @@ Quantization choices: - `none`: BF16 DiT - `tf-kernel-fp8`: TeleFuser dynamic W8A8 FP8 GEMM -- `torchao-fp8`: TorchAO dynamic-activation/FP8-weight -- `bnb-nf4`: bitsandbytes NF4 Only DiT transformer-block Linear layers are quantized; the VAE and text encoder remain BF16. Select the two optimization axes independently: @@ -222,8 +220,8 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --kv-splits auto ``` -Replace `tf-kernel-fp8` with `torchao-fp8` or `bnb-nf4` without changing -the attention mode. `sol-fp8` quantizes post-RoPE Q/K/V per 64-token block and +`tf-kernel-fp8` is the FP8-GEMM implementation benchmarked below. `sol-fp8` +quantizes post-RoPE Q/K/V per 64-token block and runs exact Sol-Attn QK/PV GEMMs with FP8 inputs and FP32 accumulation; routing summaries remain BF16/FP32. The SOL tuning options apply to both `sol` modes. The final log reports generation time, frames per second, and peak allocated and @@ -242,10 +240,10 @@ same generation interval. | --- | --- | ---: | ---: | | BF16 | Dense | 0.854 | 16.147 | | BF16 | Sol-Attn | 1.083 | 17.023 | -| TorchAO FP8 | Dense | 0.674 | 17.601 | -| TorchAO FP8 | Sol-Attn | 0.836 | 18.193 | -| tf-kernel FP8 | Dense | 0.865 | 14.855 | -| tf-kernel FP8 | Sol-Attn | 1.167 | 15.730 | +| tf-kernel FP8 | Dense | 0.877 | 14.855 | +| tf-kernel FP8 | Sol-FP8 | 0.927 | 15.730 | + +![Wan FP8-GEMM Sol-Attn H100 benchmark](assets/wan21_fp8_sol_h100_benchmark.png) The benchmark prompt is: diff --git a/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png b/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png new file mode 100644 index 0000000000000000000000000000000000000000..5b4196b86ea45fc84e656c74dafd640716744ce2 GIT binary patch literal 50062 zcmeFZ_g7Qf7YFEjK1D$V8z=}oEP%AA2uK$ckluR}klv(s@(>V_rXV1_w@@OT5FpZ} z_ufk=p@bF!2}$Pgotb}OW_}pfUEC~I?!D)pv(Mh2{n?*QxSERGb!tZH3l}b2mw)$G ztSxzooXw{Z~E2ZU~x;^LRuQ}G*xj&NcH*8#=wXK}x@Q>6n zm*(fMoP|6$>2A7E?%d`0_uo&{-Sd8LrqnnsovGa+^^lEejmbDF^X-aR z2qYhTVPhOSW)(Xgufl6b+zMYR{idmtkbs`siv6yT=d~+Q=R@3$GW5bqUWp2iQdCgr z>gs#`{CRnKjguOx44N|j==1fXgO3oL=IZL|Ji?1Y=o1xko=3QB?amQt(XI$P$6!KJ zC#?goanE}2C%Nd9T$2ORC)LhKMuT5#;hO6vJ{B`>wMb~>Y&^WH1g!99dQ%&AcAeW|R z#(Zz^>I!r{EfV$Ikqi*r=x-!;brlICg!9?+1qH{;%KGQicds*z+D|ovn{FkH>%^b2 zdX=wI%cMj%-inJA=vHN!S8oavxnNTm6f`(FS*IOgC6y?x^@Jh#4fW-gh(x$e!at@@ z?_Cjq3D}PQ8Xn1{TiyzdAnP?Yz4!6-Q~`&Pf6Hd2R73)qOeoG-hdyg97Jcbc zl*;;}IPgbl;7{b2NC}U%mRL0OXm@I7J=Ao1diuwN=7qa&+2fu(DR}&(V7AV~dalDU zifN3piG)y-v%e=KNNlwZ(6UIBsi!B!$0x<#y?@sY(;J$RSaMJN#bWWmTEgxlmARpn zf&?DQ%0c`VBl^HW2E0SdNZXc>6LzxN^vcjLsYYR}^6m^$?tX?k^6cykHQ{{fPbP+z zk+w3Tw11(yD?D4r;Z39`kN7nkpK*(df*6tkCn`h5mKV4lerFN&kzyGNo3q6a^@+bl}b#REEfuSVN zR_&u+>J{4U@7H;Brc3wKVz~}b_bzZU2=LR@;PINKk>ODhkzdsz-@86cIL!`EV)o(`*0=570HCo(j zY1nO-_Clk^#jr?fw&|1VmoHyxN%DI>f7RAXLfH?LnV|9lzh`ylEpn3xM_RllHxPWf zoTJ=T_L`x8XXS;qXa3db=e{)zv(%KMBA$e_Xu-6^3*T%SotG`>AEC@ zesMxyvXJ8_a4J^HOjt?pqlpDF52`6KrQ(QxL*vh}F>zGUw`mf$LTl~oSHIB6DnhAr z9!16T>66$0m<2b8iFex{#%dVbKGo$+7Yjc*4r^yLh23BkeQv(-TceiNMtwB2>B+QC znc7>q&h5#7_d~&Ng1sj%t+p^7^Mw zhSd%_U!(gdXEU1Cd^UFo{Bf~is=5S+_2QY;djsch#lGD9gI_I;vPv=_cqnqaAM;m@eo%Qz~&F{qT9JFexpF5(tzJ`zXzP53+oT@V;8G7v&Dezr3)y{}q zjh)_5qs33d4#OCN;V`Ne#Q~Go8|clY?QK2vG`_1;e#n@$22t`@A}=qGRw&DEb!U@U ziCd>9GwcWg6CoFq!DErZ{Z5~Nu-dW0$$&d^C-^&CS#gEf z43+GgqA0_s?NY#6yQ^Ih{&V>|hx5?d20@6xqww5x_&{t1ebw$>EJ3emZ?CXI=!VvV zN8xQfBNiQ_@*N!=H1vELrXB=`Z(MX%>tg(w<&CI?)FjOxF#!$Y416a?8{s2YuTn_c zU=OQ*?FUY*U=h*$)LHeXsD1L=gN4caRDPo1eb$tjP%o_B`ucV(Y}IVMcwKHZTki&W zwQ699N`S>OBk=|K`P%4&42~%ErV$R*>3kK1eW+*$qcJmv&SQc2pDp$Njzk$Ptv7p5Ft5us$ zb}2m>^8Yb1u-0fI|NFK0&rj(q7cX9P-;aD&K*%aupH*?o#`s4{f?98jIE~e^Jy>2f2Gk&UvY-Sn26W! z%zM3Sd?p|;*VUC(sC=a9RHn6Z&G2_-VE3zn)s6QQjhHgzF&VG0JME?xztLUiGnJH* z;^SR6*NA_u9h+#nD)R!Y)Oq}K+gc|m!oF)$4Jl^r^dir!q7$G>W!^fvmYAA?3 z)WEaMA@BInB@E?snz89-`1GMj3!RTllYYd;!i;)*yG1^3ob3Jg7b8=D5?&^21ZJ!B zOw{`|t0IjXvpt{TQ7I>C0D56ey0780+LfVxN>X_mtu!kKg+gT?7#R5Rj{w@j!eV9U z(ccaZCGiopPIKC6hH*V;M#@uqzrW1DS9Cl*{m(`snsv!wJvJuhS;UsB3pbXjigT?$ zL}wiBu6nJfi?}V>nuBGIh>BS7+S}NB?wnwrnQzQ%h?9Br=#ifvGFDJyWvzDx62txOz7PbNnmSupXoI`u3~xcvQz`@PXZiyq&Q`Yot`oZ&@lJK% zTMzoG>h0h+Wu%5xe82c*%3jY4W@{F%_Mm#Vfo4x*qH@yyQhq5C1k#^BQXeMv#A#|^ z=b&|z#qufEKyt?ne-BB1ti7${`h8*Y7B&#!hL-{eL|WQ>v8xi^B)k&`LEV;a>u9G9 zIE_NELOnJQ#(w-5oi(!c{xAyeOSlv{gCPFL2bV8ez;!oWXW5^&HVuHHr>~=_w8|M= z&}#}H94d-mPfkiABbFJWX3Di(ro9f)H-pEZhKP^*2dWX!i14Vmt`BBQ-`yk#8l

~I*yw$w`HQOz ztzhP4mETkc-ddwKGa(~`Ld>94?aZ#nL7GIpPrSW9X!*+K8}X?ssDO8DCMeHzs`}Q- zEi6qq&+?(78O*n$8!-`4jN{I;g@Gt*j!Is;?xgj6e^lXxLJd2Ie;pQ%lNO2(4i6zD ziheT{Eo)Jr$2p8eAd)a)8VKo|uDAs$Mj@xUp`Ty-P^wbn3NdbQInY!pCjlG+2Qrk;bTO;k%i(o~IMpin&TFs{h#*iDUVBKb?Qvdp#;`m9x;_9K$q(X6wc0vnTY ze$BMbCa?Vb2PRB=D{E>r#;(!C7}dI@BmmguLiAVLn{jUP-9HKn6N30t`mAMTDc8|` z0|S&T?5$ZQJ4&+!Bg(Q#IIv_lMfa;PYd}$QfC|a9dC3RO`BYdYr>*SwgcR~`yt$5} zb_d8Va9ln@4u{h4|1O3_7r~iFQC`h{Iizym#*T7)f7`?-6=rXDQ5!J42C-RfG!oBD za2#8h7F8R*=&|#;6h%4h^$^AR2PE(kj_n4^JH7bQCF-|wgt7GgL)|C2$tepBZdj>} z7PutBZ%1jc=;BDFD|Jr)i`M@9o7yVch=@SSYFB4)%EL%f#T2_?j_3xWp^%Nna-XabpbcQgzGj`o^jtAETxR% zC5(_}#T~0lm;|^{&iA(b#>0;G5h|9s8`H=gR>^89R#(JX+9s=wlCFtKKh-7sDzQSS zGuvB$AeouBwL?M9G2>;YLQ5nGWynZ0+&Wb|%-1&@FV#xmrtpH`uq!~b^Wij5{V%n# z(aG1G>5Z$Zh3MZAc3%H#&FiWK_!rqdOi~<}PZ&fgbK{Dym~0H24b-5l2Y${XERdIR z7!i?4Ci%x!R{8TLEGwN+zQn$X(OU7qgOcO2CNc9r9EVnjF;4-0erEm`@0Yg*-qtDF z>5AETop$5&-6S<3n;^WA1N`Wy&WEfLHv{Vf!`N!5>X?y6mroE68h{xTrA`&JL_g%Fy zgFy0`fk4K~rTFY|S=sjOX}^tD)h4HzB76xVFtVS?fA1#Pvc zG%eXHL-Px&B)4Ng zBQ6_L3j?qhmE!dy(aLgt^^V9gHgsRXGwl_smlDQF^1a_3r6$4E&dur%HqsF;UI!9krz6Q7;A|6^Q0Mfz?Gu10My$>RN$skEgPL$S22y38cxDp*&4wHI$b>_FV|w+M3n51-#2u z*u0-)(SgbdtZfAO0i0xC^d_oKaB{fCd_Tq%SXZ&a!NI|N2EKjG5n1g3{5EuH8LbX) z5Wtc)HAkbr_04$NpK<07XEtH?FB)p6*q4+p%D++QnC^DfdkxjzKC{3P2IW+>>^oza zv@yaeTxFIX;)JNd^f3;VbnRb*Ewg9jD7PoR*#m0>WyUk71v1=KN+d2_PLT33dtl!3dV2Pg zg9mQl)rfTW#C{DVdr*|EN&g=02&{rI`d{?(h` zDS_(6rtq?Z?3(~I>hO}8AwRw+0b;R{Dl0}n7H?aX@bkynL9sI}>){#nY{Jn1FiFfd5E zD12wW^NENWM8faW+)#7Vq;OIk2XVl2=NBPi?M(dWWA^}mVZxsqE4Mu%A`|mJ15=v5hs~4 zI+FSQ?Tf9s+JUr5;@(>5?J&=V7i*}essK7M_@hpG5fM)UOmN@570|DiG!fpUc*?K{|kQXJXCLuPjcn=k_Dr!jt7&=J)CaTT*?upO0d zt-NJ1X+sSW#!EDQm4SM3+ z%My0luCB1F+~bH>aGpbW;RvOQ&AO_o5(UrSO1@g14j$~p1yGpTGxx7GaUkRNV z2k=^#*Mzz$TLLCUdf0kQ4pdB_Wcm9)Klv*Aufte&5+OYFVpxo=KjguSUjsjCIYx2& zIx0bX1BB8;i7}k65AN!CygO^vBS6Mw;6pnooRFA%2Xhk{8?I|o8MeJ@93fx6ezhfU z-~0JV^U6ynEtHWUga*(FfayqwcsfpInuIx#w9%)hr-0N1+=Td;xxAr@iu_PT4G)!F zQT)FAllDfM$fyj`H6DTt_kvd=T}$a}_MuSe??St{r-l>ogX06vCeJao?EmhtQnXg0 zMh@G{7(Gld`=5OO9j%i>@J$n{U-0v5XIwo76gqIEnA-`_j1qbt9VISXWd`|}(tqJ7 zf@||+et?=+UTsOP#A6NgJShyYRaz1z)n^4wjV3(s0!>h=u$`;$0Z5jkK|}-~`1t}; z`&3nAzDpiPg8L2f}-Tf-Y*4EymtCYd)ll^>JdF6J4OAbLX&>)j6 zT6+u`M?V3){W)&#Nl5$>qlqeIeY>@>Rx>r}ZEIK!7xqCw&N7WCPWK}h zV`q!XWBS|`mpU+k8ndDVT=mH)XLzf6ClnK7Aw_n)idEmuBM z$X@rrDtn}5#mfFW`m>|nXQYEMC!LynVz6tzvK$uU^G5tSO$^Q6i#PvXD=HBrb`epL{L#ht&TqB0q+VRm&F|^}LdLzj6ei}D zmKC38nd6!^;kDHt(5}n1sJFPex@;A4`M+C5OF_SEM`UG;I84kMR<&t*JO$k1>_I!K9TnVA$Fq=#)z^2hcCK zs1ndy2?}`=&htK`o%@)-2LAl9*)B9g*ZCAZg|ms#dlPWARyHLE$7Jje^%$qLG3zE! zuIuj_k?CLbLQh9sb#By+_L}l7H?)qL@rM%7Exjc?QFfb~_ZKJ)N?KI$#4^AFkn1HA zVWuMYm~V@MdEEQb!f%GN1SY#eU%{HnUV_9(6^ot-^Rf+;s}Zu9V{dt(X4U7ecx;PQ_J@esgY zth0j})cO+?(Q8xGyUji#p-jvLW2EKopyOGi#TE56dJYhnGb8JT6GG3PsYz=2x^I<8 zNWYSlgtoH|%ti=+^3RG&E+`3bcG9eanx1hzF&_|m#(uPRckHfhDyi_jc#bi-{U5n- z>5XjE<6EflKAN=PvNTkNth^r-C5!G7dn(+<&<}rq4gWIoiB1-Ti`HaCc*W1XXr2(&)J(%wOb)+W0wn(}Kg%7^U$@O54jq_EOgL ztMIXZPo>b^m(QOgJooE-Wwj@q|LEcDf7b3!5($+Y9^!LzEFlsAZ#D+H9a=@r*&!7k zC!>g4|6Au%EG3&A={zA@X=LGFl#u*}3Xebfs0q<;aM0em0TFN*^64P5D&`G;k`&9a zBM=U0J%k$vk=C{MH979`Kvve!_HBbfjB>=>tMEeAweN9p$fH^m;Tq#ym2#~})Xb>| zM!@P!RFaphoOHQo4!lIu>2_KKR?`2d|A^M$1f^Htf>eRrARIwyXrhN;6|fd$6DU8h z_CR$q3{8`++E4nl)Gqi}10q;N=f=HJ-IZzMxul;zOZ2N>Wj1uW=1{x-swFP32s4Jp zaYV6-tS0#&=plwdlQ@bv@l}9Jn@wJvE=m9?qmQ z-}`O7tLBTF?hDIO9aS{h9+wH&2LO~ANxSDXDJ$%BMu?ipD+HC#jW_@DJYEAxTEJnk zSbbx4H8XI+ApTQru8@GhseL=-5>`Lg;k5P%jbSxsA04|GbbMTgi9&V+=txHSr`6-{ zN~zCw@p+Oqv#2rvB_Oi0vd*YeOw0nktHBEKQbyn*`m8%ynH&+bes(~j*maObivJ+V zp?u6bXZ30Uvj%-kq97Zs2<$t!`G(3aqhk=bwCB#(i<0hw?pqzFB&$+|J| z`To6oRXF?8!xl*K3W;Esz6)`Ot#9Qh_5q6Bac)npnX+*G+BJI0-WZ@6Y*8|r0Sh8X zLHO=aV?=_;6OZk-`%#9bb`&%7Kffgyox@58(%+{Z9B(nA^zO>wlP?^}U1xp46|!eP zH~a44P@SnADDhl!4tx?~gT>Pg9$F^ue)Lw)JETob16kt+ z$`%|Af*q}m$C6t#N$yPAaq)R*Abv=GWj5PCN7bK7`!zfq)YMBX?N@e9kcNrTtrst! zjN^*x`V>3gJ$zpK%**{3T(?#w?AY=0@s{>plN}d4n)$q>C{Co!kYI3MS=$LIkWadw zWMySlQ!`#uBd8J`t(@}GVX8qM>;dj&!;8S#iWV z7*q?xzedZ;%lCEn*E|!dt*JRaaCZi(64PROK*hPc^6alR7XI6vB*rRv74`Ayj2 zf8*-vD2OHY_S_5%>mXGG6&iC;N9iWf(=ZA}R8;-im@=N>$TD=Cs0rj9D_EOuvZqj0 z=EIu001w1ETWvOm@#s`d4>)?o!_Qw_c&g|D2fO0l>br=BoBZcuq|D33r*CBcI{z5@ zknUU@^SO0Rfuh4^Yd)jsFJHU*;_F|Dy6+5ns%5YM z#wc%VV3fA$o#Mc~(b~>jUAfIk)%Zso{ky{MVqEO~Ewi&IkHZO*8-) zmx?*R5=#3Pdgv1f`kt&}HJA)uw|wR8>GD!vAH5GpM+nnn%)AdFSJwlZRX>Ag2|Pj5 z!esdQH6gUMf#Rkcr_ypbJB_66y?uUb$mb9_S?l%H>4^##7uW64dAMPLr%nR>6W@+X zhBr-yM??;+D$aSX!9Ac%Z5pq&g4Q^-SrcWRuF-H_o$PW0HxRoT)7{%$Q(oi#t9Yz1 z$hlT!hjF}EbK?->J(;P;$XX-9;w(VMK`@%-{*r!wUGU9jr)nwLPMyjKWXkwfX!JwD zvg@hsgwwnazI4I_GcJE6OKnw|w0Njd^}ru3Ys@+TS!=Slw=j-oE~EY$Ftj^MOFAH$ zYU3=HZ<#OOQc0sFlOt#WGyWsgUU2u7_jVPmKW?v(V@bWB-ss_39FGI{N${KM&tZ1^s&UI1>~nQ)k~|OXx|vqE>IVjt5!<%@(#wgG zJlA6)b>bfvB;p9?D3n!C0gt~e|j;qP0D?W!O~$2$`8N0J1ykPP2BJZ5GyU^Sy|3-nr*@!g)TOR zUDRIqRm#IlY#c3B3=?FjHU`WOSb$$gG1X?@hdNb_Hx`Nw!wGI|Cw6;dDg5)kJIHo! z#ZgtZwvo0~Wzpl5pZ8Gs<1J7>f2W|JPw1?-uibT8@Y{(0{zhT+dQ`^JSo})mkHr7T zjq~?}jIMl-P4+xmMWBRsW^2|5_kseHlp-x(oKgby7;oOy?Ci17dX37D!WbIrvlryH zKM@o6d!EZ4U^yC~stwVV(nwNDy4)x}CwF~wqDno$kf}Jbx1>P6%ZpV^#D)Fcy?ZvB z!3nLG>p54#9F+q|5d^I}Xu=fDg+W#UVLAoN7i?|6=OF}w_f|0hWl5;yF`zfL1_6YW~Kz+ zO@kOCT=pi5203mfCFxfirx9`fQY6&sZR<}z3}rf>Y|nZ#uR8JK=**Gx z+*zNlY#{1`w!@hwTL>T_u*IwyAu{BnF)`X&C^7)NVIH)EN!;Q0_ z1s>_{;(BL*t0qJG3%^t|P?ya9X*L$Ax+z^(|0-Z_H&-ZsfJTg2D_XsDIbFshuP>0E{VVgE(xFXuPWuaujJ#y0Ys-0}JW z!y3CgoJEu5fQDkuC{^y#k?!>z-j(l0_2Oe6w^tE+e>P-sTY&=5eArDxCVOz0vcmsr ztlyec*|dRPq9$+Nz7+}# zc#&JEY_6fw5eJtRdg42Dhg!x+dO4q&7(%x*Q#CqVf+#AYYiaG1xx)7SUXvuQ@BU#E zA+QUEMWx!CJ3L^y-*}MGr_40YBz}~{V2xf#!T9>&e9C)T<;?On)`JJ1vAyMhN@|A# z`lGVrI-e;rDLKI;+wF!FAOzEpKhn}xmbWLX-^`c=`xY-t2}>wuMW;1o)_>;WV&Cnw zgig;)3EIzQ=I-3Pa*?eKkx9qa7E<1-O`4FAr-hlvXv7D1suW-T>$8k3l(NN@WHg>* z->D&1^m%xv=(6R42`hhzdz6i^gS&dR?!S-tw^|{A@7^g9YS*O_bIi@}QinbZTzk`$ z%lb+rFgm_)0PXRPp?KMZVji<+&TVtn*l$$#;4ZW79#;EXf@-^T(YgCoD@=ZHCGEjB zDqV65j+?RSB`WBfJ6Z}`P5xH@i=;TW_Cu(n@wsPPp^){_&Xl%*w8klVZML2JcE=6u zJR%K9I@sEBIBpt?R_!blrzD1@ynTqE?Cn>e#+_%}Y~E4Q*Q_YU2K4Ty)1yM29(L;m zYMeUc##7*xeJI3okBEGDG^4q*wcpd#4q8RZq}NFZhB%e}CI~FG^9*bXY+EsLyY5LY(4imJOh00b)0h_mwgk|p z<34+Gz&NJj%49d$jGt@u<-C>y(t^EAzI!WTA%RWV-Ltp>@*F zdx_Y}JD8)>MR|A>W;zQ>V9ZTNAwMtY1Y6P3{ZlM%P+5%S(I}SJ3XKG7Sjm{l#J|dhg!7R57o8urOpjWVK>g z4YE0>#K)iEs(=ZPEkW6SmsDx1{b zx&`Q8Lo(qeL5_Ymo;+r+wwpfzZnOTCu$b?y@}n3J_2(`)jJIu-em3ocH*ek;OPo6f zX4*|@Nr#zQ+als)*}HfGgI@@Pq0}kfZmMdiO;-+py3Z0fAV=6dDPn=S)TMKz!&%5q zCC&EsgS$iOv|3Wh_V!-k&+XLISe3lO2$MNcTINB;YQ2e0xBjDWSi4Ri6Y9C=$)^k` zl9iBaIU)-?U?Z7eMskV|_O~p;(wgWTY|RAyy8i)!A-1;@(T=n%lBE*d9bJ_SK*`Xb zApnHpccgzN|NQ3&x+(9ynv~fCirb$*e+F8q8o}{y_^*6+_CVegLbZcIHuKZXh=kNw zdcxxdfcH8=Ni}JSiTEDp;*7U%c~4G~;O5LEL%d(n3~(dTtq)rs=By7SZ*ROWIL;MT z)GhV%JG1cq|5dF-Q!s6d=sFRQ!g7PKP3~##FDGjZ^@i8c>eR12&YcvS#}C-r_M`%ol$Ap0rly1e z--Ky4WOyQ8cy0s&ejZ!7{`W)af{DkG;fi2qwCx~i4F#NyIvye1{S0g#T{eK+;n6A& zi{eu_qW;IY$;#34bSbhnUh}?yt)BBPGA_T5q>*DFFZt2^>;boWhN8>6L@gN~K#$%Pv}gVYC&N8{{tfc> zQqAr14eq4E^@xI9lET2Jop0c6WqpR(ILQW*CW34clY-X4CvCcKPiAss7szM z+BkEB6zWbGo}!hyx|(;_C^4e9q~E^ne3w^|1y_DL8FUn>WXQB7zhX*u8p2}xiKfL? z%9yczyQvcOa21WslWY&{a1#hl8bv(pe1XLIR^p8eI_WzNmu0!)d5tO#`bmRuiDdnW z8BL2F5ZvorSB~e1StQ@4%;(ETa1TSK@#37oa-*jNTRXdKHO@QKp}q#1pI=(O9|;W& zMGyqy(q?M+Kf=WWb`zx%;ZITe3CsFF{(1zFRLv2CKvb`ooj9Bg)-Co;bxGlSf_jyd z8^z`b#~ZL@o6gx;9&T>KNAY)ybE&k;<%;aeL14hx8w%JrVE3+~ZzSx$8xK&eR^rNW zcq}Pt8h>@m*N>~9`5MA=>o8RDt6k+}t1QD4-%HE4(ghv0OasgfA+$nBgJitK(@m=A z!5z1fw*h=TY^6D014$9*5t)vOQe~S%=zdway0dArhs`USRF(0VKHq<8x^Wkm%H!5U z=qd{uvczy7%+19d-eaj^TKX>=2PXe436y{t=^2W<{p|?muvo|eqNmL6XT_&^ySmb@ z8${W29V6rW;KlWWD9nCou zVP$TB$9SW6t7%0{9`e$CEHlKhdeI~ND+Rn z3Ye_l{L){VSJLmB)U7Y>i&Mp@`g^P{Hi>&|76cpfvb}Y8_(XVnW2k(8nS!EDjIIq! zK_uyerV-cjI|JrsPJAJ1)tRx$$$TNdem&ELy!1J4C=Pl?cPhjBkMxz&tDT)ABml&` zSj2_{Jn7ZH2E*!ds-)A8^z&vkug|ADO*qdXiHqNTqq!_K2UrB_IXAg80RfJT4P4wf z5!T6;NL>ELhjjhlj4R^k2M=Cs926CD@f=rz9NGgwo_uSoyBf(EkdSu+y*&8Wg-d_J zE2|wG=Q9Xa#;mF|Zv1cFv_`N^j1^dOGS^P`3nPZ_N(L68?aRZ%U1oIprfclj$ah9Sw@8#yfSEsGx*^gJ$nbI@7zCB0E z=tBxREC@NZp$;RuKeSt4HYVsVOb|gtbxV{s@uc8eT8<)nQ@nOXg(nly!%e^bY;31K zU}f?`n?Fr&A8D+iTON*2!7MC=(1BP-&DOwmKx0(kSqh)4NBWX#;A3m^qcZAL@dy6- z2ebKK`Q(a*t5dv2;S!C$htYH`Aj2od>1?FKwr2aWgi=3XMifj=QfTy-@w5)JoJVt} ze=%@$hQ+j2RJdtzTz!;n9;-S0LsIXD!BdIO?gL$Jovh9jzHU4Ya(o3+^QpLU1z;g5KagJOP)HY&<(6Y6~^pO2eUKw%4fh5y$!S7@? zj~fKkfpwI8cveGm+2AfH+D0BSB^kCU5QltV}eq^>jR zZz7-+!z*lYU6Lou#M{=8j*cgP&grZm z;kFwlaq6^>RgH^f8Y4O%M^nY5_n!J^(Jdnf*6!dmAS4UNvyrwb&G~!6xu)yKj)K@( zIAh`3_UbRcIcE$dkgxzMikVH~o^Yb_{cQKs6;G!H*@xd9raTsVLwhumc)(DXmg5l` zws^iLKTVg;*Li0W|4Qry2ZxXVFKN4-4*dhab7t^^ppTird3ty@cpml-Yui7{vB=ZV z2U&Q28WGQY6_DmS)kZgcJ-1djO~5|ybdwHk?1x**T8-DD&RFR0r^TiCZOylS61uBZ01RPp z=X~E?7dXAP^ocwio5^k`sq4b+bpCB8Vzeh-_t{#+84~{n3)}fl*V5Le z=Mg#|V#}Iu`N1X$iJD~K6@KCf?1gcmfVa1A-rq9U&IROb-S89VCSq?7K>CL0f9`^? zCmtZ-z4{F#NP>5yl~P{qN^PvLCXx{+GpazA;NimzmRm%45=8%6vzhr&MKUp8d#2rX z@;7CT{Pg&eyu7lg>+*Jesoy9K(+T0)_F?w!Kz1{b2x8iFo6g7+Z-T6UwEosY$8j{2 ztZ=J|02r9jAc^b2LIWPKoX%x02#g=^OjR$kPt$TOh|Ggc0V1TDi5#o}l6st@WgJ$+ ztLr>jCc(K6fFDQ$6)h`Xuyo~3jz}||I`mgTpn8N?Vh6-0#V^MNoo8Qrxa5B}r%f@( zXHViQJ}(6mQboL;u1@n>I^C2$@kSu)U)d^oX=ywmJm9v?P|O1B_+G%+1iBOU?Kt4+ z|F-o3)?;UUZj&Op9}|;^>`>NM$eJ85xp09R))h23H5AxYo8dH#pb$!tS5@O4-wKaO zB>rbI=h484itM<-YGY&baal>t)h-#hhfE3aEpIq;}wqqmEjG&j}D zZ-r+SS8Ar{^br6lamCEsp<(Zk_6Cb+0Fm6Xwb+Nmj|}tk^!9BL3D3K3?XSZnMf}ftry|&vz7Gi_G(-)VbQuczch0B->}@0q!3AqdDRoX;yMH!b zol;C63Eb{bwZ-~zpiCOuC`c5V7^Y2wJ^`~uk&Z}`Klv zn6g&(KzZyj;tsX=JT=!baN~5tO_2X#%3l>#C&!g|iV;#^mEd^s;thOyC)j+-3QP^s~-<9PDDx?$za=~ zMDgoToaSpvBH+}rA5O!^?K;^0x$qiM*kyesYpEMIgWDpw?@ZQIh-zr>ebUktSfSP3 z^*c{!c1Q-Jc--@sFB^a(`zpW`N{4=BRLNd+KaB3e1@tHWkPBQ=`u%j-YW!;fxL0Bt zjk#{3C1-08s8Aq4|V=CbG<&_6MdRAe_5p#c<1Gtfzpez7exyE*0 z6xb+A_Ggto|MVaIKmR)qYX1M{^ZEFHV;+n{LGu9t=fxLlRJt8~K8n|;&tTg(Hn{k? zw~W;GLL(9%<(TJc6vf58De!b89%Ia`tY9X*d!thq<)S;=+L8=^6002~tUml#8l_ww z)$^g^HYlXk-MPhbKR!O*(b3Y@S+`h9e4(FtW(;4dxxo(Fr_cPGLAk-=e{C&R%8b|V z4|YyS&NKBgKtt6%w&gV6g!H3iC5F*Hh3O|Lzn3Gd1|4bV|EwT04G`!tG!l>@ z_I4|3aV^Kc&7``&H-V}95s?+sdd@Hsin#O)^#z?q7csaLTiZ?0{2eOjx}@=i&KwAJ zU@!GVToGb|5=(>SAjtx)dq47?p1=D*m~6fpuo27@O(N*vG)jFSTH`EQ2gSZv3*{=V zzuW>Ej)9_CSX9_^bJ9dVHZ}EO$VU$s4fq;nnJ45O(715X$-b2V(qo|#Y%!P5EtdVI z0#I!2jJ$U5MW(a*+@#;$AHc|KMey~j#BoGBs}0l^63(9t`|>GC8V!224x~P~_a-p= zl^LNsY<<44N;-O(fkEs|@57MEd_Kbl09#X69-xq>d@0-6N%^W0rp z{5BV4onJuUDNMl4trGA|XJ`J}>___tvvmX^wzg`f+NM*N@ZVoSAdgdHP!0ZQgQ#mEW7{1^ zwws%#fq2c09FB~@SIDxML;{y6(m?vlWZgfwmdQGAwvga7WA*F4^+;f?L|WGPa*JDC zD0r?h?@XL^e`2}7`5fPUtC-LEqu~EaVF_*UNCsqI-sdrEf9&d!4VxC6WE;NGu& zhSHx>-@SQ1EiKN%&^^C!9H#v5wScqLI129_Y5(WYXjj<1>&#~}fV4{UqfJG5>$AtT zw{!3q)V$E;+8LHVq*nKsLwX*3EDit?qfS$QKMBrP!cVlb{($0||2@LV=+wV#Awh$i z2hyPrRZssFelpfPuOe)C6FmGsSk9ypTgYUc;{@PaS{^Cu6W*Vn(gj+szY ztOOmrZq~}g-Ss@Po8Kc6;KO<|x$40{@iAP1=-|mw$dUVJ+3jAfgriXSf3>P# z0nZ;o=Mm6A?@#_t_fTBiV)zKZbvk|uTxd5n*s^SI?d%K|iz-uUvFdO&c$sR32}Eb3 zNh`tb_7#uk58r2kWeb+O2pm*jU!P&EgP_%_nj#I&j-jxyi2vbf!*5N{q5Y=i(L;co zczJmP0vb#=hRcId8PT`D!C-wUBq#9yD!S`A&WFX-I{E#LacUah<*>H44uAp2JUuoS zu|kKo0o%0r~s?L>N#1zjDs$ z-o0=^OPAI{gZA^=2TivEAMtcmC$yhp!!|3%^7(0XS&?{7!M^F)Iyo9V?dNLGs^^-e z%1Q(JE_>tUTk1pMN#peF*Jte-=L+N}@~gG)9vmF1&PUF>vLE^CnVTC#N2JMge&&lK z3LLsta8b^;sQ_~j-35g=-xC+MM#P`YVI~5E0%v`cPhUMZE=N1q?6ro>9TN4Em}uBu zR9OCH${MNEtETfuzE9;{pM)RB8gidm&SPJj{&NU>L-{MUvG;oZzIy(IbVak7b)4;@ zPI06t@KHzC&mU4`;y5EwDo(6AP9+nQjFPe^3D4#A{}1-wI;_fWT_42+K@g<`1Vp7v zS{kPc(hbr|H_}}KDj*=;E!{0G(%oGu-O>%`LBH?YYpuQ3S?71{bI$r_&ud;HOn76w zV?5)D`@SEay~&5np~pW3Qp1$&>r=Ek&L|dEVuo?vnRH<2k$F6_CF5pdOwuNLm#d~f z^`(Q2PNZb9y^h3M(G?%*YJJz)TSJ$j5;tLdb%ENZyn+Q* z(&{`34GpEt>@p7l>sA3fW8kCFaIw{?_S#Weg+ujGBh?|Z#W7WfqSrRr+OG^~)7N#l zboC5LpX&H!RocM9?RNwh)mA)*&yLz-VR=58I=_&~c}N&SMY-^eqa~o#_rY6%KwVzj4zUNVa;**B!~M;v_!-6e zioM8;>G(?T+{r)9je9>2Z1t&m5BJ7@>DM1JGBXC!Y|eOd?}&!e$x$j?BlT1DiQXDI zuttw|i5sk{jxx3nwQTZinli2k3=F6+*%et5yJDG!)tw;jCwdb%uKGv!YK{LC7TD0n zvx7B=8`5?k!Bgt;%8V(){2}C6ZvQ>#GYjccV=gq(`q07z#C!G0mEQ5O$@UjNv z^fUUp z-m>-sMwRmQx$19uDiQZ+Nu3VYQwx(f6Nr{5X;^lPun?7dO%a5zgvywnAq;lc-SWNp17E|pJ zov(XhSdy`E`0h%N#v(zVXI)68JfzIbe{PNBx}mn_t3;LMwU>6^3f*efD$LeQjAS)Z z0_yYf1+x{@jUIiD@z3z|7uY{2A0@ti%>_G*JY=?v>xH%c%Ax!eIWdNONTCRLbDYeQ z9aer!OZ`bTm_NohP8tT!?QT5LR8HKDKWs41v%9}+@n2sXJ~1IlUQF+3Y~^(#V=l!0L#AnVFDD`j>^;u;!(K0X!> zXKbc59Fu)@RovDDu!3K}3hoX;BrM=P%)lnJUhIF|HmVK8*tXmnDiR8Eq0-fHi-XqU zRl-5gN}xt_+1Fldq#Kjl0S>(F`e|L&(G63@z0|TYn98li%RwofvT`VB{_q~Q`T29M z*ky1pTiF}q5)u+m?MEr3?g%$GfDhV`iw{@7@(oKBtleQwm8f<(Nj$8P;B{0-VKlJi zmlA8NZ=_S*co&hr$S}q5*=(`B$7zhKWY6UmZuMqvjhvN{aTYLuPP<6jx`_E+&mMc> zh&Y>Do0p$Gv^-jK0lv@!ffM^ZmHe61KEb_-&z3r8eqtv_yRGf*hq@JgUOl(>#(xy++CGFG)oWXi|vnP6Q^5~`WQ?1~FPoTBrzOw!i1{~<(uLEUG6#+`=IuSR&NX+NYmq|TJhnO!(f+C4T%*}i= zGmKs9E9_D;v)TKazNzny;x9N2wArlkJe*JJ3ibyN5zlcWe>3YojDi%beM! zG&l#|&H)0gwWWoT310SdeZ%AJqGcH`ot~*pjvvHLx%_;+8dnn$5s5pn(wFL0x;_~u zoD4cz$rJE^CAV(EJtB-I@&&0w{0MO4t3a{VxhMq7gp!g`t|Gm0W<*VppdGls`R4;1qP`*X>NjdJ9v7aTuAQgZP4A`U-bCI z9-M$%hk9(vS;eIZhXoOE2mB0UNZy}?gCyS`@= zg1q#+q(AEycTbJEdB)L7K1K)jo_bCe?5-Qw;VK^4P>AoZ>)^C0oQ4EPM+ZlbS9LL8 zD|DGstD!wsLJ9cEva!c^c51f4v{NiEAG$SHsgPI&Px0{W`FKYgPnO68gs~gg29%QY z$6gIl>KsizV(yHA0sy{D6uY5@wzj^8#*gV$*0tH24kbCJy=e(+zF_uvv_9SsQLrX4 zcrVU#h(_8;G^IX5bU`v%{{yzK8jqS&&*~2Al8?Va^(OZrg69R zmWK<~h|J`aOY)E@q})uCE=ZiwK8;FqY1qiB~2sq*|Vm~ z%|&%NidWx76pkTvw7VPSDbbXxStzVdzdt!V{GzE9@t!`0b3qNiJKh1qv2qh< zr&Z}zJk#7_FH;Nn5WyI3m^gJ`zJT0aE-}OI+Q>=Ojk)=`Ib5aX6T3J#2a92^-s!j^ zh{6boiA+=*<`x&r;YZ;h?f{3re$ise@u20y-n@QRB!|;x`(n_5$f;t@fy>nCLjzma zwIgK85K80>K6>jXa!v#8x5%D6egp%@c^5Mg@%%dajg|4Ns!m{$(5sg%&CGnZpOq#w zG$i#X4C>>;q6p4cg!oA55M)e_z&vtKOjs_i!k*z&Sp}56-ejhXfSX4GvXX6KHdhza z%EZ2ilh(g?;)dTr3z`@%HHV5$6Tei2mkaT~7FRBcT*BHe)ELHJ~NoQNUe0-e}HqwTBn2Ehb20e~-bE}>?Ob;JYK696j$a=cd6@3HM z|9v#K{+s&lJZy<~1W9jazr2A3u`LN{kazx9h&ga=fxC;Rd%jDj=Zc9)n2wdDHj@8d zO1J~x{#;z?sS8g%ub@C>W1hg&w99YxcowyxXPTm9W!TujptndUNt$J0P5$U~ z$7hL*-E;zaMfQjU$!)aTJC3oLBB>Y|O$s@x zaV%CT28^`a_h`8_rfTb=hE8EdT{go$}-1HFQ^k&VObZGSlKdps`vU&P4Q4 zWQH9VI&Ym-pSM0J2hZvQ!v5q5j^-|7@nsv=doX>0oGz!1q!YM86^W_1(3_1|vDgTk92j#3<(^ z7ryVs`A!b{7~j)SK!Q+qX4aR0cZP!-FXN*5ODSRYYhp1mh9tR;$%(KZXy0dK;m)(L zc*fWEm&K3{f!Sr$b7LwE7jiHWCu_l=6{rQ`ZXy_WIH`kLQp6onUE;z#Fx6s-eX7;F zv-jeGvxkWyr3hx166SG#@~>GhMx9ifBTwf`8V^I)pQS(jhK^DU7HU6xDL*R$qhD1B zIe0LlNdCZ83nf01lfc#-90vS|h*;t^NG?aRd}jq&3ZhbUjIzVK+da_{eLfj?!`h%` z9-6)X1f^%|tmbcgIs+Jl!_pltup}DN1)#^YXmgd4?jorx!NOUguq%DY4mUFHQ>M%D z6p4wN!8eHC@~Ty??16nd*_t~4s#ln8#qC5-qXy1%t~2<^s68Jm)$*DL z1RkIeoAmYycDZ%)7iXaMw6%1g-|&A?Y`;2+9_5J{*Xf{+0^1)s_=x6Z_T=rIEmAqu zW4srVpWeMJ=^Jnd;VJAupkZ!S=2$P-}bVQz$b; zPgE?+%*<*mSu+V@JvENAqm!531-o@OGzg*G#3f!-9ukz>5_;hQsFQ}|4;8<*PyU@J z7+I*Y7bKeakW0xi4j2l##Um&GMbHn5>PH@1G74SRX;GxzU^109}h;Q7A3f!3p{idqBhJxiYx+{M&W?RMn*)uu9OvMqg zH|efaYRvv^K>z4X9r4pt9TA*ZKlw{1RTSGs+YM*cllcyD1+1QeP24tTZ-@#Xc80|$ zCaV2doh>u2GaxivHg~?3!B?%yM-okXS@Ky3CvswM`Wc6?Wjd*gAQBE~*M$1_jrAR8 zuSE{!&bRhltdVS1tzKWI{p-GNVcYTYszIC={d4KjtP_f+Ou<`6tGm3c%Hc0iNfU#Z z6o|^Ojvv0-Lat&r9ZPR+W6nDI+GKcE(X%npJW~bc385^xDKl$&O`8pqLQ^s!WKBaLhi}9I7AK`n;{_d3wT!(Qw773uBoL0IYJy>k(U2*aJHxWv6<;a3(4m7|L~R))F{FAtUw=YOH_mrwZGKh@CnEe{ zqJ%~;N5GM@eAOe?*8>x+R9&bm6ta4&%LbaS46rTvPLo{d>yx!yIC3U3CkU|zf>@i@ zovOe0&fTy-8nVfWkI8=JePW&cXAQ53|FhHHDI^~oW_6q zPUKximKTs?M3cEs>XDn-5m;JYWNWikV&Na`_S5}U_Q3L<-uKguw30p-KB9n+Imr~u z61-{C8czsl1+QNH3Mx!BV$0Uf*|>D-WI-+76Au`vKnvys;3z>9)srPwKA&cSLo$aZ zz~kCGeZp{qB+(%z(t{r*J}|^G2~Y|>qQI}{S0m`*2^vSilsDXWB!;y&ZYgvU<5 zRBgT%cYORcs+=$)nnos^Km;}C@N7X(S(cri=Y0)H%;xu>gU{snscsVxoPPecb<5cN zR3lG?o`lutb(6KR2PTxhm{UX}*$gKaEgom&rHcGMwfV}g9ar|e ztEoX?{DYrRCp8OZN!IjhcBX?WIo){-)73>i{E+an*GF>cDx@CoMOt>v3wqOXo7>c# zzlDYc4Sd9~@?L5j#o8l#d$o@1AheD?;W~ok-8ItD(jA$Qe=H1g4yFr1>=R34eiv2a z76B3tg&;M1;B5k-*fQPtGB2pEb&HYj8QOm{-w(*V3G;_iWFKyzijbic=!ohj!ITx2x7P*A|h_L;Mkcziv@EYN*$buI#EB?Vn1~3%(qr)zbkGe zpHm50K8B`)t|PAJD6h0lvecLbQ^Ko%Y|nOkSQR?3ZqROaZbo%+2e;xd*rY4)R(LQq z|E?C!L@byXXe6p6|M9RdGRj>S_Pvmd)av<1F4ewj^i1{J-UE6u6sfV!#I$V3=r?*O63<0~+X|o%p;Db7E+{1;I@0xCcepeb0b&2Z+K^2)6lO_p((9WC14E{Sn4 zF#56Z(uMwKJQUSA`SX?bAFWkkTR(i@IO;mr8aVS|JpO3IWH;1PjsHbj$U9S!S#M&x z$_H<+wg%>TY;V-u32ViP;#Oy9V$&x-@o4U;?-HG8$k z5h^V0E2GoSa6z`!W!wy33`9EK$}R9~^_ z@xeY6k%D?H0(&huyCxFUbmMDE5BX(CpBf1d<{JhO5tiddM`4E~y1PW7D_gIPJH(e{ z`YaK<11_`XjHD)B^S9a94yV2vl4BXv{hB}HkJwtI0|1j<)rY&FL}*uz=Ry>0KIsXP zTLvIfhxxUq_B6%|YcOjfJc;%7MI5chc-+pCxlI2lhg)GL+9u;kWk<+$N+U0@?HI%B zyT7gP*`Hef9Nfu!>gD!0v)s>gN53~cz))(Zc-VF+;U2@;Y)&+B7G<2%$q8s}Wi<5U z*4+0mQ3#0& z6SHZ>+ZQLfQS1>v0>!Vz#qX0f61hTHYBY~txzk%oPN)(~y)kP`?&E)dCByUe&1wmc z%)Man_NA3M4DiMQ$f*4-A-r&+n$=sCk=AW+?rW@Lv`RoqI z!C_)&W~I|7Lb6}>Ij|_oDu#UD=fl!Bvpf|xt0Ps0wVQ#1>G)u+y&q#dWqrm(%Ycs` zLE@7Ct$oL9|(cn6H0#%Z!i%NgSvn{t1NN=?OnIF{N zMWA&-7{h5kWU05eKd3tFf#hxPOmKgV0v#3t_7=Ls?}TzEf9K1~o*R24S0*5=+g|0s zWF-vh4wglH?F0Qe&@m{|OI54ZnP_BPW^>-IZ68v{mjSgAu00e|=l!|1(HwdN3bRZf z*gShs9st!T5X+@PV`WL3H`Lb;2oC?0%v|1o0Vx>84bv5KhJS$suDkUgtH26eILxVf z8up#u?xlXJ{@&v4MI{fR@VdOpD<&<|CyN8s+#Ohtp1?rCS&gzjoOxd`-qX0!>wiBs zz1v8`do(jUBi{V%m7b1{Jg9TE=_*`6x{ToO3b=HEo>ewXK6CM%Aw4AtMG8$xG&;I- zIOZ+O!PeuNsGg}xE8XGnjn=VCI$CXqPWcQ>j8Dp_DOkBkS&vp?!gGq1QEqOYR!Yow zL{Np%pgPgxO}9hnYim~y>~L(QTB*Ql%LWoKQ{QG6V*!rL_-Lkr1P_hW-m=d}pBU-e z!ClOYhwty8V1FuYdHp-ldyRBcENeD9(wn<%rYAs$zR@R?HsVa8dxR^W24f?9=TeQ2pav#M1!lbsVYzc%cAY9{6OP$2~q@nKDR z-Y0wG>erE;Mk9kW9T%di%x3oC3geRk9GBRd+joetAfoKgdwilG;4j%JT6j8h$&{jv z5$htvRHu(6%X9dRw!Pi9i{^+^k|m5LE=HQ#Fs)uzNH0BT%u!|XUZGQ-oAQJXZV)PLHJ?$CV#zb0V0Yh`VIcsyNpC2pG%^8M%Bwq$Ai%FMRVtnkHC}c0 zbL$FU8m_PJsT0{WRxBqMN2pPg>!tw4WomJBV#&z+A%sHLp*p_gIY)kLHY2eF)mO~d zF)~3(v@MYLy--JUz=#qyQyw$N{k@mZpLT^3`>}b}i>Ok=l`0fp^8k_M6@T8`_o3h^ zIPY!sHmB*y2?-l02^)+8QW4qzD*eHCLkdA;QV$TYBUCt(m%Yz)P;(~JdG;g9Q{Blx zp{?XzIjUZhEaR6g*{#5Y2(JK@01AbfbltB;>4g&YQB1{M2qU16u zj~Q<4xjsH2V9zquG!BV~2nmlq{`NhoZ~hHbbc&34^GtIU1a9CVnrAjms*ur00b13C zd}VLFukZmEa_}_{%rq5Na{Thq2sF~z`qa!EQfvgbmk1W3zod}TGKfz1i6>@FdKsxIw|mw6ZN-c!c6Smf6}p*^1xy*l6J%s8|x??Md@7S@~g< zfse>Xzh5~X#bb7;0x(T?dn>*OuD;>@DgAt7qbzs>wtZ)9L{(1ah}9#5GzsxuCB10; zNbypAGFj872Rb4aWN&YAi5QXy;%R!7)Jdd#6eQylYZdJZY58y+oldl4eD#i@;)`_Z z<}K{U2{tSxOrf`nGsn`p2Xj1lK2l+u6yI5TG(aE5lVJ%|sJrN=*|-DM%FE~b zMHkN0E`&!15L)t0c1#v@qoH_WQbl+khlRzRbDLV)2T(jxO$FbPeUc#T7S(vM@1NZu zj}%*j=!-v8Vj|6`I0sHyf;4&|9=*2i{O3m;)<#p-fZ@vsS4cw(LwI0n3ZnO^Wxp^# zaKTexF&>Yn6&rYdQ|Q`vFSFJrJ5)PIC3Aamc@YV9uY^|#J2-ZBL-&YhTTf47N#$mQ z3Ynw}n3l>uR}2q_Bwq6Wp;;8Sk}c5e#$PN+M3I2sJ!trSmqui7B5PV>OIpN#I5_Lz z?6S7Vtf?jC%saN^##rmed()Fg)9p=3qinAW#o7YbKL6a8eC`pEU*FWAl(bnMbmRK< z`sU7O7|3)b7k@_UOSornY9V9|S?Mr&#YClf-+%BMDbVZ+S!U;p;cBb@)SQQR6UW%d zXgu%KMJ9shn*aXKzFavUO@XV|@w-dYI`)XU3--R<+zD*KYGO}os@Lmk4r#j=_yZmN z`f^42nDXfJ*@Eu_gal}sV~ViAPq>d5=buTm~t5N7phyU$Cdmj%IF>&IfS_> z>(hc?=FD7LlrjBzI;k4*-m8jW9o61(!}<7I?rtVKtJ6!1EnNn#4`MBx)^r7$PZSs^ zXr#h5Xpp)h1PDI-RqA7H@kpyS5Wmav@ZrL@sT7+o6W+;K^N8c!s-tB#V;8^$e(U*C zd*XmsEN5-)=UHaQ5zAHXxSCQbL3BSUa_ej8MBjElKvyJ?bp>gZNnyx^o+xyQ_^*?F zO2)=P+1lLl#DuAtCEDSM`S#ZR8xE-p!S$8Lm4Un!+!7m`2rWdUYdD}78Q z-8(_56M2uJ%j*#c~(WyoHnO}u85TZ6@aFjKG>TXqf(5dP}Jb+Vv9?IdFpJo7LcM2J1 zWK@(kO#k`$Or`|?uvBxl9?)oZ+M2$#Dv=Q>E0EBs>6&qzu*;-lJt@LeS zE1HSv&gg)mX||CpnZPZtY)@@-opMo9hHYU4Echpv>cWGcqG2(kVaeFV-lL~eO!>$) zrZ?GppYc8&DUV!0>O*EWv!W8KQVS+Wwn;ue|9lo^oBZOo=9V1S7@F93hE%3cz8>() zYA5sNkqSF^O5)s&>xyz}NWQn8eS{t@14MaIJyuXz@jpM3Xh8$d&$=W+4-9 zZ~7%};EK9un9Gr86;NC|_$-hQbAh(GudZXfotBkORhh`96dJ=!xa>c7>er)se(Lhp zq(Z5LqANQ0ty|tjp;;z)xceN;JXADQ#R@0p6A9kWFfMNkTE%%>O%t*# zQ^8CNFfhLhjj1|2TQ)ZPcMG+VU25QBvZpFfrpuUp+w`M#O-xM0#d9Je^vjCmFU`!( z5&{VQLtDu0+tdm;#d0Mq?1HOv_q;1Bjr4Qw@pU!xC(<%Pgw{G(99k8UuBhhhY+yYX zlxnufF~UeCv}WzsDn{HWIAmqcoQ`qdHiG(Uw?%TS$6U!&hWAoXEMuHwVnKzgH)8lw zQQB4YV5=gtF?uY`B(`j2;Mfak_j1{gWPx@ZN>7R$8lAG$`siYJ57r*B$GNoNg-b_l z2Iru8PA&>AHn!8hf!qF=G8_Yl2efT@?!I UUa~2AXD3NwE*?TID=V0)rz?@6xw6 zju+%dg?W0V15=rlZnmZGg<@=FgaeT_kypZ}SH@i~Bxv9Dv<+Wr*pscilGcf2!sK-( z&@|Rr(=t|GkdeJP!;eHD%o$%K)ckbzQXuIzrqB(_iY06mzN5n?nIXeh6dIV<}C1I2tZtV@}suN1Dn5+b+=me)Iy@(%-UmydDi z%xLj-fL(0!9&P^{yC^MJY*#pC44p1^TMrvs`;YbQm-;@i4B)t$ zd){NE>4OQK=Fr!MEtk~GTQ zKhvKw*KbfzDX)>MzxdbpE`}?tu!aWTBr>Tf;L$l?aH=?sHEVfVD-xM7BJ^}zRO^r z)nES<`B;`btSOo79YtfNLLL)y?s$GYK_~BLMcv+xAIlKm1%ITTUJSfGu#AZzh0031 zxHws0Pu_1nR7)Lo^@1uU$_BUXC>No~2My^!bFH#lPrPE1i%XMy0unx@9i|90W>+l< zK4?hQ(Vc7k1RCg}DLOeN7)#9cwp`_r_)!mZK54g^Z*6&&d3hy;WJkp$C72{R(gZhp z`Y3&bb4z&zCcNaQicO9a%6;lYxogmv(Jc{Ai{fD`u425ZYP{;BWJnuenKLC?RbSuS zJTxLE;@)`6>s?ilpc@w!ZbNIk$~09g5eFte**$!Os+_p}l{2hLZVlPw?WcI7%%u95 ztVnK{l#u&%rS{Z%F)f#yr*Mf|91_{c&)eazL;0 zYN~J`vFnl_KZ@QHn2|Dal1#KktlF=NeLOvQkPcn>QLx4LD4)Ye*&&)t{ga5Gk6v;Q zuy7~4GH#wdw?~R)zDWA>y7^tCD!E1>JYuM?p{VLu)^nh3cVMS6Zr{bL(_B-=y2M!r|`-B6z>bJfxV3Q zf9^G?*Ydt83U9SgP$%6-l59co=)Mo;iLydEn%OZnqq&k5NfI8sRE*vOP9kPP4ooEK zuNLBO&rgH`AdDNHZ#lJJN!|3jgMu`0<<$+O^TOVM>#m*+m<-=X5UC(ZHUuDDi1!9S zOhzBs1Qj#s`7s)dYviC|%73j)A$KH6D>zTGz7{tfdr2=Ddf+7hBU6T7D;~E1$(%B% zP_qf!YtrDrt;w?N1bH9~FV1Yk|Eb=FOk9Y6zC>|SG(IdWf)=yPejokfm#qYmUiiu{ zSBH=2=l9@8O2a0YzlN6M6dQ8^Y2gk3djd~N6r>Q5h(i_)QYlQN4BB(p6s7J1_)S=Y z02~4cA+{T)T-f1$S0Z_q41;bZCHzeC9^5wJ6=qWQl5=Uo);ET5(N-`kPUf(bZrx2q z1}vIu;E(I18Jwzt#QNvQp}caDG~>dmoj=QYv(;dXCjwGDFz*h3NrpDJTB#kj(H zg+nrL{C7y++5{2=rs5IF$%u1MHvFw5o$*wa%hv8hYg0kdHqZbk95Bnj721KS8JR(< z_7*u#cg2$1uU573hmaAStahmJ^4gg3$}KgX0LYp4?2;9`>C^m~hmS;2daASIGU|b-4|oBE+$m8Jk<$Ep`V!BIJrV(d zXZ}R2S>H0C1R7iQnu?37si7qwx?j2<;a4kF8EZC&hvnQwM&$rIGz5h%!9T&rVq^YbKJW}}li z@<7?~ak?jQgWZC;5c2oEd&Q{te+hl2IXf=fqLBjQNA|y}8VuF~5zFlB^$irzYO<$0 zZ=WCgH}@*Dj+S-c=FlI12?V#1Jni|;DrAF-^0c)l;WQz;%l*~E#{O4h0c&NuaiwD; z5%J!4%o>#8poc_nLyHh~&}7-s%heQP#Cx`ErQkDO*$a@Hhu46AlKV;bEm7b4TuK>z6%esI|#(F0tosHmt?lM$o#=7dmO+|g1_=S@*HH8p<|Ra8|j z>lsPFdim6nGUji9!V91LpHTLk|8=pZ8yplY{wXEms-#aX$fklrrQgUW7*()=1_(1@*$kRhg#dw7JSW%B1Gu)R2iQ z@yBK_6>#6RuJNNcO8F9aT_XII`{?g~dR|jU-4RWzjPC01|CE;9-q8Ua>bADFjPjbd;l~oeOY@>HpiEYrdlC!*>YdiWGG>_m{$?0ZD`OjWY!%%g2u7T%zXvxKt_YZr!85cf7exaaan(yW?;6x zypBS;bZASre+&qlzc=edAnyUJ2Yvm%;bGggEd8mIN*3CC<~^Ho1cqSKyH7mDmDm^_ zwWVQdY`;f}*#$}j&`jl^=+HoL^IVICK0TCCX1*(xEP%e0&VWkFxwF9FaTopEWlN(s=w$u1 zce~yq{k0eVC*-pKXU^mQRNnjFJD=ttkR2tXqmMm0I-K>QGLN=H-tdpDz85f@uw!RC z(u#=inH`S9q(kG4f_#?zm8k4UC_uw2Bd-3_%xc9Rh!RUcX=`kdE-+9=K1a<4oYje= z8&%wn??cg#cGei3SJKQ)OCo;|{B|R{4tzlBmiy()8o$-i+k}MB{H-ILsinVuX1gq& zde|tW0jO2*8jx6+2A$KR66cIt&@>F_J}Ont%ye`UV2aYv8C0w7cfTgDtlR;aFTd3> zjJ2sT=vcqfAuscfZ^V1DCHAME+}AE7W>nS4T_=(d44V~ zP}TiQTcBI)-zfzoo0KU1)?sm7YWmAWeN{=9Jug3jEWuP1XF4RJaOdVghylD#$Dfn3 zr1i5jL#@3=#R9+G{FC94fm_h&l`IXfv9STtu`4T%0)}4B1W9BBSJt?%%+T)~3J5cZ7aQNr6D8 z!tRK8$9idCXR&Yj*+f+Kt3R$g`R0X7paeq9O#lzw)C6c3+J%hw-+}lCww%jKa{}yv zfJ)HNnBLml1b5Z)=N@7A9@)+jW@zGwf$#4Kr+f)e5YXcQO;y|2xY%sRlzuLE98H1@4uWs3JYBVLP8p}zpv0T z&n)?R*3bf(j_cGp8?{8!NNZ;C`(UnuKdjGL;pDF5_4A`ie$P<6TlwF`TukM9X%@aF zgXHtDR9#VjkEaT~zpun2%M}mIME&Wj+WM&Hf1TlHfQu4GPEJMgRG`UdB!9jY0L`5- z+8WZflRodgAzTNE9R<0~2i!mEBx>a8{P6J~YVl7c#u~2I-gme`EJX*(ykfDBszIeF zap~8P?MNG6On?UC$Y*R-4%X1FJh*M9bbUg-RR5^1y9ok7FyR&5V*jP&QUUu5qz$YM zrsR(f8o@D~-Ly<&eDpx_{(pB4+1$?CQU0^Sn7r&=7IyZw-(s-Z2oeT~S);@ulkvm3 zCbH2uDc1B4pd1(;Tm)*-?t*@VgoHI4OL}z5cAw?@j-;QSJBnEb4$qNBU%ylcUHR>h zs(Sr^8OnManax|y*HmWzcv0~_z1*68-|x2+z8jSTfKr?6!fii&^l6eb!lYgta0sFF7d7%P>6SbSjXC!00i z7Zv(Zzhq6G8TVEhralpr7>5SU8+cP-2*=W1IRP+!C3*#B+24AFm7iv2oZk6?WF_=E z0%~pOkodWO&QF%X<_03d=i*ZxV|!_MgPF^HsY5=`j5UmmxNQcD)}7S|aBr6B9Z>rg zixIFcoFXXby3!ci(s~zeULad$KfftiIS4xZh&cKI$g1)Bxu}sNgzd%(VIUZD>9G2f zQ>VoLgGvYV{{xkisq)DahnoHZyCCyX((_}dX~D7oir$;YuHrVpJ`a^(m_&WjO}TOw zvnxr$z6AX;>gQGsxO)T=)aMfpi4yG}S)c!>-~W;tfg=S=|MMs~ks4Zq#Gs*8zuqv1 zuJN+eR_5jl{W-={YCP@G({s6!Lp!wbZ3!D24K71gnp2wWzjd$lP5|wB2c0`x>AL@W zAi!}FS}tZv%tb~<;+6~f3jX0MJRm1CovNAwwfoKbp|KK^fZ*r?^PO@aw?WLkF#G)p zz@~e@7AW|e5RDa^L8n~<1M~Sccfh<0JQaxJwn%{CTVnmgH=8dY3HboDUARwKzIleL zZL^!MZYP{2hMGo3D?e6PG+Q6YUh*k9?*uWHOl^%4S!E2aoqCQh9;KOS|*qaayonmA_##y=gwnK@xH zaOmzyF|e>Q0!7Gr3xuq+X%yZGK6TVcRXZX(07RxJkyj?c}GGGn70e@;Y*(*Azkp1bePAfM7 zh7kKsBDGLgbEU;zb&`8kD8BZOA6UF62CvM`TjxotazzNRLFxLu)Ti*P4UZ%T4^O0Z%x8PnD<>QAu} z!K0u^7qy2ORUB0Uw1Of7DJchOC09xDtzz%QyLOyBJ49EjBlS{~+_jf|$wzA&Z?B=1S)X1MIt^P4%ZcKq+x|4+x^=hb)tXUYqDJyrmeepac%x& z^TynWZmiwhMugpN1#{PV*aCZj2b2YW{qJr9=C3pW64Gvf2kxUIe-T+=@1DUWl>XJdmTnaf)IY(qfQMPpm`0MI6D>$@FTG|5x14dc2pm6ai zJ9*^e(XV|PY3HthE59xdqqlK_k~F_9NG?NPBPo}yO*C%x{ilu}CiCs8qVH6s$T2Y$ zTA$egolEg!U{qLKR2az?)#-+e{}YE&Lwy5G#EQ|FI*(0O3c}XWS;-!{mA!T0?_M8c zOHa>W8RtFU;gDRz>rpTlvhE4w(<%5|Ljgf;lL?1rEhgN~h*wkTPprk)S}ejQolt7_ zeztfp-p^3=Y?_wpiGNI36;P)_I~GAjLxqcjqdPm}_`%!m`!od^8JUI;-*w`LRzGed zQ2e_j#Kib)7UT9_-ON3{`1ZQFN$@Y7KP05lzs{@V6Y*rfmCvhZz3M4nA~5v^J~8M``*5AF-ESQvgDZ|-efb8_K7z9E(q?Cl$Y~aTj)N#UFaM=eAl5r ztDbwZyI&4T&$i{rJE|NN3BCP*(|OKyy}NhiYf8>V78%~P8_yi6|MuKzDJjAzLSk** zeFHZiq@*NcAz1qRGgP0cOYAwlcnXf>Pt_!ij{yOLgT2v=+Lcqw{JCMd2B;{{eoK@h z>G5k^MzNH?nKX=>-M}r|_$ix|w4UMa^N-S!FR*cm)6)~w`g|)@iB?cZ!*VBnY#gA= zp<}$0QEJss>-{zE{M>(YfXg1LM4WBS&EsWf3n4Cd{hLHYx*q);62b^u?ynLN%iLZZ zsq!^he(&E?V>GhU%5ay0JvS?>GB1sinsc|Tl!WxDcXy=P3win2fC#I#-s0$SZgeYF zwy1%Z4!=qYNE$c)b_Pj_q3=l5shEtHxqQnFdIQr)sZQl;e-WyC?TZS7mNh>Ada?(^ zkL)vgQ{{2zob0Xe@*9LD@RLRFa20gQ{mM4GzLht0m6VixWV2`>6I3$BmXlnD+QB^) zgo}lF3(@Fz{8&tG7R*9_za`|g#EFc_eF>BAG8oubg{Cxist-AE?`@!=cuPE|rKggT zlzhm@v@+YDo#{^0f&5WeNQkPrE$TKQ2L5o3rbFh{QNmwCmWIc8@iJ%TdnuzfGPzdc zyaPb;pTD}_6oqsAVL-d{{(sqJ_(f*F@AOu>Ps`Zkf{kXu<0vaLWHVap>1u!Ea%bwW~bBoD^1GZs^KVX_vR zuCJeMW%(9^_@Tq0?6YT{N;KqjblGX~@yW@rG_@C*(p=(Vc|Ij?R9x+UFP@tdDHbnI zR3Tm!85kOCwz+a;UBMYT5PG9-3cS7Qez-@-Ynhjk;k|!!{noequ|jRWY(8iaSX~w4 z(PwjfIzMsbADJQVE4Y1AX;B730A{Rdp=m*X;tlt^j5J*HGb3okY78>c(jmdv9in1$ z?fX6H4tSi(^`Ezl;}v8sPm60>8f}U-#+bT%LGX~?^&l`S8{hY+hc|`9#QIcaR(7)7 zGPSn0HvRpkp-4JFPAp|z5o)38|B?E6|9+E1;mFECVd_x137p&c@nd*fwx>dRVq|QD z_|&4bJW?r}<|1akJ!`^He#62*e%y~niI|Wub$+_g;9xB+ohpp$&Rc6IlQGJRtC(LD zxZ`Sy3iNCahaaO`weZm3;^DokcEUT{J?{?R>uGU~)YJ2sCPf$Ko!I0%;sQFM{<&v< zbTuD{SStrFGOn@EZ5s%?TdwFlT4UM2fTqFEj}p=m$*Vmf(RPhgg?>-Q@oS0dk>klF zVs7iGgXx(JJL;f7CPp@b?%Z|$d0v{+J?3|H>DE}8L#E=w;-EylZ?{Sfz2N}*tZcJR zLq*k_^#U5FBBTrUXlg)&`SN8?TE(*6%`z;Fb%y5`IjbW3b2WW>GNF-UR2Yb6_|&x~ zqjiY+mZ_+`d-0in*8^N|bOxtH88xj!i?V(E(EPR#oB6s2dd zSLvZH`H6Q+OG`vlWO!8M@k85h;VsM;-ymXZwijpnV-ADcW@F1tW#`M1R%k&2Hq;WY5FccQf zNDNFY)~>KCOc@&+K`o&#chsA!Ul{NZXu{1^Xma)r3k>A^Y?JEk%|!ZCEp~XsIMfpc< zW1w^>DJ@8cw6rWDCEc|k-5rvvvam>Zr?hl82#8B}r%3MtOE>I$`<-{@`6HfZ=l)@5 zHrP3w`<&~<=ej}&Evxl_y_%QMTle*IDPgDQpnJ)Hx^S6o10^xnt?rdO7)vx=t!>W> z$!wnMTjn>xfV5+lB!rh+(shu>9^lB{IbpwQuvP)0@RhsOhYtPy< zP+*4m3h;EG%5-GtguF6o=CIUKJ5ux@W+8-Dfx`PmrLK~a(&R1S5Bo}Ht}1ik9@ipX z0=;66$51s!+Guj}>`%K0d=OUV36CS9rf_E0ZfjBtz(@0zT;~KQ2R;rgpEgU$r zt6Q;r9JmQVBCf9KBryObwnZ(M|xsS+7ynsE62)&9b;7-6Y>0OYa|I z+$5*aR~(}(NZm?SZtj&@bk(5&A#Lkj%{dEQH2lHJs1CQWp1Qj0*|j4g&zcr2T3%O~ zB}IuqFQJ-|C3#P&_r8CNe=Da67Q zNnF8>#r!P8i~|3~^3H&}M{}HHXYEFf^&`QPazI#yMW>rBH_Oeg1gT`+`0aK#mnL!j z(8OfLmpmYfm5GK%h(%77V(&jGv#%^Psj)eQXFl1TTXkEqWq!&z_Jsqg77i?re9rm9 z5cH%)MUh0b7%VcKFvdr(O@9F=G$sAt3h}=(i`^D1b70U_=Y6BRk zRfVQx+V5;5Vt|J=MFg_jO1K6mIM`pVeVQnb zz7_I14V~vN)M3~B<#V=}Au^iL>9KYDYAvC;wy?i?erlDsI{R6C++#XY5|a0_a1t_7 z%^>+qfy*YDcLWj?-ZnoP{>P1d?dGDE{eevB_46SO7FI4_m%wUGO$}f^ck)W3rc=WB z{)M)W%G2UzrnUM|A|Q?b`;pUK9c5*EC9v5o$JXdfQDMhpXxhQYVg;u_Nrrca7U-Rn zISm4Mjb@{r+twC#VgBNx=d*ix)kaUy-|dr;-=3w(cXdv4bN#?hB9M=ER;JZ2JABeM zW!zOGR@YwuZlAO|~p2k^l^Tf!R2Cx(sO z1llK|Zz_eRQscLX6^Ad<-#AUB#P$@Ecvq4mg`7NhmUef#0t%TOadB`^E088{bZo{_ z8*b74_vSMmPp2JMSSYpcYk=ftyt;IM8tV0Ere1fFM_=H2T}8%o?+LZ{yKL*U{s*`o ze_*eQV&`5D8yi~Qf3QKjp}G=si~{e^1J<`sPhqDxv4sA zN!r?`dy7T9RAwU0Za2`se)Jfk+rrJ{|FyQU))V6IeRkaIK!!YDej4zEV70-|L{?Td za{y*lI)84rJR|xj<7RKU*>ev#QZLN_zQJY(fkX&k zE&|#=OKO`k{Nv(&f}SG0k_;t44ixfjStF+7=c|1zHDhIE2@94(?C!o)OK_Rcn1IIA z9M`1r?d�`LCNy7Rfv%^gygQPA`~jaT~b)J*1mbAN#c3$Smc)wYjOQoFBLY{6gpQ z$*@56NP&ZxG*}k1bGvil3|RS{*Pfl9pftQtde2qv)aShe{->W0l~=F&F%fwH*CT)& z7Y_meih^=a1HrxvD^RHWDmW|*2R@nsL9A(OX=NmvxSWi1&N{D}SEtPAka;MwvT@x} z(72zY1{qmyQk%wX_Y)Nt)~537rb~AOrX#$4jwA&u0yOHEE(Bks)w&v2tsEO!{LpoM!I{0Ba^Wf7H6Z z19c0qlA73CUP@j}V)?27-(JUQuFY}$xU$f>{KmF+uvFOi`x*SWUt;fQ{uFCsIGx#AH{;00?*)Lt)uxAVwN4N7|VR~9fstB?ChTNil zLsYtL1h_oEI5di@Og2KyjIN4a=pSYRpa#%CA(+wOFH}ANb{ifgiF-}CXlo||pc^XZ z&ji)0<4+fHE;Qn#3 ziQ~?Bvx_D0{G+^Gk`Xgr^wrJjG_vtwkPM)SI6bW;=3SF1<=0|bAullqW1cPZJ?m;0 zqm}eQ99)T?Yx-O1G<(C10I>HT6Vbc6}Op!OG5ldvw$A zpIFWBY3l}!)fE*vLSQ+>|Ln9m9AH&mlK<~cLhLql0Bt-I_-NZPp}dMhEFfNE?Ih&+ zcTSOqvQi;0>VK@04=(>+294qxd-*%lm$r)jNr?t`X9*Hx6?-%}5JD|+vv|Ep7R-_i1OVlX>> z-1&3N_^Ar@RX}Y;lvgIlg2&P-)3D@*tG_zxOJ>@oWU5$4HHWI7iP7d#PmSl~OuoTz zm9Hx$<|CmyKbMM0Q?WEI(CoFlwL6)x0UJ z7`)is+f0ArTh4*$X+(=kO^$Aaj1^txfB*Z;@)ah&pf;|1{K_)TVroNq1GOj&oD69G zyx7TXb+hkLY^h8~3qAZk0pIt^6dQBvd85u_be}|v`I|@d{PuLB<4$x((9<|AS+ss= z1SxG*uKmQ5k0{^aj^$63vxB)#m8)aN?X|jE+iDi?cRE{vpE|2Gk%$p0X*hX6HZc#z zPwyuWJ~IA2x-7A!1TM*YX_%LzhOVY9;>802MptZw%7-8w&J=F{6K&u3Gc)aSf1k%= z0wlLrz@=V&bdmijW1-X7P$`J$?^wxQyAs@vy}3}v`O{dNuF=Ynb%dNlfbu{ zg`<;$GOaT-e~G0F+t1pIJStOsg-M^PINM}gRi;E%CCKMR+FWH2ZRXRZ5rlK|ah>-& z4JR>)>Gzdn01G+c!|o7ax4zH%n!kxapTq;*46gfe(QQ6-N8Q4vP~376H;iY0bJNMg z`$;D1u1 zr=nfU*RL7%!Sng!J;j;N+RwsgQQ+2-P@?0-6-F(^&a7p?6TtQ|#2~(!no*+`%G)i8 z*ti;`sUYO~-MHV;CWM7+edJMNS%%Sb#-Lz+7Ane}-rp=)%^v)J%_B0X{pNBy%wdFN zD}uz+M3qAZ>2-E!YN29OCQQ{@l}2}$C0d)x8(1mL7OiSuAQu3#=%<^f`i3buQoMVq z$*TTlHNmMz4cWSUlz8mwvib_L`u)H9Om{v@e$Dv4O{C+uLkt2SOJqTFqlu|9O%3G_ z0hrorO>T+CjlLf5qtxK*w6H;#C&hsclV0#3kJ#=DuUQ@;?sl~(w#jObPwXO6$`?|b>js|M&`DK6OE9fK($Q{!EzR4}JRbkj3 zZOwJc{g!%Fkf!Lc9p$JKxs!8mV&ri;ezWR~>E}fEkyT)G$Y))`QeTiaHa

vv z-w^Ru7So4d&DW)-mw6A6-AQs=yt3w_xWxGF6n1)C&`6hF{Hz2n~_4fH@G=Sb-;Udk|s-+0#H z!Q3<5_0N`hKiwYZMX#)$L@SSQ>%^JQQT*VcaM(9Jx9r}ZjCm(g=s;LHs;^Q5Z=#xJf@JTENpm6Lwkh~P)jJr;NuXo~LVc!u|J0nHgXUcbGjnuZEli&ON zL5Nr`9bz+bK+K;PYo=@1iCrf(EGM3~aHFO3dmW*~wE|~T4L1XQeaeqhZcW(eBAD!Z z=am4{!G%p^;{g{p+bd{-+Z*!x=1Bv&R#+ZQrd_Byb0%}SL0N?ghcS(#ZZvuKgO3jW zW?CPXHh6pNe2tw!8j69^xmUC94@*Ww%qMK51z5A}2qnUTOI+U`E?(x7^a!<_&A*#$ zS$ocy5=IuE*7D95!K)Z=i!e*+6 zP@3)#P2-}VNNcEpKLrtaeA*}nhcz$zMTJ4BcRzAH@8>53sxN=Zk_GJMvuWN!HR=37 zRI+emi^R<#QgFwjV#)}$Ub9VmPU8O}HlcMl4x;kFdcNvhu$+`Ak=L>FCYbc@H_DZZ zbB!1jCUOhhFjTf2%^vsSLDe%+)%ym2Fr%>$lh3$U@02G72}`;qHf(J`gi16*G9fBf zAbA#Zt_xuWx_I|t*I8DKwV3%E zkj^`^$>T+;AG7to6iqI< z?*Z9YBtXa781sup%8Sy$T1^LBy@^2HRIW4cxgssgTdfkx8^V@qg}-sA_-1=Y6Fr93 zd*RK-Jf;!6k~#4;pejyu8(i@p5WN%lP}N+w6z6?3uX*39#wN<&t7aHW^ei<4_L&GH zC$;n*?q~BhRbK?vlh=f9SBKBZLxv|r02Bnw?2p|X7SEGvnWNWSnq$Yap|dZ>ir~e$ zj~Na>e1uvu&u4ETG zxas42Bevj4aztOyb_)W9#(n?lVy2)_($;1qvpl4~JC>(rI5fy%6QeAUIRA*#`i-Ft zdu`%os#3@Tg$-K>%cch1@>%4xPOMmSzN#N1zDQwN$TPnYj6q`K1xZMxJKhV@fCBL)f;0T zYh!MWW3~^SX>ID)oEI*@wl7F3eb;?chNvtGN``cF%|;QP?+?cG>gR@BylT^OgVNJzXk5`TS9vFSEuj&rSk3$HF?YVd<{EbvWW zshKFE?@aJHsa9BQL!7vhS4xn2ve&|RiO<(tz#~0nlTs_f+DnqCHFcC@rx)K}bcgZG z-e!j|69I(M{#kEXMfqiwzsO?sv$K{PE5&mI^@PR~?DrXiB4)~2))7hg1d3k?HWhcR(+iYm&&d}Gg#MhJNu0CZbWXl-Da zg1u^LTU^Ov)SrfBLOH3W%Lq+*K{YTCC?a1kFtu{$9nG4Fu53wCun}NO)GL9ov$4PN z+`pX=b~%V%2Ry#*E+{N49CnI&WoBIx_t;S-Hr||^JQ>TmZ%*ys&Vky_jR{1G>HAyI z`WPcrD+UK7uax>$@3~xUCAeVjy$r8pUd*)^`>VZugZcgA>kT*aoZd=u{B!6zOAV_;$9L|5x?`Ax;ssBAP5v9^{G=dI@#-7C=$ z@vAgXu}RIFM%DD!VZ9bjou35#u0rC;E5>iTrIm(X9bNig3wk3K^bHKI5&KW6xf|W2 zwZ}L$p#~lE;SRLCjbK9=?apVBXx73pCKFj&+!ix@(AWP)q@i~@?QJ9zV9Xs&e$QJtgA=N=pDV}4$;k|5<=~ic zf393Qu_VG)GWvy^cz)C8_Bc(-;j=PsF}qe4C2NbOf@UXyf2Q65S7}{t$PaM959#Nm zpSDS70|}V4(~!U73GGKZa6%PFdA4xq(v~XGH_(d&y9`C|c148%&xMkD^EWBBopXW+ zNS>oU-h8?wuf)vCikskP_S}@73v5+)GT)Bk(4mLTz0@H2>9@cUhcqf9hTHdwA{i4J z1nI+WC#O2v>frWI4b`z7ZPbPU4VBbR32#Wg2B=AB} z&XAv-xuUp9&n5OXHv^k6kBO#n;#T0ey1n2pP5bzn{7hRzbD02LLKMLa831prlqnfS zr38a{UUA91sfOh(r(Y0nS1c_we0-JHV0}2o^yaj(=rZk&p zu2!0qDEt;f0c(6#3s0%9zmJ31XmW6f=Po8@^1V##0ud=clNH&EyNgewzskB3GG2BW zCtRoDI7?ZQCLKN|$ChH=w%kbEDb{h1RjW&;(+F9h3e?9_dIbf-`b5c-+-1J0GLbHK z55xu{oK@NK{7bXS%o9YILr6s3)a2N^HTX~{Oa*SrZ*ko)i(CXp-(Gw&l;d|y9(e*r z#W&y9DDq?!v(?he82LV6_+>=M^9K{d=D;PmiWpFOF0!`BY;)lfV^P(VQ9ex6dT)?6 zY*np4^fyieog>ULVp3kuN*j&tl_*!ND*x1^1M|RkQ z1EqmEd^;Ry%=uO17n@psRK(gvP612*=J0HIx72SE@?@YG0XjOu=m59FUzlA80sECa zrS1B`tzbyReA|twf-To``Cm5pU$`?JMT`zc?%?0e)87XQ3@{*gkIibgg zj=28jkHWQ>?DI@C5AW1M{>!FQ6zLt?{pD5ZhbKMFx?GcTi#v$oAFPmwC{h)VgJolr zN>4qDQ-pFW`0|Ox_6A@oauPDx_r1Ptr;K&ZXJxxU~G-ZR;hyn_9G%EY7G+nlg3s%&z>O-!O{~qz&wiRa)PTvv6>g z|1|64x5QQQ83jFuv>zL^xk0}jP#6Wy&l#&8T=xF9Bssj?ENSiqqvEg`uv3GfF+gwy z+7@4}dqPJziKydC3j6r%%MH*2k1*cmb(407=(l98*pi6p zWd;d;LyuI_{FmZ-L{J+<2NRO|+`9a!O8Ilf6wB5p(E%#^3s30m+`5*kU+HTlM&S7h|;Wg;ns zcA7_#V;5^mvrYvX5qPIi6U=dXyYi4vc8(gvz85ayepgm~ivSOFtB$Y1hk9R+YCps- zqyJS@zo`Sb)&jZ0Sbg-(k0A3$LT7tH^A4Qko8&1LLjmh<9+Y)>w0D@Zu`M>Z!0Y$pa2iJF3mcY@=UnKsVFBUXJzF40YYXr=7}7ipDo zpBPHvh2n;dP;~6)(vPbt*&dCJ5~Aw)#}TIa#4pmPA?p&-VTX*O+k`bONOI&EcPfrJ z76X~%6kdITg(c!Pee12^U8e9?uNACk`L}#4SuOK5Gj=g8bwfmcJxHFsa1nnWo#o&2 zU0&C6bnSG_>#_8fO?m1XI_jA?C5j&K;$r$QUg-p!)Vkn_q#iDbZQB^5KZ>33ON_Q)4jO>ZvJgdIT~T*~_T z`arLtlzg-h43fu+jg6hJv+7^6T~}UwQI+z+N2H-T_r8gkdvIHoymE^ma{=9Er|w|% zmq+YEH;tW{9&C@*H#oSz()POm;taI7DJg9|1rxhQBra~Rr4R$O4A8Ea<%W;F3DrHP zXC4%I`oOC|iJm-~*D81HwkN2xMM1fPEfria6QZFNvs&-tNhl{KB4h}TT3)W6obepA zw6KE|2V_ne;2-@9Csq2KMHT1Jn68oUv@DFT^C8Z_ZsUol{2O&crS)h>B>7Ov)XI-c zz(x@fbzwDe?d8Mhq><(G6U8(_jUVSgEV=9;VB^|s8Y>0kzT#jE@Y3%Did3L99OShm z-K5aDwhM>N?WL;2LRHkGTwSu_WvK+$U8>$QW|1AYGAn5IbdXJ-kh9jtSHTCw)?hRjjjw@-+a zRwZoN;{_}4nx7+?o(3q(hLqJcXCZ)e)6K&`VvMhmoRqq-*ajUWZ##+UNbC{grnJi| z%Ke!Dv{K4z0_>70N6xMjVJ}e`vP?)oh{AU#eg z5}a}0VDeRhI2F^!DCjv*XtT8sBTU>eFl-z&3h-U!hy);6Ja_kRe9s%t200{bNOB(B z)wz-o!Gi1s@ zIcuww=K)5J?HUdPhQMcF_~jR!9T0IfRcWnzZ9kkMz%0qudx1=*7J3GBGXP9N8UG#< z<`F{zfv5{n(=pH{^z(+YvJwimoG_X8{_@T#^Kj)t3quRrFd1-=uc(6s5cm& zH-P0m4E(o>ii)~&|C;h1v@DO*bX$u&C5{h`Z5(uTK9*#)boW&_k~hx@coqOVryWmI z013)}5Bl8Ku>TIIV7vt}{CB_w10zNa>%YTX7#K_>5C1#ngMon^3^V}#J^t@)|C!-` cyFt)TWmIh$-&<#}{WFJxtjfCzkm=X|2cxLAQ2+n{ literal 0 HcmV?d00001 diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index 3031fa4..a68422c 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -148,11 +148,7 @@ def _prepare_sol_qkv( v: torch.Tensor, sparse_state: SparseAttentionState | None, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None]: - if ( - self._is_sol_active(sparse_state) - and sparse_state is not None - and sparse_state.config.sol_fp8 - ): + if self._is_sol_active(sparse_state) and sparse_state is not None and sparse_state.config.sol_fp8: q, q_scale = quantize_fp8_per_block(q) k, k_scale = quantize_fp8_per_block(k) v, v_scale = quantize_fp8_per_block(v) From f55779422c37b4010bed22166bd5847d00b31343 Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Thu, 13 Aug 2026 09:39:54 +0000 Subject: [PATCH 06/15] perf(wan): keep H100 Sol-Attn on CuTe --- examples/wan_video/README.md | 14 ++++++++------ .../assets/wan21_fp8_sol_h100_benchmark.png | Bin 50062 -> 43197 bytes telefuser/kernel/sol_attn/interface.py | 17 +++++++++++++++++ telefuser/models/wan_video_dit.py | 11 ++++++++++- 4 files changed, 35 insertions(+), 7 deletions(-) diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index ced15de..7f2337d 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -182,7 +182,7 @@ Attention choices: - `dense`: PyTorch SDPA - `sol`: Sol-Attn with dense warm-up and fallback calls -- `sol-fp8`: FP8 QKV inputs with FP8 Sol-Attn QK/PV GEMMs +- `sol-fp8`: FP8 Sol-Attn mode; on H100 it keeps QKV in BF16 for the optimized CuTe kernel Quantization choices: @@ -220,10 +220,12 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --kv-splits auto ``` -`tf-kernel-fp8` is the FP8-GEMM implementation benchmarked below. `sol-fp8` -quantizes post-RoPE Q/K/V per 64-token block and -runs exact Sol-Attn QK/PV GEMMs with FP8 inputs and FP32 accumulation; routing -summaries remain BF16/FP32. The SOL tuning options apply to both `sol` modes. +`tf-kernel-fp8` is the FP8-GEMM implementation benchmarked below. On H100, +`sol-fp8` keeps QKV in BF16 so it can use the optimized CuTe SM90 Sol-Attn +mainloop; this avoids a slower quantize/dequantize detour because a native FP8 +CuTe Sol-Attn mainloop is not available yet. On architectures with a native FP8 +Sol-Attn backend, it quantizes post-RoPE Q/K/V per 64-token block and runs exact +FP8 QK/PV GEMMs with FP32 accumulation. The SOL tuning options apply to both modes. The final log reports generation time, frames per second, and peak allocated and reserved CUDA memory. @@ -241,7 +243,7 @@ same generation interval. | BF16 | Dense | 0.854 | 16.147 | | BF16 | Sol-Attn | 1.083 | 17.023 | | tf-kernel FP8 | Dense | 0.877 | 14.855 | -| tf-kernel FP8 | Sol-FP8 | 0.927 | 15.730 | +| tf-kernel FP8 | Sol-FP8 (H100 CuTe BF16 QKV) | 1.158 | 15.730 | ![Wan FP8-GEMM Sol-Attn H100 benchmark](assets/wan21_fp8_sol_h100_benchmark.png) diff --git a/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png b/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png index 5b4196b86ea45fc84e656c74dafd640716744ce2..0feaaa50dbf4aeb8c2fc158bbb18cd7e9cf1374a 100644 GIT binary patch literal 43197 zcmc$`2T)Vp`!0$N_&|z^AYDYN6qP1j5u}6k4k}15(mN3W5fKn+QbG@*Bho`tdhdi1 zLZp`vLJuTB&c^RIXZ~mI+&lNoIcLs{6L7QE-fO@0d7t&JM7`8frlMe?AR!^4QdLpZ zB_X-`gM{Sr2>E5;3XL7xF$qawxT@kaeZSc)q;Jrzl`Gx5JgVce>z1XpiAZ|hKx{Ma zrVYiDm(LC&d9@U;F?{~~>GS)$fmlrwmrw;0!SwBBqi?G?Wwq-_%Jt(Ws}@V{l*WFf zXG-K6Vr^S8%Cw9gco50^O=Yzk7r&5%(q4Q%y?gcI2QB%fiyv1NER%C&^`wi_ibPf687Y9Ccf7~14Q*&J0ngSn$uQwClg$?Ch9^RSM@`rrci}Zcyau<3_~T-o_O=>V zu&t#g*%4-CwA!6dTo`L4V1?fef5f>z)EdF{Km#c@xj5DPd$Zta%(LXyZsZY zOY(kLm*C(t-Js64xMb()>YB@JC|xX}aN)VvmDi4@v%fjJKT>a6jWEatu;iwbM$mFb zQt@js1;HUH-xKGG^{`FjNe&K04h{=9;@5=nXLy1scGdDhE1_5mbMsW^DMXhGntvaf zFGKy0tpDm3PQGk&yb3|npKNUWy>&XwvTIL6Rr3r*mgO>2X;x>u!WrpmawNPXOh#Dr=U0Xcl-BgsOc~u6$#fl{`-!GLCSGictY0T z~64c0@pIVTLiG4yV6Iko=y_yawvcE9N9^jv>lX1e+ce`w+QcPt2B_T@o?huAyx zkZcl8)cAN|Ev4LCP-_mq-EGw$yt`*G69?GAU0OMwvZnMEShcW`GHT`hQ)K$}8X*~^ z>N;S^dmF9Yb>%9u;oMzaOD|}wn)e#j?c(HM!86!_g|}8g&Q8`~r`j5{MADm;7auKS)j-$cgaWRr2eJ#J&v{^pYgJh>b30-kRVhAL<#oumx?x}HCFuUwxk&qlj zt;Q1<8YCh1XZBFq7b{=mtO_H`Qrd7(z27zn*-lhlIzK*)v?kO6P8P!AwX)Y>U1Ai1 z-Wa{_#xLX%(Vr4h%D4Pg+Gv<%X1u~T&Rm2=%Hy;%a}y#oyw7k}yER=V4Y9)sL20ld zUWC0I3u3MPY7t?S<^d@1m^Yt=K{(f+4wR<~0R02V}*Zwp=*hc7a$A~-&9%cM>C`chJ~lz?9Y+5zVG69mn& zS}gU2Qr0#tf0nXSBRimB0-dFGE&pNWm9z9WA zT_$~$aU{v^cV_Es9uyD|5W__8jh@(?%Rz=$a)Mb{GnFh7Z+*P|IhZVtMSjJT>+VB- zDQ-#0nTdgkd3bH688Q&>MZ_C?MjiF07dg7u1|>+4!QN@QKo0v;wKcf38s=)ZFzj4( zoc2id#-w8nHer-=#p9%2r@htu{P`Q*pS1UO@Ph{j*9Y^+>D1nq2S%DQ0P-^}(^6O0 zR#%Vv^10=YufyOh)%;+y;aUCMOmF4RwFEs~+hL1$%hRfu#GOw)vIH8;49I>#+AaCu zL%aE?QaRMl<*kXP*33OzaK}ifdds<1ok6qrT4tS97DRe@ql&@9yEKMLu3mR5Bp~3$ zpK;%2A7@lS{)670o*wUE7o=1T|59)E20I-54Si3WLowQx4HhMIUsOO`NK54SB_o3{ zL#I4Fs0?**l7iyq;z4VDiH(YhjXFGR-D@8%vC0y2IZv;km(+0O+swrPY5I6$Z2h}2 zGHQ4CAK+izhYz?d`MKQGwH6Ffrjg>I%)LE`gJ-oJu$r#QThKW06IqwN@L_q=J8U;3 zU5?UWNhf*T9C2)s&dAC&LWMlLoXt~rYQ-Y-kvXEk#K8SL`H?p;YATSKiRrv$55b0% z8W*}mB4d|y{QIDxD@mthlHp9)$&&0EN>fE;qRDl2rlY6l2gKKN&%47sBOcs9p94?a=a#WYOl|4;H9wz z)tQYr_9i^Mv6qhzHXcb+lnB0g5thUt1yy?Gd_!7V*8~l^I{QO3qCo=lCdZ?V(Q>^> zq~3gVpWO^k9|vxKXd71nBpWA3bUd2#^?>}o$!C4-dwi;xLzh#-Id$$X!3qrml4X>h z-I}z(SA`ru&)>PaRv*kVV6vPdPJ07Wo-Stbp$|DZt;Y_gyvDO5B2HGlTN6kxkz8&2 zR#{gy=*{{kf`=w+B>3`cbn?8yr&_5E9fCtST1imQX`T8pUxKQc=za#SYfBJ4l_6FA zUM)v*Y7d~Nwg#DWD1UbhJJ!G&i7CB|X53CXOn-eB-opnMR_er0{ zZkQldFY5lzVbOeeJkMaQSHyDMxfqE zxW}%=M-M#CQgE|P9&`*Tf#fD7GE--!CZ=@N`hYoGKf6sdK0WG2H8_5k6e&0+u=oRc zDf!UJaSD&UI^AGL9Bw_+iS-_xr6QjH#0w+&4(H56EGjibMn-3r2=vVapZF=$l*-FZ z0T?_cTe`pH_*m+v zB36v+teSn!95WY6DQUt1gt26q>Ag2#f}tY|Yy=a_fmr>8{vQD78^3|ooo;Z;B6Hq| z#>J0dv?fB*oD@^x^MhPBqmEosVlg-u;u^Tl5;?DRl3bzo%Vhb!1rxK-2@ z{w2jbMb>B0Yaw0AXW~aECCc5y>!GM9i@dF3`JE*~zFU<_NJ@Fr$q)gihd2-gd8Tce7CI)`iGM1;x$;sK?(hQ?voW^b>t%aOApTIE+ zpJFkHty5pj8T0ui2CY-r8xvWHsSm3fTB?i^{;Lg?Ewg;(-sr9QpXINEebYEe-ZLmw z=(+``!v)WgwR(i3%^Eb2PJm24Yr`9_Vj3vHpdgBKbbR$H+n+XqxI&Vo#+A@3gqNM9 z=*9gL4#0RE+moAs8?-COC45U59~ap>Lt~eq*vozB!ML6?)nT#Qd{JIX_>^7VAQKR55Bw4KikqGd8sJ2W9$0>Ar>Z z*ZVQ91Pa80pOi5_gXnHFy0w-|KVSNIC|%9D2KO6mQg)a^AH6%^cF`MALVhn#0BGEby1u}Snwf~NVR2KvMv^^yt*urY^`h`4zpzEa z>Zb0RyGo(0TCi^&lbAp_?SC(h(n!Z+((98p~wKTu;TXOPwYiHJ5KDr8xbp6ew`&>Mr!@^~b~X z#EP+`r{$=4+Iv%XA1Qnq7%2Ua%)hj_B80MzM`H~&nZ$Ymqig|Bbh%5XkRjGylvC*q;xCh|GT})i-y4J=# zWN9-AWe_Q93=4Kc)h2E4^F+`he6}`bi_Xer$>`J|KDqk*tdgaRvCy#=my%C}DKWv- z@bC_xrmx+et8(A>K|a!NH9k5kTnH-_pk%i7IT?!+3l?y!krI+S-%E1J3O>PC;8@^x#q!6Fo_3p!xRtQLi<+&V4)9hqtIu^C^GJgTtw_lG<(S`(AFGO@9LUrl`V!LTTBf;Rw?brtDF3 z2Plt0Hdr`8;*@jSDy|-M);}yKaQ=dcakYi2=-3F%m)IwJ1NYm16{MlfZ|XBXUm}dwat*k_;~ldgN8Td z%J<@p%}frr=HDXIl3?SLj)$u*jSrBF-_VOs<5AZ_;+}hGJIwuf)*xW{qEm*x)Boc$ zp@#B(feKjRMBg#wvB?dhR&Q2pildHkRFWIga``zQMK8pfd(dxZ=TR9zXodbL^{`mm z@a6k6cTFB+hICRh^OqQx%leaxg3QI`gQSczk0gm+w|<=WVOyMM>vMcgVG3#HE4-3i zZx<71U^rC@j@8zaO<$|v0;9<+5uKyp;Y0i1i4zkBPH?=e|I@caGTL#}6P;-=$UV-iH{BBoHdPKX*M9Pg_BCC!|9o?>vsWAZF%928! z9_HJ9Ti!Xb|KkdNQ#tN=@zYM-7VpN;(2!lba)WUb-10whpIF3w9;ea*?3_SJZCSIY z7`_JQ@I-1FEoU(Bw?@n;`Ef5Vw+6?!dbJH2_Ey6=W7{mQAjVU6+eE@gjuWWppDSyR zv4YLZp}6*+T>EQ7d_#+evWU&t^Hw`zj+&P-Q0Xwa3VAoD2X1Kfikap`Rqno*6SjR1 zZ;Kn6Ru$O`-c0M_1Hb8&tn()A6D|Nqqz0xIHOSAgvs0?;*YSrTn;cXKpY6%ucjh+w z2RFdb=6^UoQf*sVMyI5-0VWK+;V0^1K7tacCR)h(mYR!~diSq3ICh##h#I~S{^9ED z^@HGx#KP*Hd{kDXX2<>KbrT9SnE)>ARXNX8vhx)@l*OOq#P# zmgzM<&J+~ZX2~On3&_0L+>!taoc_K(7yy0pH!3zsWg0Rq%b8h$)M7%nZ|sGgRmJuC zO;*Mz_1kWc1QJN^ zuxz{TX1Txspvv-sCc5wH@@`Zp-NT~Ht;?2Y4_z4<1IdBPt(jYq@WKF$&oW?c}>xe0-Nn$<|4~H^854W{7`Nj#I zge-3@8!wMnKac~0T~^E4&n+|~z{PE3^1i$waI$=6PQ%t?l$D8`@58!Bzcp`~Tal#) zQfWRyBsD(1a&c`^mh6lDNU22`ez0lQi&fSg{}UTBmj8S2aId4Xf$bhUU65~AiJOG$ zXrBWw+06XnBWyD zPPyQ+81;~;&J`C5<~w&vEza`7nk`zK&M{E8&*L8yl^zzqcl{OLPAVI@dt2RZABOTo zo_!zIHsWU$uP6StASPEDJu8Pw1a|eKLJl2PWII-EZPTZSavt`fWcu3N>5|@Kn^~>n ztg7_@eWM(~61}1>f4)~mBRDXa3EB<8F9ga9Sn0lH%d~v9c_hS&sIWXK0}3FCTOJr* zl0Ok6C6@_*kaOzV+r+4Cz{jx(S!Ql=8Mm0MP3fbDa-H z_T6vX7%~ud>sCVk1h8dFEoL?lV6lxAZPYwIM@0B*+Cj?!7Mn@D2Vk*h1TFFSy8GR4_QD?l zXle$aoA|OkFHMyzcXt#NG$v$X!mHV4phO?>mj_9>^db_Q~^hZSDjD z`%c3#mXP^?2nGLxgOmMpBlbuo8yj0w)9;8`;~JNRnMzH{Th@6DE&gj~6(I!ltglu2 zY+IGPl9^ej{%_o)N0M)+LYGp?k)NV8ABsONUDrPK;sel=C$&q0CA$oLlfk34oo9Wa`~}^G)PfjLs5Ug1oa@R85f|*LpIn!g*alB^wyFd z&ilHf%&{QXk|GbnQeC-n_RV!H4OENX=v9M(nGi>8Gu8X=#XG?}yUoFCzZtN@Vvj|G ze3bJ0hC%{&r?@OIkP?YaxR2!%-;WWgeDaZYzQ=tlIDo+MUpvo}#<6#Hbj7huq{Eni z4Zek{+?V{Vr|z06em3AVt#+~l`{iv%$2S>Yzq8zC_MImdzwSlApDqZQ!{}L=jfNpd z2mf$7O3gqa5O!%#_odjgz_^UDeAZO`SvaCZT^b9k0S{_u+_QdDw=Ee2DE!=7ZkD^E zs8Rdb$Ti;kM!v9RkAnG+!w1tt>Kwre zIRf~U@|Oh#kh&7H5*XySd2+JP&$<`!41Wj8X7l;QZv5cvE^asm!@ug(-j!d zXl`bTW1H)4`qTAR!@LcZ?n`FgDi}{*!A({0h1u{pnK!AJl`IzA>?g@BG(X+b1F7Br z+Ja<;H9F6_ze2QdDtr5Wg$;$f3I>FaA-bkGLTx+0ph0%wKgX zyO9zpUCS4QqU5hsH*yO)SxP(cv~-;;b#RY$aIb9@1%;GIV6wJCMx_|NBb>3z#sJFx%-?3vp6gg~^Ou2wX1Ac&G3{>m%$lZtJ zpBCxIFpia2s0(w1HV z#l=1+_zWMeK^2CMJ~+O&Ga|_5wHpnfwB}KHLX@6WIcG4dMn)FD2Tw$0`XIHQyXeiI z6D6G$iEVkgI2I{JhOG5hZX!(}ZKRWZ7BT1Q1*;YS1XBtQOD0JbL|c!MX+Rg5x>2}t zr~P~v*L^lQ|Lx6~Y~gv32#VQAcYKTibZD(%uAtT4FMLO<>W%&G-h9Kqhw z&X#Yng3F?jvB`C1?89m+k#`xLb&JjQ$>CUsbuWk;mb*si`JJXT;hor*Sx<>U(R-H#kY>av;~p z+Tu#}m+OJdXrHn)s*Q0?v#ATcZnv_ks%~yRhL?1fCdOt+L1)-FI)EU61(`u#@;6pH znZSgv0V+JcCreG0(to(wILQ}O3+N(JxpTs^NY;DqM_TMr&W-q1x9JcdykZEkS}c+e zv8U2BEYVTw_Om0wd*6Q_iNkSh5vNDEe@VC8fikb5~ov{>#*WDd(nV*Zg$F%Uj_z% zpfcr#Ybny@TmVF_45t^Kh7T zI(-^nEj+o3JI3<7-wNb5BkR8W3makvtq=xT&t5G$XuT9#adjT%g<1XCxv$;~2E0>& zoJ^V-d5sn0Tuy$2A22*qzIY0!+=ClM`v^(E^MY9Z6~3-%Mm$wN)UWlB${a z_wHlHg4}oaYJ#rhSpA{`09T!z=GcLln1^rhep1T(iJw1*)oWYFH?Fp`*_IX@;h`-} zCJ7%0(?kVq&VdZOHuSwP7(G*euAWCKXm+xOB#TQE_nOke&Q~60&s(UQ3WOMyr>EU% zm38`N>m2p*(0o2%1>(;xD|Wdfl`=&@Z~*1+{aJ&5(s}=vr3Kr^@GKl<&(IK@e&c;6 znBa{q^p|Y{4d0S$iBo7-um0H7_;MgvIK|qu=P1S0UQu6PV-CdgI^czhvCWGYw$Iw{ zbKmEd^`k4c&r3vatfZ5I@YYLtBRie@YxYcdrZO|XgY7Xu@L7a71@_1nO(J%mYYeC` z&<#e$lJj{P@41KlOskOjp@C9?R1NL~S+Tn@p)D<|d21$$Z{yKdAFR%ZWgD$!f)S3E z+IJr9lwxmNbGA=GW2UYTkGB|9aYBdC)F9pE8Ot%IY?Yf)`DJxRS*{o~+{2JJNg`~! z9Ptx3bfyd>W5zisLPRzQa*o-O4DtZd08H=W0me34SUAMOqAQ^0dy;M)br~tHSJDoT z+X7Oj=b!rF!9Rg%q3`<7US0;6p`ednsHn!r#=VJX$YVlyghYQKDmeK5+jhm2U<3e+JOm;O~#Ea+EE7IAK(2YCoKf&zF}TYdP6 z_YkPPwwbS}h`mWxUD^yu;Zh; zu10iW)3WUE&z?SF?@CI7U0fV;PSQVYzbXUZlKlIM>YQ@hhnW1u5)c>(^>QCB-l6^6$`L+f;^FyV1zejq*&40n-M+v z#!!tdCNtppvvMAuES8|y*^vx;Om-2k+a^4Gt~cFg@xr~1_dQkbJGF>F{hAs$F=Qolp z%5Rl^w@cZM$~lQ&h1F@^3peF?a@)T)rJ9u2z#mU859;rJyZDzr_mZl?>fca$LgP@O$8ggv&;+4GP(mUY2G?M{>OW zN40mB8q$%!OcvMfTqd}GCNVz*$8U#q1Bh)_t4%p)AxgCu9#Q-4aAj|mEha5c^rrU5 zKThl0KyjeQAOA__c^iwg=kK<;t(c8l=8n?Lj7%yJ-PZSLSlDa2`#}1du__6qc|Ioc zxEdVJu@DbD4>qW%9Ebq+J+ShD6%%C{x)nJ+!NT)$$y?u(QxYc*KAf9>1vXIJW`SLJ zHOOPm{i2fNdwJF=LBw>0qEq|dC9&El(t?Yc?92^4ps9)XLjQ2@J{N4HiDMgs9``mp zL2MY-!GSuALCAtx?kcUa@!SSHsw`K(}LSM6ApQT1&Q%VDnWERyLjd)T8L|1tVCxNG_}m2iaXWh z^U4Ea6Yk-C5p{Q>&icxqdmtq&U zJf1m?uV5zcbBSRZt)?Qvm2GYLK5WY!yBcg`uyb(f$5K8s!yr;*;P>Rr_|()xUeTbn z&?R|dP0{xVt7I{{7HsEC_cyljJzZ!)9_v!*wh%j;xd!4oQ1r#dXnRdDYApfw6rEg7s}S__PfgHxHtm~9lo?P> zTcwMK7!%omU{Q;a&TalM3ZHcxFMbOn2JfDmb0`woz~W`51rBJ21~AZ3FmL%d)tS(e zw|07&j^eG2*OqVVIXD%vJ^?4B(vMad!mJEUhK|~t-jB`HTNB~Uk?bGC-`d!6b3rq3 zt;PNwAL{H9blK(VU^QDVWfjQlmWR02kxxLU;5{jESHwIgS=`Bb|H{0jrlsG1a>tlE zJIaDg`#v`m*qPpv_Q3D-dKPeZr9rJ^qM!dm@)YvYB#?Z?qxo8e=9xz)mgad|%bB-J zxpttBE7bo{z8jA6+$d~Y_7t^-a*(g7s}ZuFZU@*Cf_W2AZ1|mbOJ1p*DBoDOw|Zx0 zC%BydGq-UR951eKK^SE?IUe}~+HYX02busAl&Rn?$Xcw5u7{egM_&mx zxIo&BU^pmxB^U7bU7#F%eHJL{VysR6Xde;}#7EQ0fOezkwba{K{>6Jhf%n?lz}k<< zmhQ_wb`}^K2G->3dc?nB1UjCKtHYk+0!PgWo2Jl$ZO_N*(23U4$6!nZoCK2aJuzr& zpcCZFHUTyshWMa(t~4hC$>aXFRN)wNd^=WI+iJwzyA&)JJomE7&L=eBzoR%)F(+6E znhmKQ%@f0xG46J9oto<}3D7?e$44uGh>+6=%)N>= zZ{1A5IF|vGm@%{1CJ?3HW6lH?nWu*`ae9IQ?ioUJQ(h=`H7qV z90R4dkpLD+&a`ja7lk~ln>EoP1kx8dmIVlYra9`gzBxpahg z^{yGtmUtu{VBOUb*Mi?jk(X!&fG)808xtQE0C2JL?@l!oZN31Vdz5i6mOkQytr_X} zOS0m1i9h>1Y8foABn}-fTmgg&@Jmc=j9@SrWABE5Bb;7U2*IYz03;yhvVMNjCgHOs z@3VA-Tu#&n8ZH97ooKd@oq6$rN`)o;#ufBcx|;x9w}dYT$S5G?=J*TP9hwH%5@cfxjeKS;=?hs<8!*6v{>?aMT$v6U$$9G#^OHH-1=I7RQLFY(_Ebrg50RHYYf1B#{^>ja9O!z<9YX5gV z{|jyX|F=&j&}Cgv)ezkk&>%q#}7R@&jzb0-0Yyi|Oa+}pm{r=6%uOSNzwChWO5Z@FZ zHHa>xZ|H0z3H=}h^$_5lfic}Y_sfoh+Zp6b&o|ra-s<9&fO8t}i;H%?WILS*gC-wJ z1QU`}sgK0ZFG(1lH$39tbLe<)IayrK?@LBWowtEWg_fFz3RBvFg!1&AvDlRw=^0YvBm<2qHjm~tI6Wq~UTZINjlVe2erOan$7ZooN!oCF|6#?;h>_)#+-_PrxV z^YPwLJ$7Km`*Og}Jwq;0AuUP}u|_Lw3P4UPh-GgHK*C$aF?0$SG{!zeCj)(Pe=neJ zCms_%4hVGCFPxPiT7U^{fE+7keqdr4Z%vgO?=3N4&CKA&^~(gn8;ATZW|_8@Ap#C< zN1A35fQr5N-&q!9vA&LopWgcg3gl*2&H$D@4ZlieCuF%j)x1QBs#UTQzNIkW?{HII z9=OHbiX>3QZH1ttp;NmvC6L>E@LnkvQN z2FxnVj0HD5v|T+N`NYf70nD!3l9VB=Ji?d;7$7be&B=xX6_F={;A9J{h-VA!V!Af{ z6Y1%HIJ&6Nc5j@k-3cwXOQrzqT0c?LuwG`gM8833`->A~u z+LEOI_xR56BxjahosSFd4nfyYO>LU1)gu-4vmD(atGsu%K9<#bB0x|eA0;+sflaID zBt7>gshrp|B_-XbUKg=xx;Od#8=5ASkwZw*Afh_(nO@KKqt%gi6nU%f)(>k>EF#eN9}e?I+)tkh?S|05r;+sEcvz3aJy&74 z4Y$od2Y)(7+p)v)#P&U9JFVDbr-ZL6;k4e3UfAQIws9-L8A8{CG$ocMQ!k<)pHd{n z2lW>heyi*F@SRG!q-WXO+@Yq%Ce59>SHEaE6ak^UBu1w(4u#+*iy&EjF)_++bC34t zk^-6uN>|NiMQz``P>_L;{XGKp(tDy1UF;vv!p0rmW5_Q|Y%nDUxVOe&J?$$Z`>P31 zN{(;s3Smdsoce#ycJb2^CtFMHH(|L&PR{RM^3iMiz0=-r%Qa->2$zyJ1DAyaBxlq) zCVd+t{d**=u;XYD%zBLd*r7~w+ri;kpf7U15vV2VJSDnq1;c3{2M6O?&LU`W+APtq z5J3^ijXkSYN{;2Ou5x6cUO*xF9eq+j){`p0Yn3Iua6;MM@2IffzbXF8LS>DS?z|-f5acr*3(yPi47^U zYkj@CJW%;m=`P@TiEaCJ>*%)%pN8jCwRM1}k%)e`q7!2Ln-2d^oD>6|@IUT__GPgt zT|AeryZ1=b8|5R(tdg4jGcWSuP#v#3(||9?^8`u~3H?&-Bo zPx4FIGDSbD;>;TwHzsQIrr2ysCX)+ZoXZc4f`5B&?gcT-{dnbt_*k=d#`{Fn=j2F) z!8J19j^CU`pxgH_ZMrN~@m@IX>q^!1;j3&-UTYzWO_iZ@oxeTQD(J76ij4@RCnh&6 zDE^v{uEQ#Kl}zT=f|ZDdRJMmsTvG||KS;8_!!oD@Y?kk2z90wWc(=Rg7a3xB-mfUI zD)B&|_0LF?$efvoHo-JUzb31dUz|^q{I$(pVP2|O*yRf{t zyju*h8~4JG9nq<)<*^UeuK%ogctV&?s;JFq@#5Li(Qn(j__& z1-Ut0(#ag?at=# zCJI&%(*RU-m*g>vBKX>jSwv%PSwvLMIj|^aL5*-)!r6x>x+E`IU>pzKjyS1uDY60e zT7Cye#FvkKAB8kX@~3Z>0$NvN^?+?Y^9t58Nyb{dAv(m1WqO6d)!jVfYH zrGSLWWHDKm8B%g+#2oMik?@9aPms#ryj#o`Kz`{@XFdB3&^FwfBfKo4_p49L9(Ya5 zIKve1HTr^&X?$WD*Rz7?(w%Hk*d>gxeb9eRYQKCEv z9S_B7z$Jaz1KP{KqAM3{v~Yix*#-|O?<{f&iadb+OkO=b)57*<@NJmDJYR4X44rvr ztsXWluzJf=M5%5z{gJ>R9)V@aY8+Hft~`A-e)qzJ6RQLuj_Jcx07nxVq##l8!B#}P zr=eCar*FtVN8iD6-SQ|k6Y~B1?C8_64M#`YBMW>$Bf31lAm*f5Z2#NXV;ZZL{dx9C z@ywu~$gw9#DBec#9ui=#=4AtJTXOH5lz1eP{u_cKPMO{=reTTZo%_ zBBFXXGn)!=%cKf5rcKR`pg&aer%kI{>@5qs8MThM z2W8&rAB0j{QqU`W!7QGrwHii%Dwb92vNhDsXh`y(T{!G}8aF4DbUk?}m`^Ksg$#QhyEj*i64`RyUN!Y7cnm6`W1Bl6J}cZ{~xX1wq2sFkyf z*M^q<8%M_%@S%MPKs21;>m&8Cw(cY(4z!ajINHlk{Q+ZTJ?ngdK^!M~O%?S=&}M0E zO6E8XHGi0eY7VR;ihFL07dliOp6#NXasHl@^9KTNEeK4}-M(29c@3-Mt+D3trYC_* z@q`1n8g*7Hsj^rLDvpl8JZYl;@feiI68A~_k@(lZi5lz@=G*F;E2RKor}GhFyt zCG6k4KN;bQyZfKv*-{061qi_5aZwfA-G5K?I?(<*@?YVS{6}vYWdnc@;BWx4C!7`_ zc9*HbVK)=LT`5OxPvs?Oa4-EG8YxPituhG!)NuwOAb>Y|+jnx10q~C7;p_IZY?XjX z8+q-_!;&@K#)5<^ku1OV6U%gw^6txBa7~)w3|%`rX84Z`_pa6*rN!;5S!R_o6SxnL zIh$XwrWE9L2gZVV@885+m+*D|XYN$Ht(~mz!0VQDqZAlcn<0nj7_uafPtBmKcX+<@ zyZIlQ?UG5yXww=bp#MiM$r?gy#uxf6EC>Y}RK5bls0erLAHOR-PQ68icE-J=_?}=N zKB?<)nwS(Mm5PxK!@2j^PGFbS@K>7x$gpLBheO)oID$vsD(JWkGqB)hZ#yFYbN&^A zf1gBefRw$(5{ zW>NEaZ6~e8^Vg@0clkeghTuK?BjJM&WQtD_~93lqTmmps9E; zbV_R0l46rzyeSG7fUHZI9<9AO%Omp6QuXP3&fCT=LB#vmDW z5uHqsK>>W&7iLP%aa}Xh?pHw;?Jv(4sJk!e4L~m$f!#i!mxxh|^yxM2Fa>gljw?Ap zBADP_l}z~JY1wmc+sV@AFzs&sW7VMi`ANNRiB}u2930_eRi3p8X64RBpg-EGx*1

M+kOT|Rcn#gveE#$akqDS_nOV=OkHuBfRr{-cvnw{(pgxCQ?{`wI)=u@uz&lSM* zdz66hJorVK4|-QNlTmByIjh#bT^uuzNM-g9NB#m%G^?)-OUV(gnCm=mK+ZAYm@^dO z=wD;H4^pk=Z=ICixSgnrxVR4xu%31%OTH(+J4ZM$d}C6H{B_gm?dJCh-#YH7zDPS=eAZk74W}*s zSmr@b_MRUIAc^r8m~uFF%k};4o>8?MS5XwORtk>CU4k#Ym@<9Ooq6i2eGQqn@RK`n zoi~+U)Vh=njPC#RD!(@P{oq;cbJi1xbr)*jCV#~2gzijJ;Ocva8AEE|sK)Dx4R(~x zj}tQog=%0i28}sCO{Pj+t)I}66kkVfKl>ZU-&1I=tR^+>1#yVj9Hu5t+Ub^Se3wI) zd?<18HzYckoIzKJ-sNl?E1ahJN7ve+J%uIi7#3h^H5UkFG=b?g-FM1vFeG zszKC}EQCC7{{5y20M3Q}mzYY&w*Oh)?six9V<6=Kf}W)Ndz1f()S$Bgu)D7Z22OHr z=+H%w4s%?5t%U=vfilpfG^eCq;|8=I3Wp!BCR`h$_JL$3{6*w4M)}VYW5d9M1qS^T z`0BuJ@r%Zzp^V9^DPMr^p2$`h!LZ$@y8Jii>355g|7lqGrbAL#Pwx}5$@Y;8T7QM@ zX^l#l7tO_xyA-SP@y6sA?%DLNmzP98R@8Vv`_je#kJmY_zW4)e0;I%T;5k?9$o>zX>h3l%HHnR+*Z8%zA`EdEQyWugf)NZpp^$KAWfPd8$ zx77>?CnawZ#=>b|ka|!9yyIJCrmOtqLX1igBmWi3pA7w2nj5rObL%^_ksoB7tB!4s z$_W7~otD(m)`gX4 z3&_40y;P{^@KxUV(rpUa@WQKOk8hi(CDGoK5o!RHkTSG@Cp+6$vOzmPhT)N2+(}+8 zIhCr|a!KBBKRcW5SsyWqEdTc2jP2jOvT0~^?-P2z0*8lg{YCvVPm#nu7Yhf5tALRK z-OPHS_7H&)fqE~W%Cw9a~%lDq#I z?>z*&P*@qXCLeld@$W8e?*nl7hPHgrH zbpQJ2{rdmL+gpc4)vbTySO_SBARwS3NFyNKHFS5EN)IKS11c#gEe+D$AT8Y>Eg%d^ z=Kw?JZ;d|BdEaxs=bY>J$9sL};sR#&?6vn^>t1)?pWFCBOk3f#qU54K=Aoj((D?$; z-F<_7mR1#>46TwPe7jE2{V>Q+UF#>y%jR+CPjXr)miUC?!mooJNGs@$p-g^!D zQ;TwIl-hLPy5m}FMcDh#l>#1aXPCGC8k`e=^<+OtY5wh%1YStbpzE&62B7w#j{#e7 z!zos$SeFd793=e&gn&_f4VLo}lnsxOwePEsyoYdF*;D?ToqVwy)e;-RkuS#tH$GJ*dBQMEkI{&-}%uuP9R_g=!NcYbffET`v z%xd{%%=mjW6g-gH68rJzC!AWbzwbcC_s`RSb(5)k2A(Dj_qR5WvgC4AEJ{Ca(coY2 zb}FMqotl=PCBjPn_E)Az-EEZhm6p)41NZizT&hha-)ac0X?9T3GTJry7h5i z&JR%Jaj^k@Jx7!jHv%0MQu_t`SDZ5V6R20O*@jrk?ZP|JFE zBwdI0%)jBk1MTuT#cLZqUhYn%Rq?~q8F9^X?#s?na^`X)BXo@%7H`@kvGa(v3s&w~ zF0XTM)o%q)K88I?{_;kSXixwbLplCA4}my&bg*CY7Ok&L9D0E$69u>t5oPt_2V2W) z(*vXG7-i6A+mqsx`7FK!_SJ3UE$vAWHaiJ;>@=|{jyZBAqHp*^9&oL&zuTz@r2#U9 zU;hz%B73bFlA)m0TWFChUsctG#&S@Y-}{Fs#>i7Y%#oMx;y>zQT^o6dwLA0yZ6J6J zrbFmBkOLsCAfHIqfWvc?ZRm&VQ{UVx$}C8q>EtQy+upNa_WgoSm;;o4U%$L`(-abF9_DS|4#BHN!I_c_3YPSb#Ord2 zsxH%yydJxCoF?7%Tu~?YwVpHxjq}t!?uYvQnq!7zl3Ve*aM|VMBzqbC)a|J$Zr*{B zK z$2-L($?!oecbtNSLzGRiHHzqfz9*0sGtLunaqP5yQSYz7VX62D3il2)b>G<=YOB6{ z=KIEbMHFVaal5%#%EPgFn8t9%c_aBg;pr-3e79+({Bs7SJ8OoHleNcw9?vpWUei0% zDbhReAVE7VjN!#TGAmvuB1NX|O)9Zy)yL$^mZrgc-peid1bHq#2i9Bbaw!(8dJOYb zbP$ZbSZe`qE&ZS^cE>jjN64Br9I}VJyyK6q$`|G%{A|Nr?cc2~5idj)8;LwUQ!K@%NuNBf4m~h-I?e zJE=RFYz|@QJlKz^QVKa&3lv*j&I`0PY+l)2#UUzQ8$F+?ENOmOmQvf%#vv4d_>q(x zV=`B%-B@#Fwy0M?fAe1)F(8-y)ByARFzVJlXHG~UFGsRLGH;kWned0tPk}mQ0XVpl zdISLdI->Z+!My%`{9D7Z3A_XNp9vw4jveh`W~89jg{Byb+*qY&Y;Ml4L}_oaRW)7> zYGd^+-7cAJk_;XA3#p+tkE5l;;Dp-*&2J)$^_!4ecvUKNzEbtxM-ygM(>Z)^B)V@);v@Ur1Kzwmw4YQV0T8pQS`$VKT`gQm*pAr%x zs*vXcZE*+;a$ff0g>5TFUsrEAO`{ub^KISunuxBPOyGr##C$JRu&3cmU>iKLxoedxKPWtG!dU$-2xmYPrhK%wd4U ziuZ^%lhQKOTboTcuHs#?g2CRUWpGMf9xjFBhET$40mIxC1gX9;gAg!TRa3=?csNf6 z+f)jMb5+m4etex%#)vrF@!AWugapo{HW|j;j&a-J?ype309o{>dJWH`WM=iC&FuCs zL3%yZ8?8gyN35lljlAZ;b#*#y4QsEK^o-{3$A-E*RI^U&_T=q3QSJ(>>s2p}a}H>( zgNVr^yeRXhla@MNUdCP78N8fy#(sZ%#q@|AMCkVU%1_zUO9y@m!H)03+sczo;WcM{ z0=njc)=HzM8)pm6Q`^Yp{Khry-nv5$!f=FOo+MI6Ir82Gx^C45sIb`(c={S~k6cW` zRV&5K-~4QQv`|q<;A-dVUE6UzRcX8;d!GKy3FW@3l`1a8XoP=#m)EZ*=|s?Q;yA& zu6b{?@xYz<4;GlR9)UK@xfCovho{HwRus*A15C|~&2!AmjEy@aEML!g_Z^bGo@)nr zTXSQwY$>6zt8~-Q1T1nWJ2H&_i)57A5+Mf=^?AMTUI+**v}?newsgFXraWY_pacHa zWQ=DgGm3*Ivn794;QPqz36MIsVC%nL)|y*QLI_jGn-2;v zi=1*iUlh;WUcd^$X8e()^M!|?ycM0p+6xAgGI=h&P{-qA*BuhV!8|4-UhF3KDU;b$jM-QQFn!l0?z`uMX;<#SDKij zV!fJ%UP$gVPBz!KNiOymp9Ej|`o-r}WpxzxWT>i)0_Wy391RWB8;APUVpW_v@&Qvp zjRCKbX2>-9fzcE4{i4ED@j(X;R*F^!7=6Ub^UqtyBl`@OHBV#2Sd7dt_&m>3uV}*%_3%=`Qk?Ynlmt;pU-6URf(;}?GdSAw@`vlXD>}#ClBdu z*>fsgW@_zeYh=fnS@WolB}l%49!`$vs0E3b2+nTT{In{?eZ9M*VKg(&aQXhQdVkXYz6CLvCvlFzDd51nzEbFJ_FmD^ zvq`BKSW6FA&!h9xy9fOl?@;w-Z(u~o$6#;KEe0=M&J>!EoN{P>K_c#{M~qWv_n2sb zmkFpemWuGp*OOT}_hR%k`lc&Aq7p{yI9b`En|DU1E2J-GZ!6M2kddbbxB&9r;*R?tc{P~H$O2dV>7nSB!2MO{wg{~T&`mPRFnx0WJ z)sbZx(^j9%@p-YabMv#X2=qxA4o#1J9hm!U1nassukhSKmiuE6UzBK4S6G04TR9UQuHW+?MIB`_1LjBK(kiYYt%g(fO_mUyfO z8<6Iom@a><{~*rauk>x4!8^1$vvt3akT)F~U!)Cwm2u%>1%fC!2K%*qb{{`VJwm@5 z`n^Kmh4jaA6X}uP9~UVYzac%Gl<>ljB-Y@$!XN{bFvQTH^o}Jwr9UVQdYvx`>02xa@aZipj>^T!Zo!~SL z6^OLell`(uQ(ke0m^%lTn&}`!|HsHSd>|yK;YC9YZfiqcTzW=@b9;h{W$9|gUV|~a zXgXGy)cL{c0gKQhnL(w7ozXgw?1+B42Lu8&bxu1K<>lUFGsp9O-b$N694~lGK)8{r zFRsyy0LRzerQ0kw;Nfd|E>DP7)Tt;4= zn6@_D>aNNFF2(UAH&tdtmG0x);p~o%vE555qN1bj*m|Z-!{?Ju(*_*)Tu0P~p0?|A zN>Foiy~(PcNQ710et*@Ijl*Pt*dlhz?^^VeT#oru^WLfElHL)-q;ap?o+s2mPY)D9 zYNG6-ea5uYEc~Q&qCKwz{zO1^)Srxr!_HySz)%Mx=T!#q+LsMu#=&GYL-6AJjZTI* z^*z?(bywpPrVrK;WmZ{YI`@EE(~RMY!dJ#LP*G8F6H8WE~>vEW4rU{y4R2JfD-s*904 zhi4|cP;4mxPgZnYOA;c-rRjpg5$R4YwiRbq*YBC}X=4nwSx=oKaC<7~Io!^NwH3lK zR8?ZWq-S%4+nX85X>)kdux{1ps)=ED@@gd=8(`wcP#l^ zdJ|_rA&=X}h5*lvMky*hd{H+652Og=dYFZ-VuImS2_)v?kT><&J90>1io9j9{z+C# zlqJ+@t#CpNx@hs5R&jvSr4FB~vr|SV(q`%7muPqxoS*2?{rFGRQ8F9T2?xl0fjre4 zTX7Q9*;jV!%WJwn!L9h)3k*$9lV%8mjWKa?uUe;4f?q;O6t1eVl8{>-x&QW1%d6a1l6`H6Ok2s^MtRb(Ce9#vvYsX zv!8sNz^Yb*!aZKzg1hGF!JQ^_xdfBQV0%*Q7kpqR(vL029Hp@;o|1C#JbOWD(_7M| z;>bh>me56ja(l6;RDHB=Jv$<4zfglF$4uV~*WpA&&RlQ({MvwhDpv0%*F=h-8=D*k zOMd9gub)MG#gwT^24m1W_6N$FW9nV1Y*jOp zKj^MJc=SExEiYX7vQ%PV=MsZd-OMgFA8k9)rF>|!fx3!3Y>5Qs=ToKF==&arYQT(8 zl#iBLZ!G)zrP#6q?m3eJtKZ+$WvMCpaXo!Ga`-}%%wmqS!04IpF}xE=M~o#}yEzlb zTj}5;HK68IHtIdJ6kz5<;d}}wo9k7_V5yRA%hM-;BIg=&+ogrbVksW4{79zYTCk~;#vjd;FF9gbjT{EE5?9Y`F*2X2 zfNiib&Q8H{QPr$KRI8f4%kyjqRF7KFbW)lb)2by*Vn1f&U{-+NORYU^_rCICw?8$3 z66HVph_2i(_o?PQLPhRWbU9dp8L z<~UuCv&5SXCXg2&aaqZPO=QdX%V0cnKgmE+we39ID_f-l=LU#kX+G~y#I%S6uDG-Z z&vk2RtY(`+80=Dyn$8-I==$yN`Re%F$5+Hy9CZ$|)p4-0jWj<_QR?mo!UGhy>0InF zcVQ){?mDtc52;|0%V94v>IXga$Pg=Fl(Lc;jH0z32;D!hZ%vFc8GLs3Ig0Idq=RW` z&5Mc57CdVS=N84@5NQSG^3`9dRw-3^ujU3Ad=Ec7O*kBtnQ$nbBsSyKjbT)^RwNKR1t( zshobVB{X%pnhA1as;M3Q1Zw9-I{UwMJ~#94Zm(S2FKhtzZX$UPGa+Hl>hsio{KUuz z7bMoMVYd7!v-5gS<}3*x0>_ooSIc2<;#(|JM?*egZl=XYR&xC7t&fdl-%8Z*Kd7#;fveG5TQ5Q^ALyEi-1tJa9KO(>ZT{Ms|w{&^gF%>(~KOg z*cPKM=5Z_NzSwi_A-=}(f>OcdloZA{ob2Yb(azsgtU05~G-*2DC=)Vuwp~n86nZ7CsrzwA zvL^m;X17vq32yfOQV>5gd9HiP-t?#w5}0K)V@M9?_TS zZ?7$yL*j7eZsS@NmzS=GG5meXezM^*ikIU}6&u&LDD8(EE;g&jeU(fd@VjiHCvgkY z<1U5d-lup26u}A@!xF{->=&eda+Q4{5y7glN;_ltf6RTvsgkK;qcXtFGMNvm|~ zrni!X=C>{l7}uv0NZ0x@@fh%ZPQy4(-NrBNt?I#)@A;rsf0#1=DcBpsj{*PuiX>MS zac>YQkAOS^aTnjr%FgKW=1Vel`lR_;eb{UDI{pcQF_W5?S3HHrs*Q>uah zgvkhUTJ&Hb<=H8-ga%gM9Ebbs&Aib7k`ZlM)HUTx7H-<%4J8o}T^UyuDw)1$fU4e0 z1G*XMo!Z}rc3uyQV!S=XH87@rzQ=tXd>Iw~m=*;gr!g;7F#KZWN5T_86nqmdBI-JBMF^uz}x{b*++(H zA9+iIE`(hbtlWOy@1ng2@-l%a`N@hR2&()W(mrV~M4p0Bm6&?4pIbOz8E)>h)6;AI z6dlMesth!9%=BNT5GJx)Ktg-8Xw+Cr#>m86jc;_pJVBnAjn$;jg`Yk%4hrqKH=0;AnDN@Pj1~zsdcRdl_ByZUkC)v z9Ibws!Hew+bS)3K$e5G(7LvTbl9ASE*8HNty`CFA@KgsfPLFRdOAZ2RK_x{slb}vt z*OfC?CK!Xy11FW0lhxH?@u}zyACx!#?wZH`zTdGgB`R>XZ;H#94t;nBnL^MGLB)AH zsuEk|N$M;eGR!hg0h;BG#tY#3Hb+m#Glqz0lIMc2h#1Qi7R6*87MWl>#LE!^Gz73X z2_hgYrRF27o~=9Pu2FdJ{|dz6TXMy3V--Mqb1qZVF~Y~p#OEw0@G=KB{cGWpL5nqF z0Q-;}5r6W6(rR?@h5&C>S^g{KA}q0RhwH+AGCu4+Nfr1%8pY-M=nb^K~9jy)VA1g4_Ggxmb zsQ^E9q6Qge|48Arr-iL>ey*)2sltcaJ02*h%qVt)=YhLB98vED?*B}s&z;zweN zm|&oiL)hOcU9gfX(%p<9NA+v@h6bq$^K%n$iw@ppU`I2edA^|AuXB1W9yd)86~9H(5z0M9h2oU2DavA9Lw(r-*Y(wwtqom!a6Voh&NXmf zYP-%j7Pv7K_*ivVyFW&sqQ+1wL49TnQDRV`x}K}FPkSA$(nBek?NzaZIx?1xa4ZNd z9}{f3Z`;?(N&_7prfK}X+_V57CvXhRrEvXibte2hF)Ss9>$(14KqB_mllkVTSY50t zwW%q|;vch)>(hCbIOXF|S=DD(WXp@m*8iO1vW2DdlQ4DTDhb0CH~wQvR%yYj>G3v` z!?zQM9C93~#X|RYTBe`X5)#A5krmR7LFdsr_fc1-SIP)*#>rQq9!vT;z7s7-aQyOY zij>*OHWnnB6Fo{m@P~iKho3Dd*-aeITx({&6rl4xJ{fOeoK2?ju5j8K?xQK06*@n8 zATZ@U$xzFy^;nxVD=p^1B^{8hGVefT2uP*+tT+#KJu&A}oHR5b9RX2P>X=17A4s12 z*2-+pT6~o?W02lVN^p%YZkUECaJv@`gn~3}kPPxL2KV?yIj>sCpJ~b$C+R{*;yjP{9wXcb>j&WvAp1~*lBSB#Z7ZCNU!+5)2J{hQ zm1PgdvVTS!FgoHJ{Y+J1`AtKiPv)mgAMni))LaP~%}jYm=l z7Im-S2vMLBJo>#=u2b2JY9jk2OXk9*6o#zSo0k`#Fb36jYw9X^pAOj>-MDy6Cii7; z!SX_9qS=wlYNu=Cm+A-y+g*zp^X-p^+X@rZ{--`OoFpUF5xA|)-21PUq#{x^`s*Ts z?*F0fTnZGmBlobe>=LVgfcgvD>NE4~yXUJ2J42gZz8sa9#?!P^+|TuK1v$Y!wPh_J zm&gwnWa@Y@%?-LN72k_ZAOgi0R2P-+$3oGR=umlKz;64Y1#%=vPleKl-+bq!cWD-#Pmmr$psgBjYBo(WPj;@4onQ@0JcNR zAzduXz2o`()IA>eSB1;6W*wuUJ$+EyQONU6D^2Z@m*eK!Z&)>lqE4nRn53?c<6C@$ z#zW)L9ZY^p+ikS02*e9EJwcg+Kb+C!Ei?i1R?Pzyf#1|O@c6A9M!w{4_EvjC?Lw|w z6)W$_wY13m9~mL^|1)XPNk{--pR5R_y7O;G1DWi+@m2Y@%-BSF?TrU)gnQAT`UHif z!_fRzm<-O)SYN41ACrHcV#P2ys#{}E7eEYj9cD`WAo^1=ET|N?u)4%jlX2z$C4y@Y z30$eIA1Ck$OS8R-XnwBt(+C^I)75FPRj7s6?6226&^VEaB*7DoN!P4$QOr}3_x7;U z(1>kJmLF2E_HeS0G^7A>Jw*|#bHvCLaPEH17yR$pK zU5UjEJ|BK3gNJ7Dv~j(nfT=2J9O~p~)!}$7q~SY~+||`(wq|=K%ueTh zZ;@xMEnnWjap=y4*I*&^)FZ4Y7`Nm_lZW5yuE#Cf#=dgF+UaY!P`%m zTYBV{4)yvdhn6xDGDTce#X+LEXAfLf55nf>^v_KOj`tawXz zJom+{k8p>@(zkEdQf@GjQDI*fzNJ>{@h5Tl6j6&bG94s>cp`hQo89IkZwJ@{)ms9m z8}LK|EI$Cz1~6mlU7&5`x6PN(m|x(t_TN3A1S{GX`j6aC1^JvB1Gf)Gcfa*sqgSKu zKa-`zS5XAHRjr~YXgWM-26E5)50Hy{%Ing<$tvasN=Sam`sj{_5)xosAl(#E)F*tx zZU^2H@Zc|doX}FKUaR2!Edra^r5JAKhV1(2Vi~@)mKMN7iEOOw7EeA3G_~(2f$J$! zMTMru;Qrv6lKjY1lj~~da$2!D512s84RI_a;HYk+=|J1P@Q&a0G~V<8TtIZW!tte! zH4h@uL1viBC)AAqxo`01dH-2G6D1+u%U%|2mCV0Brq`o>aLbVZPrp|HTcz~x59O)F zYIA-o0=dS@AOTFR2enzG|7Z_lB>gjinObl2AKXD+u7B|RqcP0C@)713K({6j}H__@JQgd+Ns>9qqr1?72pE}L=JwZ(RZ~rSOjQkdHxv!{8pz&DF}D6by@#> z6vKxViGJ-PrfeMgHE>dzeNM}-N}xHUqO!TDWKc=AU~+RQEIxM0qS4YWg3vyYpCHGR zK+z`}^KEN}0>_Z=KSnnO_xAfRP*8+9x_wU!1Z7ATf1ujPfnB7e|N7P3TQ%70KSO}; zpv9L4W{Q#A1S|C}i1ZeC{_ej98)v~@PJn=O>g2RwHgq5zSU7J@`KGX+ypI^rW5fMw zTO*gP*C?Xc&r@284NY3&D7A^FoU1B$u~i+PkW)s!Uye(KGYB>6lN@M8#zg41NlazJ zG*Ww@cSy;7K|1;gQ{>^YjQRy(NBq0L=L?N$&^G~BePo^zyV>mg;I6c(&grHJeO$JL zP^0x1i+Fa<=MKlDCxa#4#*Ho21dzb}cv*EDgHDF!_0;c$E*E;g3o)48mQJor)#>Hn zWY49;h#(j+(Sb?|Gg@;qo?mE6-W7z%4h28^%3P3 zx2rz*ecmu9_q`nU21)YBWW%AWKwOBmMTymfy0vMZIY@LQ54{&5(3I>k_%Y61tM>qfJ%$yiJ%D>@?`zG>HNfRQU_#0 zYf1f}V-4CT=RNYKn;0g{=)rndL_()YmiBkugs4;&D4Tep|4owwuh9OW`_q>flUDZD z2*XKbh+>M1z{wyWqqlplX{^G2f$zb1Lu~l5uAhi+WF!&MXDwev(lGm`+g-|T>g9o= zgXwPei%_!iEhusE83eJ%vXI4`bAe#4`p^+ImisXkPUEYzK}yFP-p#DB%yt7%Y{d2t z?@DK6F#sS62@T<`vS6{%@)i)#ZgNQ3*`4*+R|7y6ezg|i3f2LK8%%VrJB#!mouN7V`AjKpD%=~%CzUk7n1=!rb!K#*`>njdy8y}#^Ig~Z?;1mF0Xl8 z{YXJ1fnuohYOlUhF9EN!jlqSGcqy`pXA|qGQ7CgF=zYHleVMRQ(5>LH&U)nn++0P3>zg^6=@aLMkwvG> zh~z0IUtE>E&s|?UYhFy=1ykKocC#ipCOCl6<9Ht;@9;3;-4+&V}}@r+pOJ6iOHny zRkXr`7(0x=(r<=fGI8Ti|n>CV@xiFR6q*HXSfgs6}Ge(AhG;ZG9wjlYJv2B48 ztt~qI87?Y~_$j_1)l3iLR`R$^xPrg;bIuF6u(YYf@~RuSH^R#45NR^wdr9flm0Xvm zv1@!VW0xDz1NAwL6}&UOB5ZBm86Epc9}y&uW@(o8FRgl4m8Mhvy$JP;tTKQjYIt)ba6ri4*N^)psA0W+4fB1!t(6xL=KDZ zKQey$NWKF2ZJ_dJnahj-D3qvea$b=AS}^M)IK2aOaQi2-e~1q`FR0&zLssfG*#b;x zgDRW8=rU11+)fGq@%(mBH<4ng>OXP!EmjJq?70{gU{zS1bDg}rOa^D1g5=U!DpNCB zOUXbVySF;!2EZs2YD8SfFvtN^T(L%n_@C=jnrXBN>)EYJuBqb-aP3(>* ziU5Q@={9$9Cf8quB^k+~^|IE$M&48O;WxN0Ox5d4@ za=vG5$X8O|vy*STJ3@`r+aSm_01n;RpF~D-pUQWq5$_A4d#!yjdU34s@mYAXptr?x zrtogED*PC1Oear?B=TP-@8jybrEH4 zr%QY}W((*5L&N65c|27r^^`D{4<0pf?Bq&1}U( zz;0~>*&$%eopS&XLr-gQEa0Qx-20wcx||j8Ff%8T>{Wiq1K!oB^K*lfwXKoduE@fW zO%l#t>aB`XISt+K803pNHN=-l_2LtPWPg(okE1+z+mew)Gvio;g8moIdee&$j@Ra0 zXu0|EJ5#eNpjZRk&yFDp`=z@-B^x8`kTphDH6wM7^GBr|327|PdFXq>;eRSRWL_X} zI8d~jsP|$zXbXkJI;5D*OiJrFRl48XGtYDBLe4eQ}|(9wrsVP&Qfd0s~N~=NP!TyY(8DrSj^_ zDNNd3raIJtn3Sc`2uDiPg)BN@O{;&KZK9fyURpIK0Okk^i^_q9(1-pvmF=?A6Rx~fmI~-nBni^ zoYVElOv+8xd+J@an^rZw>TDPHgJ(-ULr23ZIg*LS1ecljTX+IdT=NAAibEND-IV6v#1jyl`8q@d3fC?Zor`)85(?#|YK z7m0qKoyfM5f;3>pT+F>gh!{}GH(N^~{UI(hCwHBrZ=E*?WvX`jx-m#tQ}+|xazg%; zrdL7BMFTY*1PY)AZonxqvNAJCI**50?H&p_n241!dJG4YLxG1udtMndUw^Y&Y1uC?(9l2axMJO@__ZRR#o=CKS zc&`KOCYtNuD!G_=G|zvmJv?Df`>(Z!mFohiy}F7kZCdx^C=LAJLZ&KxddD+9CAOn3 z%pG6a&Hl+UXgdk*?MgVH6R{};8w*4s=l6qxpJc&QK_ON%I`J7RCQiiK-4w)8g5A!! z&zc?kj}x^aSxSjI^ocG*k6thW$%GNX;}~{6=6n&QOUd#I0vA>$cD_@n=| zr>#aao%TgG&??i$Dr@c$dhz6P;xlbpB9vF*X zb^88mEHX*8GrDKJpnMc%Z(dx#4)ZJl7_tg^YWZqYCxaOp1)7&-0RL}K*_^wJ6c{RE zK)K9#|3qMjjRr04eJ()o0hQy+0|`G`E6UV#=EF9H+^Naw+@#<(kF3Ds%%6PQ+V4xe zaLoB8fPu!WLC_rVf}H}oFn9qVz^Ir6BYdz?_cA1x3w!!XWnH#bu! z&POwh5>JbrxJQ6G@fA7bn{^4M!&40uqHlSGgSN$CkEg@i?t&2-zRKlp(NkgpM!&;@ z86GlMjaY519mT%8&#)=sFTTV4`c}|;W`1mY!L;6aV~M7+k8H&A#^~>MPCk5)1rwI) zg)G=^uMLi^-Z9_b-Wa9tmH^3O>P?Z@`4j{)w>%rfviQm9tDasuEgCP&p_{SNu+gw# z_^L-sK0A^xh_!&H95tVAl~^`kZ3+N9n>tfODQ zQtiwclyX7>Z3VUGaAnL#%pC^Y5`$em#nrnbXpdU28253&m45$FJ6 z2Tfm(HHbb1{mv=Mp@oJ9VoSO17N`hM9hnmkX)Y}NB=GkE3{bzdH2Sib4kQ=@E)Kp5 zD&^7|EJ}CQ!2c*iv%PZG$1g{4E5{EGMGXbK*r+4GXM=hH&o_9XT;kvbY(^qzPaf^S zoh>9fGV4N(4}~HH4(5lwXMOck8wINbJnbl=b+o(qomJO#6j_L88zI~T) zj0`0L0pu0nEA%m#vKMsVjS@EM-6zZ{eLpeO37{};{ox(dT|5Z;jVJUkkTk&hN4QGfpT5B+0mQ1|;EX6-*7QquxH5OCVt{6&D4Y{`YFJ9^Rn9d!LpBm@on zCP$-|_*MXRv@;p+{U-!QR_5O`-_3madBds@s#Z|``8NiS@c8}5q(@UV1wYm&bS?(E z?yL-QuaNC@b#9#ut^s`RAUKz!1sn?caRjdn-3t#o`4uppziIVn^3-JhW4tW0!6BvD zu_`+Q29PT6V&8qVWi5%l{uqz&J_(`pZ*q7m^KZ=nMZ!oaM|eI5eiq)}7x^Cw`y9`i z;|4Bh?SWWb1K8+0-Kzi_*3E;Xcx_RsC@5IO1T~jx_W`hTx>E0wV#gy`t$~Jy#$5dD z`a}Y+*YUcO?$vZ9z-j`$G5X^q)NAnm01^ok{2RR+V2uHP;bCXU2TRT8C%eH%MU0lu z_4Z))GW36_o1O|F)USSV1wbUelrb_Uwni~7gZ~Ff4ITXyAgPzgkpSc7&GIjkzn=<# zGnT-y2EF$70}Hr(DDkAD{15g9>3TSFLv$GTLfA$BnkA$E7>omv^6&9?fJYA}G!PHs zXg+(s@74_Z1F*4G0PI!Gi^h)bZ)vrX8Ble-yYv9311)X6w5nQXe}cjLqTcdUH#ju( zE??Eb6TU*jCWkQ6iJ|uI<_m`~28|LfR`xe}stzDz_kGF5j_$AL3_!KnfE`tN3a!tT z$sdJTvzLHIdZm{F{s#!)~~fD&~~ z-uNScCxeF8t<0DrnyZ$-&F|_se;hB;Shu>?!@Z6=q{MjrXFCsqE*hl&^qI>0Pp;+yLuHS5Hd58-fe1w%CeQDo$^h-(!HngEkzKZo5KP*_ z53KZ>-&<*Ei9N`9%1!pQ=k6Qo8%h13v%7;#mI|tAH~k=JTMcA|{oT)ypPsq&-0?i| zJW(z2{XP9XATyGyL@sHKK4+ z=5HSJ-x>RVWKsWB9Q?mJ>wlkZ|NoXA|6UIMzk2ZhB0ZGc-DBK0{!MyCU8=vh7=q7z z9dt19%{0EPn^pGLy(WdqJ4V%Jq2P25eNZ_@`V?Wg1P*?wxp|L(uHr`sxM)CP|^Qncr zgy}Nq1%Vtq`4@yC;GR&g{pW4%>6{599B3d?VPkCuyjK7&*#)`)J6`1RB1cgyLG^!h ze9?R60Vs5UO3GJr5ZhyXvz7j}dGmIi$gr?sN^%YnN&^vAx8E{-6Xm}>{dE3s=ak;D zN)EH0iiw%a#<%=B0!=4kHEIsVizGu~Q5Vg>E(%gREGQ_u#N{d)HXz@q9NK1yLTv8s_*}p`sH>z9B;fK~MoS?}E|*)ugK@G*FnrCEhOb-j<*z-Yd{@ z!6@K>!F)M$F}m+v4S>?7i;&7Uvy&c!a?pToyWk|z|6N?*&-TZgeYTJE63j0PX1ePw zQ`5bRXSBZEO}O$8g=u4?E72Zq3h}>Mw_0&eNuWbv%OkvX#+!7n>OvcChkbgp{oeK` z>HO++Ws}*=_Jx1r)o>(eKmW#j=Unml{otWJ|4%kHw&p7f^_4jv{lrSEx#F{Z9$SN* z1Lnb4X4k8QmJDk9n>@A-$?dNuO!x(?EKHko9#<^dfm<$iPgy}-8dY%Ks0O2$S(!MZ-q1%E7LG6A}N z?i;hNzfzlXXwp+~&3~Ko{fbw<%MZxj${<+0mpV79*7_#`RG zKT~P7+f^cTcXb9CdU6N!tmdZ8eNA%? znsj%C6VGMnn&BH6ky;iA3e;P}k7?v_qwmD=Rsqm9Z8SSm-eZ41W8Zvw?#No1_r-Yl3O2K&ZWmC2Oq*v9C#)PV^ z!{HS}6H`-rM?wG=BSquTtc53)J z1H0xU-ADW6sTm%(aaQYJjaL6E#7^27+V{O$M|QX%gm$cEO;$hkkTIX=9BH({uXL_D z$`SD7qtgKfGfeFhL;<5||g&DRd9>0yNSFjUTq>yh}IF zVj4Xht(BS$=+=Ky6vzW=D#lOAe|QrzPX`tx;jCgdq0-%<<#P{y7oZ9TI5GRxV?MX+ zZl~}Mo_%X0R!0=#X=?_8iusvtA z9CL9w0}qf`v*o_eEym_hlrWgec$o?spk|qb)|L$34G;9(nOC;Am+|z3gxe&84NzM8 zg6wYXT38N;c>c?klJiHGp#%7wl`AAlaPoPceR5Yi|fRG zJO&?Y&xFlr+=06}EG%!m`g#8m>jPxJ=g!XV?Nl!dYm@PMNlkJw(B)HB=wyC03;u<} zVm$o}?SokPW^j1$Rp_+)7w&QDgBK}RJchBTNr`#j(>t#{JeQgr6GPy-Fi~LJ#OJa7 zUR{W3h);Bvn3xzeuq+Wg`}Xy#FvjkdW-90Ib|q}G$s&s8ftg*EPI6Kt)W($F7XuSJ zI4Epwys_YLblKnuYaaVfXVG*O6pLJMVMiM*vTW#dzXXwHDU!^^9= zEuEUpA7#lIJ}|opf2db$+%GHUHB4J$k}JMtv#PAys8&@?0aIw-Mod4l9FObyS_K zNVn$G42MC7)Ri@!A)aIe&yd^prdx1u6zS?tYI&Q7vjel`Yfr(YaH8U|v9U$wt|>{{ zWhV!n^_a>rhB$?*a?_4T-YTffB(qLk>r94hm$S8_W90z7W&#^*uT9b-vD(TTj(lUM zkeC#(tM1~e&~dD@}vGVzUMM6a?J7)meOUQZGFk;ExWIvPo<6b`3}+ zXngV7^^+9~AD?d0iHv4Vh{cJ$goJ<2V3I9jnSR!BSuZFdL1*iA^X@HLB~4A>=05 z6p|=>=m(Z@LkX+Z5+{Fn#cGCy;GLrM!HOA9i@Y81G;j!vKXsW1n+<`zfRmU|4Bo zGSt*UN5s6cxcCEfDCM<^&P@9{IU|bedtVg2NC!U-?4yFok|@RG_$M6j$r&FJ8CsU` z!N*)y3e#mCh8h!zXE;y79`PyH$i+_-B{U=4zO?7a54Ih@=Ht(K?V05Ya9x9}xSO*5 z{qrnNCKwaLACgcA89?&J_*@bzH=pW8FTZhwa~7z<@_Gg% zTGEG(m)bj6^H}JN#S~0d%35;Sn?J09T6}Q0@%Yln>3EBDvh(r*87up*mJD{#kL;9l zLK;(uqqK+y1JeA~;N;}iXvT7Ye>f`(Nc*a71B_tGkhYBF{!g%qqo0@ke8Wc=GmI>a z8eeS%xin_xobv0Hk5aF^@+^CtqRP^w0=@ehrK-NYqV`9@;-R5kLzyFJZ{PlLg8tK9kioeSCB3*YvUPph08dI!C_E*6MVNkw;?=@DFjE6Z+GBq;Z{GMl0TuVz!)-$y=n)7MB8r9`7 zdwX+D$<5r^76oD8%k>_Iymov0_3JyFLPDlX>l2UbHH(aF)k~B$5p&WrE_7-mBhw;+ z7isBmjRH&W)$UzEkyit=yhMX+zv`u(Hj|Khd(?`lY%i=RFR*1UR+>I}O*1QAez5AQ zi_T#K&(aWwofz5vx-2{{;3>w*$;pMNf3l#5SL^o(Yn@v2xCucACX>DWDVOaz!|FGO z_4c$G8BwvPl4CAN4QK`D_LVN{WTWWJQPGF6u){3|v&{RZza}0(lTdt~m9;ea|221| z(QLKtx~)o-&yPYIX}*yv)1{Qm6f%3_TJBZU-xxg&-3g%2ldPN9BPk8X9{>}HX3T_ zYDt(xhTgowA+^;}GJH^lz#|L%7L19wr3z3QSG;!c%noDyF$p>(6y@PEC@wEA??(Ds ze%;3KzInz2<7a${sj13*ANDsFhe*bf-tOhS-QAbi9_*X~+UDne@4cGd|- zhKADOb(tE{$RT?}t~Jg9()W+Te_Z7KN35O2JmG7YJn7fzSBFYsy$tSyx-SaxG-D)n9&;RUqb{FGLq*DtXF- zPws72P5^_1gak>chE*GEk^;s=@G^6$a&T4+GzF)A{aTxqrJhNE$K2GGG;Jn!RoI5M znVLe688q}WLE~&lT=!4?Yat^g<~{MU;b5IOaIHr_{fb*&84?%}0I<@P>S*QuEEZSK z{`&?5()8+mXdNY3?#XAjO-jN&T*ezMrzf@?b>r{hJ7-J|QC5!!zQx$sr2_3+pIi-`P+JVd2JS%@WuGg^47r@STCT?p(zc}GiQtwgby9VWK-wB?eveO?!r;#@IVO##VhY&gVwae2AtC9suY zqzLF(kzt`+Gt)$w?M-C=@aWn#_<1pt&hGQ+dKcOxt!kEC;PeY`l^xCEqH=n}OrY!8 zxt08@Tj;=wM%vKrt)CK`Mc{Nr`FH-dZuIHm$?}7(_#)K| z3$^kyiGj4f%9Qqew^*kMxl)^zsVeA34N0C+SPvZNOCfDiD7XLlm-1qITNK zkAkeZxycnRSi=*k)fvlX^YLGbagtX0;@CKe<&_mH6ujzFyj`v?%dM9qg^u`^+6?F6 z-@xAxiNp+xX}e2Tt^j1_1C-DJ>SQ=QXaDwlnY_c}%)BUWo6+JN*^OQ8;Jbv&r#{BL zPAfnl#88iKoK;UtKB$mFpUlq1BKoJyAw459#ZbYQCTJkXb%#j59rP#<2u)t_&gD#I zZdEx42hF(>9PcP=UxeWWmo*%ae{Vs_1|w~C*5`)ms|1Funy1pLb2i$u_%7-S8wPAG zI*wO%RJOm&t+q~-P;eLbot?cN+L>|y^W##5&2Ppcg815?&z(o}a${n${LffeOkZ&m z;$}V@Vif|^r3gSZOSJ{@nKyZ;A!OUlW1c$(eFB*;Rl!V@ohR!NrkvhiCcL;+j_+I47dHID zWx>FiDpbmCYsa-&Ty1anko`IJG4I*;U%V_N-c9=Mk&i^y+}wHsW{O|QB+3*4wdULv zHuLETH%yzTvKOVupwDcse{)skJCN{Jo-+G)+d=?7b##>rIN?BfIO)VX@a0(6mY=#O zGc&iG?qA+M`+KW5ZwW!3Pc-W_+q3duiW`I+qceBN(&c6rets)*y~f1}C*LEojkq*V zZFmljj`m{w@6x8by2>N!#{v2iak3c{bS55hH;6CfX=JR*NRhK$Ds`{qm)iQT>gsAT z$JkXe?>B&;pm>B?4e|Gu&-}&@No=2AWv|k#E&d>=PPG-CM&u z$=|Xp{NOU~zr?(~$D_c-mB)4y_F?r$R#r^8;;MNMAdp@LG(EZ{<-bdem)Spi@uC_U zQh+9{5;@p@Th+r#45=APN)vvyLgqTEV})AM&=dOYlVfa}rK)A%RB=Pb4dlDk1z|9a z;eu$u!TZfhmR7-z{xqq-#%5=KLB3>y=Nty)`MPQ|b_=Y$DC4#K%F3P7MWexC(LW%P z7WA5&uuQ&-&q$*`(jKvKaIvA}EIr)-j1Obx$uC0)gWl8IPCFXko0x0s9%6VB=Dj|! zejHBou-C>$T8i8!E+=RTB+JOYYx}0Y63zSGz5M8H%PbH~z$^)Jy)#Jzb(2>G_Q-2~ zSfCde=m_Wg;hIvww{PE@(H06118{YYZ&@T*$NzZIk&+SN9Whx~`?~%EeLW!TJkx=k)w(D^gBf54eyU{4&D%WZvG$)mAsClMaqxyBkC~Kf*k7rqkt9%>sStoM;^| zns&s!YHjcaI-1ZI0He;7D~hVn?7Y18Vr;tCGn(|%j%_x+dfY^4JpIb6m~gxd^4cnp zmR`wE`KoSVK|_7s5S3nWovfD^)DXvFo~wH1QCj%5RQWNzy~=@exJs5H628R66hE&) zZ)qXGCnjDeyU|AriC70CgU_AEaKd`9sniS+S`5(H0M-v z#qG?}PfMi8N~96`YT8c{Fw-^mfjavl2T22?lM9DY5I2&%ukVNJ;hI~#iXQj1EG(J| zVi~xZBTk6R83NnG8S&ga0Wg2P1!nCNwN5FTmQAI z>pJd)mh`P|_h<0avZ~4-Vw9<-(U!evy|v@xAaxDFt8{l6c%S>~?LiSETUFhq#lgx5EnJNErnY^-tywS>w-tr&ZI zU>WIDUCfR*(AXMCEeNL*2-K(MUS4cwV6ZVeC+O0PE1X2{o34)H45!V5`0@)1L>t{_ zDWsY=-CF=5g#kKUMC5hix3LnBr_)k)3a?c|bndu2(nn9QK%i~~dn{6)#Gsy^3)ghN zzcjy5j^^otyn0ww{C3k))^WPB-njxKmRPqRMO{}D!IBqGZBPtmIvRi=E}8UZW@7p$ zL(W73at#SLB&T@hbZY6I4G*oahmcq+?O|{N57Llwg-7+UFO>`%scAjG%Uj|;GLWH~ z(b1u;XKnl`icNyX(ek|o3WU2+%4;Q;NWP0H92|LhOAw^9sr8vibKT1Y~< zi#1n~YZ}>C2$NO6c7=&aj>e_T@2)MFSUZthlg_nvW5Gm>ySjk?2CbSH#!}(5g0Yfv za0uCXyV+=JxQMu9G*AjEN@^ce&uVSe&X-J+w&0-GE6A6=s`v%?5%5seq-q=Q$&m;s z%bBwwnQ>$?-}cV(64d?ekyv0+brw6mLGD=-17KE8n>w2Os zebVD=>`#7nL~*?W519iaZueQ5@$%)K6t>zZg0X|Ll<#d zHex{BeGT)*%JbU9}m|Ood!j4?~ zPZ0shF{!CeTh#s(Hg*=Y&>^g$QkgM`Pe<9;Y+}LtT*yDnbS3dq`)5MWpTnA#-;t@y z+zAoA5e_@@rS5GTr@X#%1H>k6{VVBurEF!HNL*#1rG&(8#Bs`=Yw6l_1bh@Wl|Q$j z%_RAASpJbuhvTbo27l(?vQ)KwSA z3~CdJypPA@`?+Vz7R^X@9ktwk^Kf(RM2)|$u+!3Zc0Xtd3?f_4i$%HyemIIIEwq&= zGx#0VxGrUvArN_a#`}1o0F~)FBRz125@f&K+E;5iC+_d&jE088?c0H=T!1mNg7ecm ze+c6ch{3T}5ReKiYi}GJzUSud9v^%`kIsb>uQ1N^8^XfMy7naD##(ZTJDOJehE0BZ z3K{-~RAuT&sa|=hVbNa!oL|2%|2T9lJ+>5aPh4FkHpA2JqIbJv5jd;3JyvQT6Sy#tW#9q&-7b)pN7EMwDIssESHiN^n%W#pY2c3tTtD@yskbo6 z1irEH!gz#p)f&)|-t;tA{RwGiZpHgY+pp;ry9%$ldV{N@8U(Y=1kyix1&7@|6Gpi zR3p%tG=Svw4%3650jJ+RKBycb*6jkaO+q7$9ksmDN%7H~U3`Od-31Iq{j;I>!Y=_V z?XT;t7p$7RY@RG)5d9f5^?qtTT{<7O$7rqb70ld5!Mmy}PSiGJi z%xY>5R6sJBY$L4S-OpN*&E3qztm3)nKRl%c&=crXBaIekDmUV(s(zICk+}R)SL4ps z_ImU33KIuM_WJacqJZi{JPMPehd4$;5!jSeerzLb?Ql4GaKzmsFp!f623RJFjBp)5 zT5cN?sfIa(BKs6JPUu+m(s1(jJO%?mEJP_zgmcvilkq#1@qU1(ybIgDT3=iW`_1>S;9>6BCD!SWl)zE9` zxAhcxb%rOOB&@h3PF!(*OF7@Qs`A__^z8c~3#;k$w-N;n?_U+EsqXn`~#5RDT#@){~!ej%`ARrvc z&ie5F!H%M!MsUUT*5;Vh*XEvsy37KA2Y18Zs0{S#!CIp5iDs>hMbA*OfmEk{+7>Xs zc9f@pmu3if5fOO_9pSLC(9;7ODlc$)EXlj-avd)O`l375xQko0`h5a)x7@bw64_A9 zMWEAIWy`vJtXm3*rYD| z%Ms|bf4~_9PzV%0JX}_~h06!ey_gFm{`}sYVQDDbx$)Ue-j#qG9;k*opdPv!88Lb& z$pJQ0!~@%%g*ROyf855hs~XyX5%)jaj>pQh6`}Y)KcCXs4EeV&dGWs%Z2qx>fK`!C v{{8X4bS{Avi~q-m{eQcjYya_W>05ykzhfAEX>8B`Jg;Y3`kJLr-UR&%@c%7l literal 50062 zcmeFZ_g7Qf7YFEjK1D$V8z=}oEP%AA2uK$ckluR}klv(s@(>V_rXV1_w@@OT5FpZ} z_ufk=p@bF!2}$Pgotb}OW_}pfUEC~I?!D)pv(Mh2{n?*QxSERGb!tZH3l}b2mw)$G ztSxzooXw{Z~E2ZU~x;^LRuQ}G*xj&NcH*8#=wXK}x@Q>6n zm*(fMoP|6$>2A7E?%d`0_uo&{-Sd8LrqnnsovGa+^^lEejmbDF^X-aR z2qYhTVPhOSW)(Xgufl6b+zMYR{idmtkbs`siv6yT=d~+Q=R@3$GW5bqUWp2iQdCgr z>gs#`{CRnKjguOx44N|j==1fXgO3oL=IZL|Ji?1Y=o1xko=3QB?amQt(XI$P$6!KJ zC#?goanE}2C%Nd9T$2ORC)LhKMuT5#;hO6vJ{B`>wMb~>Y&^WH1g!99dQ%&AcAeW|R z#(Zz^>I!r{EfV$Ikqi*r=x-!;brlICg!9?+1qH{;%KGQicds*z+D|ovn{FkH>%^b2 zdX=wI%cMj%-inJA=vHN!S8oavxnNTm6f`(FS*IOgC6y?x^@Jh#4fW-gh(x$e!at@@ z?_Cjq3D}PQ8Xn1{TiyzdAnP?Yz4!6-Q~`&Pf6Hd2R73)qOeoG-hdyg97Jcbc zl*;;}IPgbl;7{b2NC}U%mRL0OXm@I7J=Ao1diuwN=7qa&+2fu(DR}&(V7AV~dalDU zifN3piG)y-v%e=KNNlwZ(6UIBsi!B!$0x<#y?@sY(;J$RSaMJN#bWWmTEgxlmARpn zf&?DQ%0c`VBl^HW2E0SdNZXc>6LzxN^vcjLsYYR}^6m^$?tX?k^6cykHQ{{fPbP+z zk+w3Tw11(yD?D4r;Z39`kN7nkpK*(df*6tkCn`h5mKV4lerFN&kzyGNo3q6a^@+bl}b#REEfuSVN zR_&u+>J{4U@7H;Brc3wKVz~}b_bzZU2=LR@;PINKk>ODhkzdsz-@86cIL!`EV)o(`*0=570HCo(j zY1nO-_Clk^#jr?fw&|1VmoHyxN%DI>f7RAXLfH?LnV|9lzh`ylEpn3xM_RllHxPWf zoTJ=T_L`x8XXS;qXa3db=e{)zv(%KMBA$e_Xu-6^3*T%SotG`>AEC@ zesMxyvXJ8_a4J^HOjt?pqlpDF52`6KrQ(QxL*vh}F>zGUw`mf$LTl~oSHIB6DnhAr z9!16T>66$0m<2b8iFex{#%dVbKGo$+7Yjc*4r^yLh23BkeQv(-TceiNMtwB2>B+QC znc7>q&h5#7_d~&Ng1sj%t+p^7^Mw zhSd%_U!(gdXEU1Cd^UFo{Bf~is=5S+_2QY;djsch#lGD9gI_I;vPv=_cqnqaAM;m@eo%Qz~&F{qT9JFexpF5(tzJ`zXzP53+oT@V;8G7v&Dezr3)y{}q zjh)_5qs33d4#OCN;V`Ne#Q~Go8|clY?QK2vG`_1;e#n@$22t`@A}=qGRw&DEb!U@U ziCd>9GwcWg6CoFq!DErZ{Z5~Nu-dW0$$&d^C-^&CS#gEf z43+GgqA0_s?NY#6yQ^Ih{&V>|hx5?d20@6xqww5x_&{t1ebw$>EJ3emZ?CXI=!VvV zN8xQfBNiQ_@*N!=H1vELrXB=`Z(MX%>tg(w<&CI?)FjOxF#!$Y416a?8{s2YuTn_c zU=OQ*?FUY*U=h*$)LHeXsD1L=gN4caRDPo1eb$tjP%o_B`ucV(Y}IVMcwKHZTki&W zwQ699N`S>OBk=|K`P%4&42~%ErV$R*>3kK1eW+*$qcJmv&SQc2pDp$Njzk$Ptv7p5Ft5us$ zb}2m>^8Yb1u-0fI|NFK0&rj(q7cX9P-;aD&K*%aupH*?o#`s4{f?98jIE~e^Jy>2f2Gk&UvY-Sn26W! z%zM3Sd?p|;*VUC(sC=a9RHn6Z&G2_-VE3zn)s6QQjhHgzF&VG0JME?xztLUiGnJH* z;^SR6*NA_u9h+#nD)R!Y)Oq}K+gc|m!oF)$4Jl^r^dir!q7$G>W!^fvmYAA?3 z)WEaMA@BInB@E?snz89-`1GMj3!RTllYYd;!i;)*yG1^3ob3Jg7b8=D5?&^21ZJ!B zOw{`|t0IjXvpt{TQ7I>C0D56ey0780+LfVxN>X_mtu!kKg+gT?7#R5Rj{w@j!eV9U z(ccaZCGiopPIKC6hH*V;M#@uqzrW1DS9Cl*{m(`snsv!wJvJuhS;UsB3pbXjigT?$ zL}wiBu6nJfi?}V>nuBGIh>BS7+S}NB?wnwrnQzQ%h?9Br=#ifvGFDJyWvzDx62txOz7PbNnmSupXoI`u3~xcvQz`@PXZiyq&Q`Yot`oZ&@lJK% zTMzoG>h0h+Wu%5xe82c*%3jY4W@{F%_Mm#Vfo4x*qH@yyQhq5C1k#^BQXeMv#A#|^ z=b&|z#qufEKyt?ne-BB1ti7${`h8*Y7B&#!hL-{eL|WQ>v8xi^B)k&`LEV;a>u9G9 zIE_NELOnJQ#(w-5oi(!c{xAyeOSlv{gCPFL2bV8ez;!oWXW5^&HVuHHr>~=_w8|M= z&}#}H94d-mPfkiABbFJWX3Di(ro9f)H-pEZhKP^*2dWX!i14Vmt`BBQ-`yk#8l

~I*yw$w`HQOz ztzhP4mETkc-ddwKGa(~`Ld>94?aZ#nL7GIpPrSW9X!*+K8}X?ssDO8DCMeHzs`}Q- zEi6qq&+?(78O*n$8!-`4jN{I;g@Gt*j!Is;?xgj6e^lXxLJd2Ie;pQ%lNO2(4i6zD ziheT{Eo)Jr$2p8eAd)a)8VKo|uDAs$Mj@xUp`Ty-P^wbn3NdbQInY!pCjlG+2Qrk;bTO;k%i(o~IMpin&TFs{h#*iDUVBKb?Qvdp#;`m9x;_9K$q(X6wc0vnTY ze$BMbCa?Vb2PRB=D{E>r#;(!C7}dI@BmmguLiAVLn{jUP-9HKn6N30t`mAMTDc8|` z0|S&T?5$ZQJ4&+!Bg(Q#IIv_lMfa;PYd}$QfC|a9dC3RO`BYdYr>*SwgcR~`yt$5} zb_d8Va9ln@4u{h4|1O3_7r~iFQC`h{Iizym#*T7)f7`?-6=rXDQ5!J42C-RfG!oBD za2#8h7F8R*=&|#;6h%4h^$^AR2PE(kj_n4^JH7bQCF-|wgt7GgL)|C2$tepBZdj>} z7PutBZ%1jc=;BDFD|Jr)i`M@9o7yVch=@SSYFB4)%EL%f#T2_?j_3xWp^%Nna-XabpbcQgzGj`o^jtAETxR% zC5(_}#T~0lm;|^{&iA(b#>0;G5h|9s8`H=gR>^89R#(JX+9s=wlCFtKKh-7sDzQSS zGuvB$AeouBwL?M9G2>;YLQ5nGWynZ0+&Wb|%-1&@FV#xmrtpH`uq!~b^Wij5{V%n# z(aG1G>5Z$Zh3MZAc3%H#&FiWK_!rqdOi~<}PZ&fgbK{Dym~0H24b-5l2Y${XERdIR z7!i?4Ci%x!R{8TLEGwN+zQn$X(OU7qgOcO2CNc9r9EVnjF;4-0erEm`@0Yg*-qtDF z>5AETop$5&-6S<3n;^WA1N`Wy&WEfLHv{Vf!`N!5>X?y6mroE68h{xTrA`&JL_g%Fy zgFy0`fk4K~rTFY|S=sjOX}^tD)h4HzB76xVFtVS?fA1#Pvc zG%eXHL-Px&B)4Ng zBQ6_L3j?qhmE!dy(aLgt^^V9gHgsRXGwl_smlDQF^1a_3r6$4E&dur%HqsF;UI!9krz6Q7;A|6^Q0Mfz?Gu10My$>RN$skEgPL$S22y38cxDp*&4wHI$b>_FV|w+M3n51-#2u z*u0-)(SgbdtZfAO0i0xC^d_oKaB{fCd_Tq%SXZ&a!NI|N2EKjG5n1g3{5EuH8LbX) z5Wtc)HAkbr_04$NpK<07XEtH?FB)p6*q4+p%D++QnC^DfdkxjzKC{3P2IW+>>^oza zv@yaeTxFIX;)JNd^f3;VbnRb*Ewg9jD7PoR*#m0>WyUk71v1=KN+d2_PLT33dtl!3dV2Pg zg9mQl)rfTW#C{DVdr*|EN&g=02&{rI`d{?(h` zDS_(6rtq?Z?3(~I>hO}8AwRw+0b;R{Dl0}n7H?aX@bkynL9sI}>){#nY{Jn1FiFfd5E zD12wW^NENWM8faW+)#7Vq;OIk2XVl2=NBPi?M(dWWA^}mVZxsqE4Mu%A`|mJ15=v5hs~4 zI+FSQ?Tf9s+JUr5;@(>5?J&=V7i*}essK7M_@hpG5fM)UOmN@570|DiG!fpUc*?K{|kQXJXCLuPjcn=k_Dr!jt7&=J)CaTT*?upO0d zt-NJ1X+sSW#!EDQm4SM3+ z%My0luCB1F+~bH>aGpbW;RvOQ&AO_o5(UrSO1@g14j$~p1yGpTGxx7GaUkRNV z2k=^#*Mzz$TLLCUdf0kQ4pdB_Wcm9)Klv*Aufte&5+OYFVpxo=KjguSUjsjCIYx2& zIx0bX1BB8;i7}k65AN!CygO^vBS6Mw;6pnooRFA%2Xhk{8?I|o8MeJ@93fx6ezhfU z-~0JV^U6ynEtHWUga*(FfayqwcsfpInuIx#w9%)hr-0N1+=Td;xxAr@iu_PT4G)!F zQT)FAllDfM$fyj`H6DTt_kvd=T}$a}_MuSe??St{r-l>ogX06vCeJao?EmhtQnXg0 zMh@G{7(Gld`=5OO9j%i>@J$n{U-0v5XIwo76gqIEnA-`_j1qbt9VISXWd`|}(tqJ7 zf@||+et?=+UTsOP#A6NgJShyYRaz1z)n^4wjV3(s0!>h=u$`;$0Z5jkK|}-~`1t}; z`&3nAzDpiPg8L2f}-Tf-Y*4EymtCYd)ll^>JdF6J4OAbLX&>)j6 zT6+u`M?V3){W)&#Nl5$>qlqeIeY>@>Rx>r}ZEIK!7xqCw&N7WCPWK}h zV`q!XWBS|`mpU+k8ndDVT=mH)XLzf6ClnK7Aw_n)idEmuBM z$X@rrDtn}5#mfFW`m>|nXQYEMC!LynVz6tzvK$uU^G5tSO$^Q6i#PvXD=HBrb`epL{L#ht&TqB0q+VRm&F|^}LdLzj6ei}D zmKC38nd6!^;kDHt(5}n1sJFPex@;A4`M+C5OF_SEM`UG;I84kMR<&t*JO$k1>_I!K9TnVA$Fq=#)z^2hcCK zs1ndy2?}`=&htK`o%@)-2LAl9*)B9g*ZCAZg|ms#dlPWARyHLE$7Jje^%$qLG3zE! zuIuj_k?CLbLQh9sb#By+_L}l7H?)qL@rM%7Exjc?QFfb~_ZKJ)N?KI$#4^AFkn1HA zVWuMYm~V@MdEEQb!f%GN1SY#eU%{HnUV_9(6^ot-^Rf+;s}Zu9V{dt(X4U7ecx;PQ_J@esgY zth0j})cO+?(Q8xGyUji#p-jvLW2EKopyOGi#TE56dJYhnGb8JT6GG3PsYz=2x^I<8 zNWYSlgtoH|%ti=+^3RG&E+`3bcG9eanx1hzF&_|m#(uPRckHfhDyi_jc#bi-{U5n- z>5XjE<6EflKAN=PvNTkNth^r-C5!G7dn(+<&<}rq4gWIoiB1-Ti`HaCc*W1XXr2(&)J(%wOb)+W0wn(}Kg%7^U$@O54jq_EOgL ztMIXZPo>b^m(QOgJooE-Wwj@q|LEcDf7b3!5($+Y9^!LzEFlsAZ#D+H9a=@r*&!7k zC!>g4|6Au%EG3&A={zA@X=LGFl#u*}3Xebfs0q<;aM0em0TFN*^64P5D&`G;k`&9a zBM=U0J%k$vk=C{MH979`Kvve!_HBbfjB>=>tMEeAweN9p$fH^m;Tq#ym2#~})Xb>| zM!@P!RFaphoOHQo4!lIu>2_KKR?`2d|A^M$1f^Htf>eRrARIwyXrhN;6|fd$6DU8h z_CR$q3{8`++E4nl)Gqi}10q;N=f=HJ-IZzMxul;zOZ2N>Wj1uW=1{x-swFP32s4Jp zaYV6-tS0#&=plwdlQ@bv@l}9Jn@wJvE=m9?qmQ z-}`O7tLBTF?hDIO9aS{h9+wH&2LO~ANxSDXDJ$%BMu?ipD+HC#jW_@DJYEAxTEJnk zSbbx4H8XI+ApTQru8@GhseL=-5>`Lg;k5P%jbSxsA04|GbbMTgi9&V+=txHSr`6-{ zN~zCw@p+Oqv#2rvB_Oi0vd*YeOw0nktHBEKQbyn*`m8%ynH&+bes(~j*maObivJ+V zp?u6bXZ30Uvj%-kq97Zs2<$t!`G(3aqhk=bwCB#(i<0hw?pqzFB&$+|J| z`To6oRXF?8!xl*K3W;Esz6)`Ot#9Qh_5q6Bac)npnX+*G+BJI0-WZ@6Y*8|r0Sh8X zLHO=aV?=_;6OZk-`%#9bb`&%7Kffgyox@58(%+{Z9B(nA^zO>wlP?^}U1xp46|!eP zH~a44P@SnADDhl!4tx?~gT>Pg9$F^ue)Lw)JETob16kt+ z$`%|Af*q}m$C6t#N$yPAaq)R*Abv=GWj5PCN7bK7`!zfq)YMBX?N@e9kcNrTtrst! zjN^*x`V>3gJ$zpK%**{3T(?#w?AY=0@s{>plN}d4n)$q>C{Co!kYI3MS=$LIkWadw zWMySlQ!`#uBd8J`t(@}GVX8qM>;dj&!;8S#iWV z7*q?xzedZ;%lCEn*E|!dt*JRaaCZi(64PROK*hPc^6alR7XI6vB*rRv74`Ayj2 zf8*-vD2OHY_S_5%>mXGG6&iC;N9iWf(=ZA}R8;-im@=N>$TD=Cs0rj9D_EOuvZqj0 z=EIu001w1ETWvOm@#s`d4>)?o!_Qw_c&g|D2fO0l>br=BoBZcuq|D33r*CBcI{z5@ zknUU@^SO0Rfuh4^Yd)jsFJHU*;_F|Dy6+5ns%5YM z#wc%VV3fA$o#Mc~(b~>jUAfIk)%Zso{ky{MVqEO~Ewi&IkHZO*8-) zmx?*R5=#3Pdgv1f`kt&}HJA)uw|wR8>GD!vAH5GpM+nnn%)AdFSJwlZRX>Ag2|Pj5 z!esdQH6gUMf#Rkcr_ypbJB_66y?uUb$mb9_S?l%H>4^##7uW64dAMPLr%nR>6W@+X zhBr-yM??;+D$aSX!9Ac%Z5pq&g4Q^-SrcWRuF-H_o$PW0HxRoT)7{%$Q(oi#t9Yz1 z$hlT!hjF}EbK?->J(;P;$XX-9;w(VMK`@%-{*r!wUGU9jr)nwLPMyjKWXkwfX!JwD zvg@hsgwwnazI4I_GcJE6OKnw|w0Njd^}ru3Ys@+TS!=Slw=j-oE~EY$Ftj^MOFAH$ zYU3=HZ<#OOQc0sFlOt#WGyWsgUU2u7_jVPmKW?v(V@bWB-ss_39FGI{N${KM&tZ1^s&UI1>~nQ)k~|OXx|vqE>IVjt5!<%@(#wgG zJlA6)b>bfvB;p9?D3n!C0gt~e|j;qP0D?W!O~$2$`8N0J1ykPP2BJZ5GyU^Sy|3-nr*@!g)TOR zUDRIqRm#IlY#c3B3=?FjHU`WOSb$$gG1X?@hdNb_Hx`Nw!wGI|Cw6;dDg5)kJIHo! z#ZgtZwvo0~Wzpl5pZ8Gs<1J7>f2W|JPw1?-uibT8@Y{(0{zhT+dQ`^JSo})mkHr7T zjq~?}jIMl-P4+xmMWBRsW^2|5_kseHlp-x(oKgby7;oOy?Ci17dX37D!WbIrvlryH zKM@o6d!EZ4U^yC~stwVV(nwNDy4)x}CwF~wqDno$kf}Jbx1>P6%ZpV^#D)Fcy?ZvB z!3nLG>p54#9F+q|5d^I}Xu=fDg+W#UVLAoN7i?|6=OF}w_f|0hWl5;yF`zfL1_6YW~Kz+ zO@kOCT=pi5203mfCFxfirx9`fQY6&sZR<}z3}rf>Y|nZ#uR8JK=**Gx z+*zNlY#{1`w!@hwTL>T_u*IwyAu{BnF)`X&C^7)NVIH)EN!;Q0_ z1s>_{;(BL*t0qJG3%^t|P?ya9X*L$Ax+z^(|0-Z_H&-ZsfJTg2D_XsDIbFshuP>0E{VVgE(xFXuPWuaujJ#y0Ys-0}JW z!y3CgoJEu5fQDkuC{^y#k?!>z-j(l0_2Oe6w^tE+e>P-sTY&=5eArDxCVOz0vcmsr ztlyec*|dRPq9$+Nz7+}# zc#&JEY_6fw5eJtRdg42Dhg!x+dO4q&7(%x*Q#CqVf+#AYYiaG1xx)7SUXvuQ@BU#E zA+QUEMWx!CJ3L^y-*}MGr_40YBz}~{V2xf#!T9>&e9C)T<;?On)`JJ1vAyMhN@|A# z`lGVrI-e;rDLKI;+wF!FAOzEpKhn}xmbWLX-^`c=`xY-t2}>wuMW;1o)_>;WV&Cnw zgig;)3EIzQ=I-3Pa*?eKkx9qa7E<1-O`4FAr-hlvXv7D1suW-T>$8k3l(NN@WHg>* z->D&1^m%xv=(6R42`hhzdz6i^gS&dR?!S-tw^|{A@7^g9YS*O_bIi@}QinbZTzk`$ z%lb+rFgm_)0PXRPp?KMZVji<+&TVtn*l$$#;4ZW79#;EXf@-^T(YgCoD@=ZHCGEjB zDqV65j+?RSB`WBfJ6Z}`P5xH@i=;TW_Cu(n@wsPPp^){_&Xl%*w8klVZML2JcE=6u zJR%K9I@sEBIBpt?R_!blrzD1@ynTqE?Cn>e#+_%}Y~E4Q*Q_YU2K4Ty)1yM29(L;m zYMeUc##7*xeJI3okBEGDG^4q*wcpd#4q8RZq}NFZhB%e}CI~FG^9*bXY+EsLyY5LY(4imJOh00b)0h_mwgk|p z<34+Gz&NJj%49d$jGt@u<-C>y(t^EAzI!WTA%RWV-Ltp>@*F zdx_Y}JD8)>MR|A>W;zQ>V9ZTNAwMtY1Y6P3{ZlM%P+5%S(I}SJ3XKG7Sjm{l#J|dhg!7R57o8urOpjWVK>g z4YE0>#K)iEs(=ZPEkW6SmsDx1{b zx&`Q8Lo(qeL5_Ymo;+r+wwpfzZnOTCu$b?y@}n3J_2(`)jJIu-em3ocH*ek;OPo6f zX4*|@Nr#zQ+als)*}HfGgI@@Pq0}kfZmMdiO;-+py3Z0fAV=6dDPn=S)TMKz!&%5q zCC&EsgS$iOv|3Wh_V!-k&+XLISe3lO2$MNcTINB;YQ2e0xBjDWSi4Ri6Y9C=$)^k` zl9iBaIU)-?U?Z7eMskV|_O~p;(wgWTY|RAyy8i)!A-1;@(T=n%lBE*d9bJ_SK*`Xb zApnHpccgzN|NQ3&x+(9ynv~fCirb$*e+F8q8o}{y_^*6+_CVegLbZcIHuKZXh=kNw zdcxxdfcH8=Ni}JSiTEDp;*7U%c~4G~;O5LEL%d(n3~(dTtq)rs=By7SZ*ROWIL;MT z)GhV%JG1cq|5dF-Q!s6d=sFRQ!g7PKP3~##FDGjZ^@i8c>eR12&YcvS#}C-r_M`%ol$Ap0rly1e z--Ky4WOyQ8cy0s&ejZ!7{`W)af{DkG;fi2qwCx~i4F#NyIvye1{S0g#T{eK+;n6A& zi{eu_qW;IY$;#34bSbhnUh}?yt)BBPGA_T5q>*DFFZt2^>;boWhN8>6L@gN~K#$%Pv}gVYC&N8{{tfc> zQqAr14eq4E^@xI9lET2Jop0c6WqpR(ILQW*CW34clY-X4CvCcKPiAss7szM z+BkEB6zWbGo}!hyx|(;_C^4e9q~E^ne3w^|1y_DL8FUn>WXQB7zhX*u8p2}xiKfL? z%9yczyQvcOa21WslWY&{a1#hl8bv(pe1XLIR^p8eI_WzNmu0!)d5tO#`bmRuiDdnW z8BL2F5ZvorSB~e1StQ@4%;(ETa1TSK@#37oa-*jNTRXdKHO@QKp}q#1pI=(O9|;W& zMGyqy(q?M+Kf=WWb`zx%;ZITe3CsFF{(1zFRLv2CKvb`ooj9Bg)-Co;bxGlSf_jyd z8^z`b#~ZL@o6gx;9&T>KNAY)ybE&k;<%;aeL14hx8w%JrVE3+~ZzSx$8xK&eR^rNW zcq}Pt8h>@m*N>~9`5MA=>o8RDt6k+}t1QD4-%HE4(ghv0OasgfA+$nBgJitK(@m=A z!5z1fw*h=TY^6D014$9*5t)vOQe~S%=zdway0dArhs`USRF(0VKHq<8x^Wkm%H!5U z=qd{uvczy7%+19d-eaj^TKX>=2PXe436y{t=^2W<{p|?muvo|eqNmL6XT_&^ySmb@ z8${W29V6rW;KlWWD9nCou zVP$TB$9SW6t7%0{9`e$CEHlKhdeI~ND+Rn z3Ye_l{L){VSJLmB)U7Y>i&Mp@`g^P{Hi>&|76cpfvb}Y8_(XVnW2k(8nS!EDjIIq! zK_uyerV-cjI|JrsPJAJ1)tRx$$$TNdem&ELy!1J4C=Pl?cPhjBkMxz&tDT)ABml&` zSj2_{Jn7ZH2E*!ds-)A8^z&vkug|ADO*qdXiHqNTqq!_K2UrB_IXAg80RfJT4P4wf z5!T6;NL>ELhjjhlj4R^k2M=Cs926CD@f=rz9NGgwo_uSoyBf(EkdSu+y*&8Wg-d_J zE2|wG=Q9Xa#;mF|Zv1cFv_`N^j1^dOGS^P`3nPZ_N(L68?aRZ%U1oIprfclj$ah9Sw@8#yfSEsGx*^gJ$nbI@7zCB0E z=tBxREC@NZp$;RuKeSt4HYVsVOb|gtbxV{s@uc8eT8<)nQ@nOXg(nly!%e^bY;31K zU}f?`n?Fr&A8D+iTON*2!7MC=(1BP-&DOwmKx0(kSqh)4NBWX#;A3m^qcZAL@dy6- z2ebKK`Q(a*t5dv2;S!C$htYH`Aj2od>1?FKwr2aWgi=3XMifj=QfTy-@w5)JoJVt} ze=%@$hQ+j2RJdtzTz!;n9;-S0LsIXD!BdIO?gL$Jovh9jzHU4Ya(o3+^QpLU1z;g5KagJOP)HY&<(6Y6~^pO2eUKw%4fh5y$!S7@? zj~fKkfpwI8cveGm+2AfH+D0BSB^kCU5QltV}eq^>jR zZz7-+!z*lYU6Lou#M{=8j*cgP&grZm z;kFwlaq6^>RgH^f8Y4O%M^nY5_n!J^(Jdnf*6!dmAS4UNvyrwb&G~!6xu)yKj)K@( zIAh`3_UbRcIcE$dkgxzMikVH~o^Yb_{cQKs6;G!H*@xd9raTsVLwhumc)(DXmg5l` zws^iLKTVg;*Li0W|4Qry2ZxXVFKN4-4*dhab7t^^ppTird3ty@cpml-Yui7{vB=ZV z2U&Q28WGQY6_DmS)kZgcJ-1djO~5|ybdwHk?1x**T8-DD&RFR0r^TiCZOylS61uBZ01RPp z=X~E?7dXAP^ocwio5^k`sq4b+bpCB8Vzeh-_t{#+84~{n3)}fl*V5Le z=Mg#|V#}Iu`N1X$iJD~K6@KCf?1gcmfVa1A-rq9U&IROb-S89VCSq?7K>CL0f9`^? zCmtZ-z4{F#NP>5yl~P{qN^PvLCXx{+GpazA;NimzmRm%45=8%6vzhr&MKUp8d#2rX z@;7CT{Pg&eyu7lg>+*Jesoy9K(+T0)_F?w!Kz1{b2x8iFo6g7+Z-T6UwEosY$8j{2 ztZ=J|02r9jAc^b2LIWPKoX%x02#g=^OjR$kPt$TOh|Ggc0V1TDi5#o}l6st@WgJ$+ ztLr>jCc(K6fFDQ$6)h`Xuyo~3jz}||I`mgTpn8N?Vh6-0#V^MNoo8Qrxa5B}r%f@( zXHViQJ}(6mQboL;u1@n>I^C2$@kSu)U)d^oX=ywmJm9v?P|O1B_+G%+1iBOU?Kt4+ z|F-o3)?;UUZj&Op9}|;^>`>NM$eJ85xp09R))h23H5AxYo8dH#pb$!tS5@O4-wKaO zB>rbI=h484itM<-YGY&baal>t)h-#hhfE3aEpIq;}wqqmEjG&j}D zZ-r+SS8Ar{^br6lamCEsp<(Zk_6Cb+0Fm6Xwb+Nmj|}tk^!9BL3D3K3?XSZnMf}ftry|&vz7Gi_G(-)VbQuczch0B->}@0q!3AqdDRoX;yMH!b zol;C63Eb{bwZ-~zpiCOuC`c5V7^Y2wJ^`~uk&Z}`Klv zn6g&(KzZyj;tsX=JT=!baN~5tO_2X#%3l>#C&!g|iV;#^mEd^s;thOyC)j+-3QP^s~-<9PDDx?$za=~ zMDgoToaSpvBH+}rA5O!^?K;^0x$qiM*kyesYpEMIgWDpw?@ZQIh-zr>ebUktSfSP3 z^*c{!c1Q-Jc--@sFB^a(`zpW`N{4=BRLNd+KaB3e1@tHWkPBQ=`u%j-YW!;fxL0Bt zjk#{3C1-08s8Aq4|V=CbG<&_6MdRAe_5p#c<1Gtfzpez7exyE*0 z6xb+A_Ggto|MVaIKmR)qYX1M{^ZEFHV;+n{LGu9t=fxLlRJt8~K8n|;&tTg(Hn{k? zw~W;GLL(9%<(TJc6vf58De!b89%Ia`tY9X*d!thq<)S;=+L8=^6002~tUml#8l_ww z)$^g^HYlXk-MPhbKR!O*(b3Y@S+`h9e4(FtW(;4dxxo(Fr_cPGLAk-=e{C&R%8b|V z4|YyS&NKBgKtt6%w&gV6g!H3iC5F*Hh3O|Lzn3Gd1|4bV|EwT04G`!tG!l>@ z_I4|3aV^Kc&7``&H-V}95s?+sdd@Hsin#O)^#z?q7csaLTiZ?0{2eOjx}@=i&KwAJ zU@!GVToGb|5=(>SAjtx)dq47?p1=D*m~6fpuo27@O(N*vG)jFSTH`EQ2gSZv3*{=V zzuW>Ej)9_CSX9_^bJ9dVHZ}EO$VU$s4fq;nnJ45O(715X$-b2V(qo|#Y%!P5EtdVI z0#I!2jJ$U5MW(a*+@#;$AHc|KMey~j#BoGBs}0l^63(9t`|>GC8V!224x~P~_a-p= zl^LNsY<<44N;-O(fkEs|@57MEd_Kbl09#X69-xq>d@0-6N%^W0rp z{5BV4onJuUDNMl4trGA|XJ`J}>___tvvmX^wzg`f+NM*N@ZVoSAdgdHP!0ZQgQ#mEW7{1^ zwws%#fq2c09FB~@SIDxML;{y6(m?vlWZgfwmdQGAwvga7WA*F4^+;f?L|WGPa*JDC zD0r?h?@XL^e`2}7`5fPUtC-LEqu~EaVF_*UNCsqI-sdrEf9&d!4VxC6WE;NGu& zhSHx>-@SQ1EiKN%&^^C!9H#v5wScqLI129_Y5(WYXjj<1>&#~}fV4{UqfJG5>$AtT zw{!3q)V$E;+8LHVq*nKsLwX*3EDit?qfS$QKMBrP!cVlb{($0||2@LV=+wV#Awh$i z2hyPrRZssFelpfPuOe)C6FmGsSk9ypTgYUc;{@PaS{^Cu6W*Vn(gj+szY ztOOmrZq~}g-Ss@Po8Kc6;KO<|x$40{@iAP1=-|mw$dUVJ+3jAfgriXSf3>P# z0nZ;o=Mm6A?@#_t_fTBiV)zKZbvk|uTxd5n*s^SI?d%K|iz-uUvFdO&c$sR32}Eb3 zNh`tb_7#uk58r2kWeb+O2pm*jU!P&EgP_%_nj#I&j-jxyi2vbf!*5N{q5Y=i(L;co zczJmP0vb#=hRcId8PT`D!C-wUBq#9yD!S`A&WFX-I{E#LacUah<*>H44uAp2JUuoS zu|kKo0o%0r~s?L>N#1zjDs$ z-o0=^OPAI{gZA^=2TivEAMtcmC$yhp!!|3%^7(0XS&?{7!M^F)Iyo9V?dNLGs^^-e z%1Q(JE_>tUTk1pMN#peF*Jte-=L+N}@~gG)9vmF1&PUF>vLE^CnVTC#N2JMge&&lK z3LLsta8b^;sQ_~j-35g=-xC+MM#P`YVI~5E0%v`cPhUMZE=N1q?6ro>9TN4Em}uBu zR9OCH${MNEtETfuzE9;{pM)RB8gidm&SPJj{&NU>L-{MUvG;oZzIy(IbVak7b)4;@ zPI06t@KHzC&mU4`;y5EwDo(6AP9+nQjFPe^3D4#A{}1-wI;_fWT_42+K@g<`1Vp7v zS{kPc(hbr|H_}}KDj*=;E!{0G(%oGu-O>%`LBH?YYpuQ3S?71{bI$r_&ud;HOn76w zV?5)D`@SEay~&5np~pW3Qp1$&>r=Ek&L|dEVuo?vnRH<2k$F6_CF5pdOwuNLm#d~f z^`(Q2PNZb9y^h3M(G?%*YJJz)TSJ$j5;tLdb%ENZyn+Q* z(&{`34GpEt>@p7l>sA3fW8kCFaIw{?_S#Weg+ujGBh?|Z#W7WfqSrRr+OG^~)7N#l zboC5LpX&H!RocM9?RNwh)mA)*&yLz-VR=58I=_&~c}N&SMY-^eqa~o#_rY6%KwVzj4zUNVa;**B!~M;v_!-6e zioM8;>G(?T+{r)9je9>2Z1t&m5BJ7@>DM1JGBXC!Y|eOd?}&!e$x$j?BlT1DiQXDI zuttw|i5sk{jxx3nwQTZinli2k3=F6+*%et5yJDG!)tw;jCwdb%uKGv!YK{LC7TD0n zvx7B=8`5?k!Bgt;%8V(){2}C6ZvQ>#GYjccV=gq(`q07z#C!G0mEQ5O$@UjNv z^fUUp z-m>-sMwRmQx$19uDiQZ+Nu3VYQwx(f6Nr{5X;^lPun?7dO%a5zgvywnAq;lc-SWNp17E|pJ zov(XhSdy`E`0h%N#v(zVXI)68JfzIbe{PNBx}mn_t3;LMwU>6^3f*efD$LeQjAS)Z z0_yYf1+x{@jUIiD@z3z|7uY{2A0@ti%>_G*JY=?v>xH%c%Ax!eIWdNONTCRLbDYeQ z9aer!OZ`bTm_NohP8tT!?QT5LR8HKDKWs41v%9}+@n2sXJ~1IlUQF+3Y~^(#V=l!0L#AnVFDD`j>^;u;!(K0X!> zXKbc59Fu)@RovDDu!3K}3hoX;BrM=P%)lnJUhIF|HmVK8*tXmnDiR8Eq0-fHi-XqU zRl-5gN}xt_+1Fldq#Kjl0S>(F`e|L&(G63@z0|TYn98li%RwofvT`VB{_q~Q`T29M z*ky1pTiF}q5)u+m?MEr3?g%$GfDhV`iw{@7@(oKBtleQwm8f<(Nj$8P;B{0-VKlJi zmlA8NZ=_S*co&hr$S}q5*=(`B$7zhKWY6UmZuMqvjhvN{aTYLuPP<6jx`_E+&mMc> zh&Y>Do0p$Gv^-jK0lv@!ffM^ZmHe61KEb_-&z3r8eqtv_yRGf*hq@JgUOl(>#(xy++CGFG)oWXi|vnP6Q^5~`WQ?1~FPoTBrzOw!i1{~<(uLEUG6#+`=IuSR&NX+NYmq|TJhnO!(f+C4T%*}i= zGmKs9E9_D;v)TKazNzny;x9N2wArlkJe*JJ3ibyN5zlcWe>3YojDi%beM! zG&l#|&H)0gwWWoT310SdeZ%AJqGcH`ot~*pjvvHLx%_;+8dnn$5s5pn(wFL0x;_~u zoD4cz$rJE^CAV(EJtB-I@&&0w{0MO4t3a{VxhMq7gp!g`t|Gm0W<*VppdGls`R4;1qP`*X>NjdJ9v7aTuAQgZP4A`U-bCI z9-M$%hk9(vS;eIZhXoOE2mB0UNZy}?gCyS`@= zg1q#+q(AEycTbJEdB)L7K1K)jo_bCe?5-Qw;VK^4P>AoZ>)^C0oQ4EPM+ZlbS9LL8 zD|DGstD!wsLJ9cEva!c^c51f4v{NiEAG$SHsgPI&Px0{W`FKYgPnO68gs~gg29%QY z$6gIl>KsizV(yHA0sy{D6uY5@wzj^8#*gV$*0tH24kbCJy=e(+zF_uvv_9SsQLrX4 zcrVU#h(_8;G^IX5bU`v%{{yzK8jqS&&*~2Al8?Va^(OZrg69R zmWK<~h|J`aOY)E@q})uCE=ZiwK8;FqY1qiB~2sq*|Vm~ z%|&%NidWx76pkTvw7VPSDbbXxStzVdzdt!V{GzE9@t!`0b3qNiJKh1qv2qh< zr&Z}zJk#7_FH;Nn5WyI3m^gJ`zJT0aE-}OI+Q>=Ojk)=`Ib5aX6T3J#2a92^-s!j^ zh{6boiA+=*<`x&r;YZ;h?f{3re$ise@u20y-n@QRB!|;x`(n_5$f;t@fy>nCLjzma zwIgK85K80>K6>jXa!v#8x5%D6egp%@c^5Mg@%%dajg|4Ns!m{$(5sg%&CGnZpOq#w zG$i#X4C>>;q6p4cg!oA55M)e_z&vtKOjs_i!k*z&Sp}56-ejhXfSX4GvXX6KHdhza z%EZ2ilh(g?;)dTr3z`@%HHV5$6Tei2mkaT~7FRBcT*BHe)ELHJ~NoQNUe0-e}HqwTBn2Ehb20e~-bE}>?Ob;JYK696j$a=cd6@3HM z|9v#K{+s&lJZy<~1W9jazr2A3u`LN{kazx9h&ga=fxC;Rd%jDj=Zc9)n2wdDHj@8d zO1J~x{#;z?sS8g%ub@C>W1hg&w99YxcowyxXPTm9W!TujptndUNt$J0P5$U~ z$7hL*-E;zaMfQjU$!)aTJC3oLBB>Y|O$s@x zaV%CT28^`a_h`8_rfTb=hE8EdT{go$}-1HFQ^k&VObZGSlKdps`vU&P4Q4 zWQH9VI&Ym-pSM0J2hZvQ!v5q5j^-|7@nsv=doX>0oGz!1q!YM86^W_1(3_1|vDgTk92j#3<(^ z7ryVs`A!b{7~j)SK!Q+qX4aR0cZP!-FXN*5ODSRYYhp1mh9tR;$%(KZXy0dK;m)(L zc*fWEm&K3{f!Sr$b7LwE7jiHWCu_l=6{rQ`ZXy_WIH`kLQp6onUE;z#Fx6s-eX7;F zv-jeGvxkWyr3hx166SG#@~>GhMx9ifBTwf`8V^I)pQS(jhK^DU7HU6xDL*R$qhD1B zIe0LlNdCZ83nf01lfc#-90vS|h*;t^NG?aRd}jq&3ZhbUjIzVK+da_{eLfj?!`h%` z9-6)X1f^%|tmbcgIs+Jl!_pltup}DN1)#^YXmgd4?jorx!NOUguq%DY4mUFHQ>M%D z6p4wN!8eHC@~Ty??16nd*_t~4s#ln8#qC5-qXy1%t~2<^s68Jm)$*DL z1RkIeoAmYycDZ%)7iXaMw6%1g-|&A?Y`;2+9_5J{*Xf{+0^1)s_=x6Z_T=rIEmAqu zW4srVpWeMJ=^Jnd;VJAupkZ!S=2$P-}bVQz$b; zPgE?+%*<*mSu+V@JvENAqm!531-o@OGzg*G#3f!-9ukz>5_;hQsFQ}|4;8<*PyU@J z7+I*Y7bKeakW0xi4j2l##Um&GMbHn5>PH@1G74SRX;GxzU^109}h;Q7A3f!3p{idqBhJxiYx+{M&W?RMn*)uu9OvMqg zH|efaYRvv^K>z4X9r4pt9TA*ZKlw{1RTSGs+YM*cllcyD1+1QeP24tTZ-@#Xc80|$ zCaV2doh>u2GaxivHg~?3!B?%yM-okXS@Ky3CvswM`Wc6?Wjd*gAQBE~*M$1_jrAR8 zuSE{!&bRhltdVS1tzKWI{p-GNVcYTYszIC={d4KjtP_f+Ou<`6tGm3c%Hc0iNfU#Z z6o|^Ojvv0-Lat&r9ZPR+W6nDI+GKcE(X%npJW~bc385^xDKl$&O`8pqLQ^s!WKBaLhi}9I7AK`n;{_d3wT!(Qw773uBoL0IYJy>k(U2*aJHxWv6<;a3(4m7|L~R))F{FAtUw=YOH_mrwZGKh@CnEe{ zqJ%~;N5GM@eAOe?*8>x+R9&bm6ta4&%LbaS46rTvPLo{d>yx!yIC3U3CkU|zf>@i@ zovOe0&fTy-8nVfWkI8=JePW&cXAQ53|FhHHDI^~oW_6q zPUKximKTs?M3cEs>XDn-5m;JYWNWikV&Na`_S5}U_Q3L<-uKguw30p-KB9n+Imr~u z61-{C8czsl1+QNH3Mx!BV$0Uf*|>D-WI-+76Au`vKnvys;3z>9)srPwKA&cSLo$aZ zz~kCGeZp{qB+(%z(t{r*J}|^G2~Y|>qQI}{S0m`*2^vSilsDXWB!;y&ZYgvU<5 zRBgT%cYORcs+=$)nnos^Km;}C@N7X(S(cri=Y0)H%;xu>gU{snscsVxoPPecb<5cN zR3lG?o`lutb(6KR2PTxhm{UX}*$gKaEgom&rHcGMwfV}g9ar|e ztEoX?{DYrRCp8OZN!IjhcBX?WIo){-)73>i{E+an*GF>cDx@CoMOt>v3wqOXo7>c# zzlDYc4Sd9~@?L5j#o8l#d$o@1AheD?;W~ok-8ItD(jA$Qe=H1g4yFr1>=R34eiv2a z76B3tg&;M1;B5k-*fQPtGB2pEb&HYj8QOm{-w(*V3G;_iWFKyzijbic=!ohj!ITx2x7P*A|h_L;Mkcziv@EYN*$buI#EB?Vn1~3%(qr)zbkGe zpHm50K8B`)t|PAJD6h0lvecLbQ^Ko%Y|nOkSQR?3ZqROaZbo%+2e;xd*rY4)R(LQq z|E?C!L@byXXe6p6|M9RdGRj>S_Pvmd)av<1F4ewj^i1{J-UE6u6sfV!#I$V3=r?*O63<0~+X|o%p;Db7E+{1;I@0xCcepeb0b&2Z+K^2)6lO_p((9WC14E{Sn4 zF#56Z(uMwKJQUSA`SX?bAFWkkTR(i@IO;mr8aVS|JpO3IWH;1PjsHbj$U9S!S#M&x z$_H<+wg%>TY;V-u32ViP;#Oy9V$&x-@o4U;?-HG8$k z5h^V0E2GoSa6z`!W!wy33`9EK$}R9~^_ z@xeY6k%D?H0(&huyCxFUbmMDE5BX(CpBf1d<{JhO5tiddM`4E~y1PW7D_gIPJH(e{ z`YaK<11_`XjHD)B^S9a94yV2vl4BXv{hB}HkJwtI0|1j<)rY&FL}*uz=Ry>0KIsXP zTLvIfhxxUq_B6%|YcOjfJc;%7MI5chc-+pCxlI2lhg)GL+9u;kWk<+$N+U0@?HI%B zyT7gP*`Hef9Nfu!>gD!0v)s>gN53~cz))(Zc-VF+;U2@;Y)&+B7G<2%$q8s}Wi<5U z*4+0mQ3#0& z6SHZ>+ZQLfQS1>v0>!Vz#qX0f61hTHYBY~txzk%oPN)(~y)kP`?&E)dCByUe&1wmc z%)Man_NA3M4DiMQ$f*4-A-r&+n$=sCk=AW+?rW@Lv`RoqI z!C_)&W~I|7Lb6}>Ij|_oDu#UD=fl!Bvpf|xt0Ps0wVQ#1>G)u+y&q#dWqrm(%Ycs` zLE@7Ct$oL9|(cn6H0#%Z!i%NgSvn{t1NN=?OnIF{N zMWA&-7{h5kWU05eKd3tFf#hxPOmKgV0v#3t_7=Ls?}TzEf9K1~o*R24S0*5=+g|0s zWF-vh4wglH?F0Qe&@m{|OI54ZnP_BPW^>-IZ68v{mjSgAu00e|=l!|1(HwdN3bRZf z*gShs9st!T5X+@PV`WL3H`Lb;2oC?0%v|1o0Vx>84bv5KhJS$suDkUgtH26eILxVf z8up#u?xlXJ{@&v4MI{fR@VdOpD<&<|CyN8s+#Ohtp1?rCS&gzjoOxd`-qX0!>wiBs zz1v8`do(jUBi{V%m7b1{Jg9TE=_*`6x{ToO3b=HEo>ewXK6CM%Aw4AtMG8$xG&;I- zIOZ+O!PeuNsGg}xE8XGnjn=VCI$CXqPWcQ>j8Dp_DOkBkS&vp?!gGq1QEqOYR!Yow zL{Np%pgPgxO}9hnYim~y>~L(QTB*Ql%LWoKQ{QG6V*!rL_-Lkr1P_hW-m=d}pBU-e z!ClOYhwty8V1FuYdHp-ldyRBcENeD9(wn<%rYAs$zR@R?HsVa8dxR^W24f?9=TeQ2pav#M1!lbsVYzc%cAY9{6OP$2~q@nKDR z-Y0wG>erE;Mk9kW9T%di%x3oC3geRk9GBRd+joetAfoKgdwilG;4j%JT6j8h$&{jv z5$htvRHu(6%X9dRw!Pi9i{^+^k|m5LE=HQ#Fs)uzNH0BT%u!|XUZGQ-oAQJXZV)PLHJ?$CV#zb0V0Yh`VIcsyNpC2pG%^8M%Bwq$Ai%FMRVtnkHC}c0 zbL$FU8m_PJsT0{WRxBqMN2pPg>!tw4WomJBV#&z+A%sHLp*p_gIY)kLHY2eF)mO~d zF)~3(v@MYLy--JUz=#qyQyw$N{k@mZpLT^3`>}b}i>Ok=l`0fp^8k_M6@T8`_o3h^ zIPY!sHmB*y2?-l02^)+8QW4qzD*eHCLkdA;QV$TYBUCt(m%Yz)P;(~JdG;g9Q{Blx zp{?XzIjUZhEaR6g*{#5Y2(JK@01AbfbltB;>4g&YQB1{M2qU16u zj~Q<4xjsH2V9zquG!BV~2nmlq{`NhoZ~hHbbc&34^GtIU1a9CVnrAjms*ur00b13C zd}VLFukZmEa_}_{%rq5Na{Thq2sF~z`qa!EQfvgbmk1W3zod}TGKfz1i6>@FdKsxIw|mw6ZN-c!c6Smf6}p*^1xy*l6J%s8|x??Md@7S@~g< zfse>Xzh5~X#bb7;0x(T?dn>*OuD;>@DgAt7qbzs>wtZ)9L{(1ah}9#5GzsxuCB10; zNbypAGFj872Rb4aWN&YAi5QXy;%R!7)Jdd#6eQylYZdJZY58y+oldl4eD#i@;)`_Z z<}K{U2{tSxOrf`nGsn`p2Xj1lK2l+u6yI5TG(aE5lVJ%|sJrN=*|-DM%FE~b zMHkN0E`&!15L)t0c1#v@qoH_WQbl+khlRzRbDLV)2T(jxO$FbPeUc#T7S(vM@1NZu zj}%*j=!-v8Vj|6`I0sHyf;4&|9=*2i{O3m;)<#p-fZ@vsS4cw(LwI0n3ZnO^Wxp^# zaKTexF&>Yn6&rYdQ|Q`vFSFJrJ5)PIC3Aamc@YV9uY^|#J2-ZBL-&YhTTf47N#$mQ z3Ynw}n3l>uR}2q_Bwq6Wp;;8Sk}c5e#$PN+M3I2sJ!trSmqui7B5PV>OIpN#I5_Lz z?6S7Vtf?jC%saN^##rmed()Fg)9p=3qinAW#o7YbKL6a8eC`pEU*FWAl(bnMbmRK< z`sU7O7|3)b7k@_UOSornY9V9|S?Mr&#YClf-+%BMDbVZ+S!U;p;cBb@)SQQR6UW%d zXgu%KMJ9shn*aXKzFavUO@XV|@w-dYI`)XU3--R<+zD*KYGO}os@Lmk4r#j=_yZmN z`f^42nDXfJ*@Eu_gal}sV~ViAPq>d5=buTm~t5N7phyU$Cdmj%IF>&IfS_> z>(hc?=FD7LlrjBzI;k4*-m8jW9o61(!}<7I?rtVKtJ6!1EnNn#4`MBx)^r7$PZSs^ zXr#h5Xpp)h1PDI-RqA7H@kpyS5Wmav@ZrL@sT7+o6W+;K^N8c!s-tB#V;8^$e(U*C zd*XmsEN5-)=UHaQ5zAHXxSCQbL3BSUa_ej8MBjElKvyJ?bp>gZNnyx^o+xyQ_^*?F zO2)=P+1lLl#DuAtCEDSM`S#ZR8xE-p!S$8Lm4Un!+!7m`2rWdUYdD}78Q z-8(_56M2uJ%j*#c~(WyoHnO}u85TZ6@aFjKG>TXqf(5dP}Jb+Vv9?IdFpJo7LcM2J1 zWK@(kO#k`$Or`|?uvBxl9?)oZ+M2$#Dv=Q>E0EBs>6&qzu*;-lJt@LeS zE1HSv&gg)mX||CpnZPZtY)@@-opMo9hHYU4Echpv>cWGcqG2(kVaeFV-lL~eO!>$) zrZ?GppYc8&DUV!0>O*EWv!W8KQVS+Wwn;ue|9lo^oBZOo=9V1S7@F93hE%3cz8>() zYA5sNkqSF^O5)s&>xyz}NWQn8eS{t@14MaIJyuXz@jpM3Xh8$d&$=W+4-9 zZ~7%};EK9un9Gr86;NC|_$-hQbAh(GudZXfotBkORhh`96dJ=!xa>c7>er)se(Lhp zq(Z5LqANQ0ty|tjp;;z)xceN;JXADQ#R@0p6A9kWFfMNkTE%%>O%t*# zQ^8CNFfhLhjj1|2TQ)ZPcMG+VU25QBvZpFfrpuUp+w`M#O-xM0#d9Je^vjCmFU`!( z5&{VQLtDu0+tdm;#d0Mq?1HOv_q;1Bjr4Qw@pU!xC(<%Pgw{G(99k8UuBhhhY+yYX zlxnufF~UeCv}WzsDn{HWIAmqcoQ`qdHiG(Uw?%TS$6U!&hWAoXEMuHwVnKzgH)8lw zQQB4YV5=gtF?uY`B(`j2;Mfak_j1{gWPx@ZN>7R$8lAG$`siYJ57r*B$GNoNg-b_l z2Iru8PA&>AHn!8hf!qF=G8_Yl2efT@?!I UUa~2AXD3NwE*?TID=V0)rz?@6xw6 zju+%dg?W0V15=rlZnmZGg<@=FgaeT_kypZ}SH@i~Bxv9Dv<+Wr*pscilGcf2!sK-( z&@|Rr(=t|GkdeJP!;eHD%o$%K)ckbzQXuIzrqB(_iY06mzN5n?nIXeh6dIV<}C1I2tZtV@}suN1Dn5+b+=me)Iy@(%-UmydDi z%xLj-fL(0!9&P^{yC^MJY*#pC44p1^TMrvs`;YbQm-;@i4B)t$ zd){NE>4OQK=Fr!MEtk~GTQ zKhvKw*KbfzDX)>MzxdbpE`}?tu!aWTBr>Tf;L$l?aH=?sHEVfVD-xM7BJ^}zRO^r z)nES<`B;`btSOo79YtfNLLL)y?s$GYK_~BLMcv+xAIlKm1%ITTUJSfGu#AZzh0031 zxHws0Pu_1nR7)Lo^@1uU$_BUXC>No~2My^!bFH#lPrPE1i%XMy0unx@9i|90W>+l< zK4?hQ(Vc7k1RCg}DLOeN7)#9cwp`_r_)!mZK54g^Z*6&&d3hy;WJkp$C72{R(gZhp z`Y3&bb4z&zCcNaQicO9a%6;lYxogmv(Jc{Ai{fD`u425ZYP{;BWJnuenKLC?RbSuS zJTxLE;@)`6>s?ilpc@w!ZbNIk$~09g5eFte**$!Os+_p}l{2hLZVlPw?WcI7%%u95 ztVnK{l#u&%rS{Z%F)f#yr*Mf|91_{c&)eazL;0 zYN~J`vFnl_KZ@QHn2|Dal1#KktlF=NeLOvQkPcn>QLx4LD4)Ye*&&)t{ga5Gk6v;Q zuy7~4GH#wdw?~R)zDWA>y7^tCD!E1>JYuM?p{VLu)^nh3cVMS6Zr{bL(_B-=y2M!r|`-B6z>bJfxV3Q zf9^G?*Ydt83U9SgP$%6-l59co=)Mo;iLydEn%OZnqq&k5NfI8sRE*vOP9kPP4ooEK zuNLBO&rgH`AdDNHZ#lJJN!|3jgMu`0<<$+O^TOVM>#m*+m<-=X5UC(ZHUuDDi1!9S zOhzBs1Qj#s`7s)dYviC|%73j)A$KH6D>zTGz7{tfdr2=Ddf+7hBU6T7D;~E1$(%B% zP_qf!YtrDrt;w?N1bH9~FV1Yk|Eb=FOk9Y6zC>|SG(IdWf)=yPejokfm#qYmUiiu{ zSBH=2=l9@8O2a0YzlN6M6dQ8^Y2gk3djd~N6r>Q5h(i_)QYlQN4BB(p6s7J1_)S=Y z02~4cA+{T)T-f1$S0Z_q41;bZCHzeC9^5wJ6=qWQl5=Uo);ET5(N-`kPUf(bZrx2q z1}vIu;E(I18Jwzt#QNvQp}caDG~>dmoj=QYv(;dXCjwGDFz*h3NrpDJTB#kj(H zg+nrL{C7y++5{2=rs5IF$%u1MHvFw5o$*wa%hv8hYg0kdHqZbk95Bnj721KS8JR(< z_7*u#cg2$1uU573hmaAStahmJ^4gg3$}KgX0LYp4?2;9`>C^m~hmS;2daASIGU|b-4|oBE+$m8Jk<$Ep`V!BIJrV(d zXZ}R2S>H0C1R7iQnu?37si7qwx?j2<;a4kF8EZC&hvnQwM&$rIGz5h%!9T&rVq^YbKJW}}li z@<7?~ak?jQgWZC;5c2oEd&Q{te+hl2IXf=fqLBjQNA|y}8VuF~5zFlB^$irzYO<$0 zZ=WCgH}@*Dj+S-c=FlI12?V#1Jni|;DrAF-^0c)l;WQz;%l*~E#{O4h0c&NuaiwD; z5%J!4%o>#8poc_nLyHh~&}7-s%heQP#Cx`ErQkDO*$a@Hhu46AlKV;bEm7b4TuK>z6%esI|#(F0tosHmt?lM$o#=7dmO+|g1_=S@*HH8p<|Ra8|j z>lsPFdim6nGUji9!V91LpHTLk|8=pZ8yplY{wXEms-#aX$fklrrQgUW7*()=1_(1@*$kRhg#dw7JSW%B1Gu)R2iQ z@yBK_6>#6RuJNNcO8F9aT_XII`{?g~dR|jU-4RWzjPC01|CE;9-q8Ua>bADFjPjbd;l~oeOY@>HpiEYrdlC!*>YdiWGG>_m{$?0ZD`OjWY!%%g2u7T%zXvxKt_YZr!85cf7exaaan(yW?;6x zypBS;bZASre+&qlzc=edAnyUJ2Yvm%;bGggEd8mIN*3CC<~^Ho1cqSKyH7mDmDm^_ zwWVQdY`;f}*#$}j&`jl^=+HoL^IVICK0TCCX1*(xEP%e0&VWkFxwF9FaTopEWlN(s=w$u1 zce~yq{k0eVC*-pKXU^mQRNnjFJD=ttkR2tXqmMm0I-K>QGLN=H-tdpDz85f@uw!RC z(u#=inH`S9q(kG4f_#?zm8k4UC_uw2Bd-3_%xc9Rh!RUcX=`kdE-+9=K1a<4oYje= z8&%wn??cg#cGei3SJKQ)OCo;|{B|R{4tzlBmiy()8o$-i+k}MB{H-ILsinVuX1gq& zde|tW0jO2*8jx6+2A$KR66cIt&@>F_J}Ont%ye`UV2aYv8C0w7cfTgDtlR;aFTd3> zjJ2sT=vcqfAuscfZ^V1DCHAME+}AE7W>nS4T_=(d44V~ zP}TiQTcBI)-zfzoo0KU1)?sm7YWmAWeN{=9Jug3jEWuP1XF4RJaOdVghylD#$Dfn3 zr1i5jL#@3=#R9+G{FC94fm_h&l`IXfv9STtu`4T%0)}4B1W9BBSJt?%%+T)~3J5cZ7aQNr6D8 z!tRK8$9idCXR&Yj*+f+Kt3R$g`R0X7paeq9O#lzw)C6c3+J%hw-+}lCww%jKa{}yv zfJ)HNnBLml1b5Z)=N@7A9@)+jW@zGwf$#4Kr+f)e5YXcQO;y|2xY%sRlzuLE98H1@4uWs3JYBVLP8p}zpv0T z&n)?R*3bf(j_cGp8?{8!NNZ;C`(UnuKdjGL;pDF5_4A`ie$P<6TlwF`TukM9X%@aF zgXHtDR9#VjkEaT~zpun2%M}mIME&Wj+WM&Hf1TlHfQu4GPEJMgRG`UdB!9jY0L`5- z+8WZflRodgAzTNE9R<0~2i!mEBx>a8{P6J~YVl7c#u~2I-gme`EJX*(ykfDBszIeF zap~8P?MNG6On?UC$Y*R-4%X1FJh*M9bbUg-RR5^1y9ok7FyR&5V*jP&QUUu5qz$YM zrsR(f8o@D~-Ly<&eDpx_{(pB4+1$?CQU0^Sn7r&=7IyZw-(s-Z2oeT~S);@ulkvm3 zCbH2uDc1B4pd1(;Tm)*-?t*@VgoHI4OL}z5cAw?@j-;QSJBnEb4$qNBU%ylcUHR>h zs(Sr^8OnManax|y*HmWzcv0~_z1*68-|x2+z8jSTfKr?6!fii&^l6eb!lYgta0sFF7d7%P>6SbSjXC!00i z7Zv(Zzhq6G8TVEhralpr7>5SU8+cP-2*=W1IRP+!C3*#B+24AFm7iv2oZk6?WF_=E z0%~pOkodWO&QF%X<_03d=i*ZxV|!_MgPF^HsY5=`j5UmmxNQcD)}7S|aBr6B9Z>rg zixIFcoFXXby3!ci(s~zeULad$KfftiIS4xZh&cKI$g1)Bxu}sNgzd%(VIUZD>9G2f zQ>VoLgGvYV{{xkisq)DahnoHZyCCyX((_}dX~D7oir$;YuHrVpJ`a^(m_&WjO}TOw zvnxr$z6AX;>gQGsxO)T=)aMfpi4yG}S)c!>-~W;tfg=S=|MMs~ks4Zq#Gs*8zuqv1 zuJN+eR_5jl{W-={YCP@G({s6!Lp!wbZ3!D24K71gnp2wWzjd$lP5|wB2c0`x>AL@W zAi!}FS}tZv%tb~<;+6~f3jX0MJRm1CovNAwwfoKbp|KK^fZ*r?^PO@aw?WLkF#G)p zz@~e@7AW|e5RDa^L8n~<1M~Sccfh<0JQaxJwn%{CTVnmgH=8dY3HboDUARwKzIleL zZL^!MZYP{2hMGo3D?e6PG+Q6YUh*k9?*uWHOl^%4S!E2aoqCQh9;KOS|*qaayonmA_##y=gwnK@xH zaOmzyF|e>Q0!7Gr3xuq+X%yZGK6TVcRXZX(07RxJkyj?c}GGGn70e@;Y*(*Azkp1bePAfM7 zh7kKsBDGLgbEU;zb&`8kD8BZOA6UF62CvM`TjxotazzNRLFxLu)Ti*P4UZ%T4^O0Z%x8PnD<>QAu} z!K0u^7qy2ORUB0Uw1Of7DJchOC09xDtzz%QyLOyBJ49EjBlS{~+_jf|$wzA&Z?B=1S)X1MIt^P4%ZcKq+x|4+x^=hb)tXUYqDJyrmeepac%x& z^TynWZmiwhMugpN1#{PV*aCZj2b2YW{qJr9=C3pW64Gvf2kxUIe-T+=@1DUWl>XJdmTnaf)IY(qfQMPpm`0MI6D>$@FTG|5x14dc2pm6ai zJ9*^e(XV|PY3HthE59xdqqlK_k~F_9NG?NPBPo}yO*C%x{ilu}CiCs8qVH6s$T2Y$ zTA$egolEg!U{qLKR2az?)#-+e{}YE&Lwy5G#EQ|FI*(0O3c}XWS;-!{mA!T0?_M8c zOHa>W8RtFU;gDRz>rpTlvhE4w(<%5|Ljgf;lL?1rEhgN~h*wkTPprk)S}ejQolt7_ zeztfp-p^3=Y?_wpiGNI36;P)_I~GAjLxqcjqdPm}_`%!m`!od^8JUI;-*w`LRzGed zQ2e_j#Kib)7UT9_-ON3{`1ZQFN$@Y7KP05lzs{@V6Y*rfmCvhZz3M4nA~5v^J~8M``*5AF-ESQvgDZ|-efb8_K7z9E(q?Cl$Y~aTj)N#UFaM=eAl5r ztDbwZyI&4T&$i{rJE|NN3BCP*(|OKyy}NhiYf8>V78%~P8_yi6|MuKzDJjAzLSk** zeFHZiq@*NcAz1qRGgP0cOYAwlcnXf>Pt_!ij{yOLgT2v=+Lcqw{JCMd2B;{{eoK@h z>G5k^MzNH?nKX=>-M}r|_$ix|w4UMa^N-S!FR*cm)6)~w`g|)@iB?cZ!*VBnY#gA= zp<}$0QEJss>-{zE{M>(YfXg1LM4WBS&EsWf3n4Cd{hLHYx*q);62b^u?ynLN%iLZZ zsq!^he(&E?V>GhU%5ay0JvS?>GB1sinsc|Tl!WxDcXy=P3win2fC#I#-s0$SZgeYF zwy1%Z4!=qYNE$c)b_Pj_q3=l5shEtHxqQnFdIQr)sZQl;e-WyC?TZS7mNh>Ada?(^ zkL)vgQ{{2zob0Xe@*9LD@RLRFa20gQ{mM4GzLht0m6VixWV2`>6I3$BmXlnD+QB^) zgo}lF3(@Fz{8&tG7R*9_za`|g#EFc_eF>BAG8oubg{Cxist-AE?`@!=cuPE|rKggT zlzhm@v@+YDo#{^0f&5WeNQkPrE$TKQ2L5o3rbFh{QNmwCmWIc8@iJ%TdnuzfGPzdc zyaPb;pTD}_6oqsAVL-d{{(sqJ_(f*F@AOu>Ps`Zkf{kXu<0vaLWHVap>1u!Ea%bwW~bBoD^1GZs^KVX_vR zuCJeMW%(9^_@Tq0?6YT{N;KqjblGX~@yW@rG_@C*(p=(Vc|Ij?R9x+UFP@tdDHbnI zR3Tm!85kOCwz+a;UBMYT5PG9-3cS7Qez-@-Ynhjk;k|!!{noequ|jRWY(8iaSX~w4 z(PwjfIzMsbADJQVE4Y1AX;B730A{Rdp=m*X;tlt^j5J*HGb3okY78>c(jmdv9in1$ z?fX6H4tSi(^`Ezl;}v8sPm60>8f}U-#+bT%LGX~?^&l`S8{hY+hc|`9#QIcaR(7)7 zGPSn0HvRpkp-4JFPAp|z5o)38|B?E6|9+E1;mFECVd_x137p&c@nd*fwx>dRVq|QD z_|&4bJW?r}<|1akJ!`^He#62*e%y~niI|Wub$+_g;9xB+ohpp$&Rc6IlQGJRtC(LD zxZ`Sy3iNCahaaO`weZm3;^DokcEUT{J?{?R>uGU~)YJ2sCPf$Ko!I0%;sQFM{<&v< zbTuD{SStrFGOn@EZ5s%?TdwFlT4UM2fTqFEj}p=m$*Vmf(RPhgg?>-Q@oS0dk>klF zVs7iGgXx(JJL;f7CPp@b?%Z|$d0v{+J?3|H>DE}8L#E=w;-EylZ?{Sfz2N}*tZcJR zLq*k_^#U5FBBTrUXlg)&`SN8?TE(*6%`z;Fb%y5`IjbW3b2WW>GNF-UR2Yb6_|&x~ zqjiY+mZ_+`d-0in*8^N|bOxtH88xj!i?V(E(EPR#oB6s2dd zSLvZH`H6Q+OG`vlWO!8M@k85h;VsM;-ymXZwijpnV-ADcW@F1tW#`M1R%k&2Hq;WY5FccQf zNDNFY)~>KCOc@&+K`o&#chsA!Ul{NZXu{1^Xma)r3k>A^Y?JEk%|!ZCEp~XsIMfpc< zW1w^>DJ@8cw6rWDCEc|k-5rvvvam>Zr?hl82#8B}r%3MtOE>I$`<-{@`6HfZ=l)@5 zHrP3w`<&~<=ej}&Evxl_y_%QMTle*IDPgDQpnJ)Hx^S6o10^xnt?rdO7)vx=t!>W> z$!wnMTjn>xfV5+lB!rh+(shu>9^lB{IbpwQuvP)0@RhsOhYtPy< zP+*4m3h;EG%5-GtguF6o=CIUKJ5ux@W+8-Dfx`PmrLK~a(&R1S5Bo}Ht}1ik9@ipX z0=;66$51s!+Guj}>`%K0d=OUV36CS9rf_E0ZfjBtz(@0zT;~KQ2R;rgpEgU$r zt6Q;r9JmQVBCf9KBryObwnZ(M|xsS+7ynsE62)&9b;7-6Y>0OYa|I z+$5*aR~(}(NZm?SZtj&@bk(5&A#Lkj%{dEQH2lHJs1CQWp1Qj0*|j4g&zcr2T3%O~ zB}IuqFQJ-|C3#P&_r8CNe=Da67Q zNnF8>#r!P8i~|3~^3H&}M{}HHXYEFf^&`QPazI#yMW>rBH_Oeg1gT`+`0aK#mnL!j z(8OfLmpmYfm5GK%h(%77V(&jGv#%^Psj)eQXFl1TTXkEqWq!&z_Jsqg77i?re9rm9 z5cH%)MUh0b7%VcKFvdr(O@9F=G$sAt3h}=(i`^D1b70U_=Y6BRk zRfVQx+V5;5Vt|J=MFg_jO1K6mIM`pVeVQnb zz7_I14V~vN)M3~B<#V=}Au^iL>9KYDYAvC;wy?i?erlDsI{R6C++#XY5|a0_a1t_7 z%^>+qfy*YDcLWj?-ZnoP{>P1d?dGDE{eevB_46SO7FI4_m%wUGO$}f^ck)W3rc=WB z{)M)W%G2UzrnUM|A|Q?b`;pUK9c5*EC9v5o$JXdfQDMhpXxhQYVg;u_Nrrca7U-Rn zISm4Mjb@{r+twC#VgBNx=d*ix)kaUy-|dr;-=3w(cXdv4bN#?hB9M=ER;JZ2JABeM zW!zOGR@YwuZlAO|~p2k^l^Tf!R2Cx(sO z1llK|Zz_eRQscLX6^Ad<-#AUB#P$@Ecvq4mg`7NhmUef#0t%TOadB`^E088{bZo{_ z8*b74_vSMmPp2JMSSYpcYk=ftyt;IM8tV0Ere1fFM_=H2T}8%o?+LZ{yKL*U{s*`o ze_*eQV&`5D8yi~Qf3QKjp}G=si~{e^1J<`sPhqDxv4sA zN!r?`dy7T9RAwU0Za2`se)Jfk+rrJ{|FyQU))V6IeRkaIK!!YDej4zEV70-|L{?Td za{y*lI)84rJR|xj<7RKU*>ev#QZLN_zQJY(fkX&k zE&|#=OKO`k{Nv(&f}SG0k_;t44ixfjStF+7=c|1zHDhIE2@94(?C!o)OK_Rcn1IIA z9M`1r?d�`LCNy7Rfv%^gygQPA`~jaT~b)J*1mbAN#c3$Smc)wYjOQoFBLY{6gpQ z$*@56NP&ZxG*}k1bGvil3|RS{*Pfl9pftQtde2qv)aShe{->W0l~=F&F%fwH*CT)& z7Y_meih^=a1HrxvD^RHWDmW|*2R@nsL9A(OX=NmvxSWi1&N{D}SEtPAka;MwvT@x} z(72zY1{qmyQk%wX_Y)Nt)~537rb~AOrX#$4jwA&u0yOHEE(Bks)w&v2tsEO!{LpoM!I{0Ba^Wf7H6Z z19c0qlA73CUP@j}V)?27-(JUQuFY}$xU$f>{KmF+uvFOi`x*SWUt;fQ{uFCsIGx#AH{;00?*)Lt)uxAVwN4N7|VR~9fstB?ChTNil zLsYtL1h_oEI5di@Og2KyjIN4a=pSYRpa#%CA(+wOFH}ANb{ifgiF-}CXlo||pc^XZ z&ji)0<4+fHE;Qn#3 ziQ~?Bvx_D0{G+^Gk`Xgr^wrJjG_vtwkPM)SI6bW;=3SF1<=0|bAullqW1cPZJ?m;0 zqm}eQ99)T?Yx-O1G<(C10I>HT6Vbc6}Op!OG5ldvw$A zpIFWBY3l}!)fE*vLSQ+>|Ln9m9AH&mlK<~cLhLql0Bt-I_-NZPp}dMhEFfNE?Ih&+ zcTSOqvQi;0>VK@04=(>+294qxd-*%lm$r)jNr?t`X9*Hx6?-%}5JD|+vv|Ep7R-_i1OVlX>> z-1&3N_^Ar@RX}Y;lvgIlg2&P-)3D@*tG_zxOJ>@oWU5$4HHWI7iP7d#PmSl~OuoTz zm9Hx$<|CmyKbMM0Q?WEI(CoFlwL6)x0UJ z7`)is+f0ArTh4*$X+(=kO^$Aaj1^txfB*Z;@)ah&pf;|1{K_)TVroNq1GOj&oD69G zyx7TXb+hkLY^h8~3qAZk0pIt^6dQBvd85u_be}|v`I|@d{PuLB<4$x((9<|AS+ss= z1SxG*uKmQ5k0{^aj^$63vxB)#m8)aN?X|jE+iDi?cRE{vpE|2Gk%$p0X*hX6HZc#z zPwyuWJ~IA2x-7A!1TM*YX_%LzhOVY9;>802MptZw%7-8w&J=F{6K&u3Gc)aSf1k%= z0wlLrz@=V&bdmijW1-X7P$`J$?^wxQyAs@vy}3}v`O{dNuF=Ynb%dNlfbu{ zg`<;$GOaT-e~G0F+t1pIJStOsg-M^PINM}gRi;E%CCKMR+FWH2ZRXRZ5rlK|ah>-& z4JR>)>Gzdn01G+c!|o7ax4zH%n!kxapTq;*46gfe(QQ6-N8Q4vP~376H;iY0bJNMg z`$;D1u1 zr=nfU*RL7%!Sng!J;j;N+RwsgQQ+2-P@?0-6-F(^&a7p?6TtQ|#2~(!no*+`%G)i8 z*ti;`sUYO~-MHV;CWM7+edJMNS%%Sb#-Lz+7Ane}-rp=)%^v)J%_B0X{pNBy%wdFN zD}uz+M3qAZ>2-E!YN29OCQQ{@l}2}$C0d)x8(1mL7OiSuAQu3#=%<^f`i3buQoMVq z$*TTlHNmMz4cWSUlz8mwvib_L`u)H9Om{v@e$Dv4O{C+uLkt2SOJqTFqlu|9O%3G_ z0hrorO>T+CjlLf5qtxK*w6H;#C&hsclV0#3kJ#=DuUQ@;?sl~(w#jObPwXO6$`?|b>js|M&`DK6OE9fK($Q{!EzR4}JRbkj3 zZOwJc{g!%Fkf!Lc9p$JKxs!8mV&ri;ezWR~>E}fEkyT)G$Y))`QeTiaHa

vv z-w^Ru7So4d&DW)-mw6A6-AQs=yt3w_xWxGF6n1)C&`6hF{Hz2n~_4fH@G=Sb-;Udk|s-+0#H z!Q3<5_0N`hKiwYZMX#)$L@SSQ>%^JQQT*VcaM(9Jx9r}ZjCm(g=s;LHs;^Q5Z=#xJf@JTENpm6Lwkh~P)jJr;NuXo~LVc!u|J0nHgXUcbGjnuZEli&ON zL5Nr`9bz+bK+K;PYo=@1iCrf(EGM3~aHFO3dmW*~wE|~T4L1XQeaeqhZcW(eBAD!Z z=am4{!G%p^;{g{p+bd{-+Z*!x=1Bv&R#+ZQrd_Byb0%}SL0N?ghcS(#ZZvuKgO3jW zW?CPXHh6pNe2tw!8j69^xmUC94@*Ww%qMK51z5A}2qnUTOI+U`E?(x7^a!<_&A*#$ zS$ocy5=IuE*7D95!K)Z=i!e*+6 zP@3)#P2-}VNNcEpKLrtaeA*}nhcz$zMTJ4BcRzAH@8>53sxN=Zk_GJMvuWN!HR=37 zRI+emi^R<#QgFwjV#)}$Ub9VmPU8O}HlcMl4x;kFdcNvhu$+`Ak=L>FCYbc@H_DZZ zbB!1jCUOhhFjTf2%^vsSLDe%+)%ym2Fr%>$lh3$U@02G72}`;qHf(J`gi16*G9fBf zAbA#Zt_xuWx_I|t*I8DKwV3%E zkj^`^$>T+;AG7to6iqI< z?*Z9YBtXa781sup%8Sy$T1^LBy@^2HRIW4cxgssgTdfkx8^V@qg}-sA_-1=Y6Fr93 zd*RK-Jf;!6k~#4;pejyu8(i@p5WN%lP}N+w6z6?3uX*39#wN<&t7aHW^ei<4_L&GH zC$;n*?q~BhRbK?vlh=f9SBKBZLxv|r02Bnw?2p|X7SEGvnWNWSnq$Yap|dZ>ir~e$ zj~Na>e1uvu&u4ETG zxas42Bevj4aztOyb_)W9#(n?lVy2)_($;1qvpl4~JC>(rI5fy%6QeAUIRA*#`i-Ft zdu`%os#3@Tg$-K>%cch1@>%4xPOMmSzN#N1zDQwN$TPnYj6q`K1xZMxJKhV@fCBL)f;0T zYh!MWW3~^SX>ID)oEI*@wl7F3eb;?chNvtGN``cF%|;QP?+?cG>gR@BylT^OgVNJzXk5`TS9vFSEuj&rSk3$HF?YVd<{EbvWW zshKFE?@aJHsa9BQL!7vhS4xn2ve&|RiO<(tz#~0nlTs_f+DnqCHFcC@rx)K}bcgZG z-e!j|69I(M{#kEXMfqiwzsO?sv$K{PE5&mI^@PR~?DrXiB4)~2))7hg1d3k?HWhcR(+iYm&&d}Gg#MhJNu0CZbWXl-Da zg1u^LTU^Ov)SrfBLOH3W%Lq+*K{YTCC?a1kFtu{$9nG4Fu53wCun}NO)GL9ov$4PN z+`pX=b~%V%2Ry#*E+{N49CnI&WoBIx_t;S-Hr||^JQ>TmZ%*ys&Vky_jR{1G>HAyI z`WPcrD+UK7uax>$@3~xUCAeVjy$r8pUd*)^`>VZugZcgA>kT*aoZd=u{B!6zOAV_;$9L|5x?`Ax;ssBAP5v9^{G=dI@#-7C=$ z@vAgXu}RIFM%DD!VZ9bjou35#u0rC;E5>iTrIm(X9bNig3wk3K^bHKI5&KW6xf|W2 zwZ}L$p#~lE;SRLCjbK9=?apVBXx73pCKFj&+!ix@(AWP)q@i~@?QJ9zV9Xs&e$QJtgA=N=pDV}4$;k|5<=~ic zf393Qu_VG)GWvy^cz)C8_Bc(-;j=PsF}qe4C2NbOf@UXyf2Q65S7}{t$PaM959#Nm zpSDS70|}V4(~!U73GGKZa6%PFdA4xq(v~XGH_(d&y9`C|c148%&xMkD^EWBBopXW+ zNS>oU-h8?wuf)vCikskP_S}@73v5+)GT)Bk(4mLTz0@H2>9@cUhcqf9hTHdwA{i4J z1nI+WC#O2v>frWI4b`z7ZPbPU4VBbR32#Wg2B=AB} z&XAv-xuUp9&n5OXHv^k6kBO#n;#T0ey1n2pP5bzn{7hRzbD02LLKMLa831prlqnfS zr38a{UUA91sfOh(r(Y0nS1c_we0-JHV0}2o^yaj(=rZk&p zu2!0qDEt;f0c(6#3s0%9zmJ31XmW6f=Po8@^1V##0ud=clNH&EyNgewzskB3GG2BW zCtRoDI7?ZQCLKN|$ChH=w%kbEDb{h1RjW&;(+F9h3e?9_dIbf-`b5c-+-1J0GLbHK z55xu{oK@NK{7bXS%o9YILr6s3)a2N^HTX~{Oa*SrZ*ko)i(CXp-(Gw&l;d|y9(e*r z#W&y9DDq?!v(?he82LV6_+>=M^9K{d=D;PmiWpFOF0!`BY;)lfV^P(VQ9ex6dT)?6 zY*np4^fyieog>ULVp3kuN*j&tl_*!ND*x1^1M|RkQ z1EqmEd^;Ry%=uO17n@psRK(gvP612*=J0HIx72SE@?@YG0XjOu=m59FUzlA80sECa zrS1B`tzbyReA|twf-To``Cm5pU$`?JMT`zc?%?0e)87XQ3@{*gkIibgg zj=28jkHWQ>?DI@C5AW1M{>!FQ6zLt?{pD5ZhbKMFx?GcTi#v$oAFPmwC{h)VgJolr zN>4qDQ-pFW`0|Ox_6A@oauPDx_r1Ptr;K&ZXJxxU~G-ZR;hyn_9G%EY7G+nlg3s%&z>O-!O{~qz&wiRa)PTvv6>g z|1|64x5QQQ83jFuv>zL^xk0}jP#6Wy&l#&8T=xF9Bssj?ENSiqqvEg`uv3GfF+gwy z+7@4}dqPJziKydC3j6r%%MH*2k1*cmb(407=(l98*pi6p zWd;d;LyuI_{FmZ-L{J+<2NRO|+`9a!O8Ilf6wB5p(E%#^3s30m+`5*kU+HTlM&S7h|;Wg;ns zcA7_#V;5^mvrYvX5qPIi6U=dXyYi4vc8(gvz85ayepgm~ivSOFtB$Y1hk9R+YCps- zqyJS@zo`Sb)&jZ0Sbg-(k0A3$LT7tH^A4Qko8&1LLjmh<9+Y)>w0D@Zu`M>Z!0Y$pa2iJF3mcY@=UnKsVFBUXJzF40YYXr=7}7ipDo zpBPHvh2n;dP;~6)(vPbt*&dCJ5~Aw)#}TIa#4pmPA?p&-VTX*O+k`bONOI&EcPfrJ z76X~%6kdITg(c!Pee12^U8e9?uNACk`L}#4SuOK5Gj=g8bwfmcJxHFsa1nnWo#o&2 zU0&C6bnSG_>#_8fO?m1XI_jA?C5j&K;$r$QUg-p!)Vkn_q#iDbZQB^5KZ>33ON_Q)4jO>ZvJgdIT~T*~_T z`arLtlzg-h43fu+jg6hJv+7^6T~}UwQI+z+N2H-T_r8gkdvIHoymE^ma{=9Er|w|% zmq+YEH;tW{9&C@*H#oSz()POm;taI7DJg9|1rxhQBra~Rr4R$O4A8Ea<%W;F3DrHP zXC4%I`oOC|iJm-~*D81HwkN2xMM1fPEfria6QZFNvs&-tNhl{KB4h}TT3)W6obepA zw6KE|2V_ne;2-@9Csq2KMHT1Jn68oUv@DFT^C8Z_ZsUol{2O&crS)h>B>7Ov)XI-c zz(x@fbzwDe?d8Mhq><(G6U8(_jUVSgEV=9;VB^|s8Y>0kzT#jE@Y3%Did3L99OShm z-K5aDwhM>N?WL;2LRHkGTwSu_WvK+$U8>$QW|1AYGAn5IbdXJ-kh9jtSHTCw)?hRjjjw@-+a zRwZoN;{_}4nx7+?o(3q(hLqJcXCZ)e)6K&`VvMhmoRqq-*ajUWZ##+UNbC{grnJi| z%Ke!Dv{K4z0_>70N6xMjVJ}e`vP?)oh{AU#eg z5}a}0VDeRhI2F^!DCjv*XtT8sBTU>eFl-z&3h-U!hy);6Ja_kRe9s%t200{bNOB(B z)wz-o!Gi1s@ zIcuww=K)5J?HUdPhQMcF_~jR!9T0IfRcWnzZ9kkMz%0qudx1=*7J3GBGXP9N8UG#< z<`F{zfv5{n(=pH{^z(+YvJwimoG_X8{_@T#^Kj)t3quRrFd1-=uc(6s5cm& zH-P0m4E(o>ii)~&|C;h1v@DO*bX$u&C5{h`Z5(uTK9*#)boW&_k~hx@coqOVryWmI z013)}5Bl8Ku>TIIV7vt}{CB_w10zNa>%YTX7#K_>5C1#ngMon^3^V}#J^t@)|C!-` cyFt)TWmIh$-&<#}{WFJxtjfCzkm=X|2cxLAQ2+n{ diff --git a/telefuser/kernel/sol_attn/interface.py b/telefuser/kernel/sol_attn/interface.py index 61aa569..6fc48a2 100644 --- a/telefuser/kernel/sol_attn/interface.py +++ b/telefuser/kernel/sol_attn/interface.py @@ -370,6 +370,23 @@ def sol_attn( raise ValueError(f"{name} must have shape {expected_scale_shape} on the Q/K/V device") if not tensor.is_contiguous(): raise ValueError(f"{name} must be contiguous") + # Use the production CuTe path on SM90 until its native FP8 mainloop + # is available; the pointer Triton fallback is substantially slower. + arch = tuple(torch.cuda.get_device_capability(q.device)) + if arch == (9, 0) and _cute_runtime_available(): + from telefuser.ops.fp8_attention import dequantize_fp8_per_block + + return sol_attn( + dequantize_fp8_per_block(q, q_scale, torch.bfloat16), + dequantize_fp8_per_block(k, k_scale, torch.bfloat16), + dequantize_fp8_per_block(v, v_scale, torch.bfloat16), + scale=scale, + tau=tau, + thresh_type=thresh_type, + kv_splits=kv_splits, + sink_tokens=sink_tokens, + sink_start=sink_start, + ) from .triton_ref import sol_attn as triton_sol_attn return triton_sol_attn( diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index a68422c..dcbc535 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -148,7 +148,16 @@ def _prepare_sol_qkv( v: torch.Tensor, sparse_state: SparseAttentionState | None, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None]: - if self._is_sol_active(sparse_state) and sparse_state is not None and sparse_state.config.sol_fp8: + # H100 currently has no native FP8 CuTe Sol-Attn mainloop. Keeping + # QKV in BF16 lets the production SM90 kernel run without paying the + # quantize/dequantize overhead that otherwise makes this path slower. + native_fp8_sol = not (q.is_cuda and torch.cuda.get_device_capability(q.device) == (9, 0)) + if ( + self._is_sol_active(sparse_state) + and sparse_state is not None + and sparse_state.config.sol_fp8 + and native_fp8_sol + ): q, q_scale = quantize_fp8_per_block(q) k, k_scale = quantize_fp8_per_block(k) v, v_scale = quantize_fp8_per_block(v) From d397790e2872a87a770709f7e1e2eedfb8bd8c1c Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Thu, 13 Aug 2026 09:59:38 +0000 Subject: [PATCH 07/15] bench(wan): refresh clean H100 FP8 Sol results --- examples/wan_video/README.md | 10 +++++----- .../assets/wan21_fp8_sol_h100_benchmark.png | Bin 43197 -> 42973 bytes 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 7f2337d..13ceb2f 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -231,8 +231,8 @@ reserved CUDA memory. ##### H100 benchmark -This generation cold-start benchmark runs each configuration in a separate process -on one H100 80GB. It uses the official Wan2.1 T2V-1.3B example prompt, `832x480`, +This clean-process cold-start benchmark runs each configuration in a separate process +with no other GPU processes on one H100 80GB. It uses the official Wan2.1 T2V-1.3B example prompt, `832x480`, 81 frames, 50 UniPC steps, CFG 5.0, sigma shift 5.0, and seed 42. Generation timing starts after pipeline loading, so it includes first-execution kernel/JIT costs but excludes model loading. Peak memory is `torch.cuda.max_memory_allocated()` over the @@ -241,9 +241,9 @@ same generation interval. | Quantization | Attention | Throughput (frames/s) | Peak allocated (GiB) | | --- | --- | ---: | ---: | | BF16 | Dense | 0.854 | 16.147 | -| BF16 | Sol-Attn | 1.083 | 17.023 | -| tf-kernel FP8 | Dense | 0.877 | 14.855 | -| tf-kernel FP8 | Sol-FP8 (H100 CuTe BF16 QKV) | 1.158 | 15.730 | +| BF16 | Sol-Attn | 1.105 | 17.023 | +| tf-kernel FP8 | Dense | 0.879 | 14.855 | +| tf-kernel FP8 | Sol-FP8 (H100 CuTe BF16 QKV) | 1.146 | 15.730 | ![Wan FP8-GEMM Sol-Attn H100 benchmark](assets/wan21_fp8_sol_h100_benchmark.png) diff --git a/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png b/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png index 0feaaa50dbf4aeb8c2fc158bbb18cd7e9cf1374a..64c49be8b4cfd247ef47d3c8d93a574224112e94 100644 GIT binary patch literal 42973 zcmc$`cTm$&7cGjSf+B)S5fBiS-UI}cs(>K9gLDCcR4y9i;aTkrI$z zLI}NuB=6w&&7HY--hcPLcV{voCck#h-h1t}_SwfE6(t$sThzDk@bHLb-%6?B;a&TM zhj(>^@GAI*#Ek9~4=*oUR_e8e+sroH)lFR!zk4q_dy~%T6=jxh#q3oRRFjEa1o5~0 zj7R&Af8gIG|D2Iem`_O9H}4rg8K%lH4YKV=-EkGqW?>LaXh2;oo9S5+eWui&ZlQN?toYO#>1QcdOn5 zc|M5{EW18m>n!{(Q^Rt7s#No!<16iO!ypB%hlty0tvUJtwnc6rT+Kb?n*PG3D-9D+ zZ8xZ(h>uZDOn9(Fw$185{d4ePlVGcI4rNoTzXg*N2HZ4%z(;w67bA>iv|<#1zY>Q&%@(z5&lr z4lhk3-Xi~S;^5OU)|nngE%0JJN=7+_?&S_KoK@VX!R<2+W+>%u*yQ3pTiTT8H7t5F zTtPvhs-|>hd*x5L0pc|VUE5ICyfJAsSq==D|M{y5SG?f_Z5dN`Z!KZ!dU-OG+rG zh#g>**&AFpPurv3Xlccj9cNHhibSicVNXWe24`IhCM}U2J^35pC9hqI`LzUFnsDFndC9 zhn_~jv_dY2t$|tcquSWWXfzgHqb@ikdOusU&`1mO(thiiQGABOXAWS4ue&+d zCFtQ$<+djh675cTEM!0WVWo%?XVmZZjRZ5@Vb+2ILEE`LX3IDqnXuFg1Pa2wwpnxn zU%hbyV}vMZXx+P?XS}?$uj`EnGrRiXiQRD0PN_-JwZ~aMTX37ukG3oOldYV>F7fd3 z_w(IG`i)-I&+HKm^bpRyNqH?DaWETSUY$lgKeEVowMaLVS;%4kS}#atnF^_i9>Fb{ ztX4w8?X;3S2!WH!+@`Iy>~y6Ze(3Imj>2t9$~RwBRiu{8f3$+xv9)z6hJ@0%dxp{S zTldDN)6Ja?3)?AiVeh01H;Z3jotto7ArA|of8Yq8mz3jkuig!y#GFR>47}uGExSuG zbvs+JRxs3D6KcwF@Kv7EB}4)24u^AWL4fQyi|9uB;E{@ zQc&AU%w=#66*ExkkfO-LFEap1>)#!Zb2H&E9L8 zG27r4uj9VVi`H+H<}^%GPZrH+7wA(rXOx5vxTV#AaEU`{L)8L}Sy$v_>R7e9rxijjr{%MPLI}-7Q5Fp$Cnx8RUy+uY zsdn>s#*|pIpqZwhK5YxTNxAm5`ZV#|E42?z;E{~0kv1wK~6sVRMAmXyVlejoIx!rYEb$Ou0a!mKaYdagY%KkM{ zbyQYXc8y}~IF=pe08~uD|y4G`Y^z+<0*sG-=ky;i_IjNyDTwsO`ZWstz!GR${D2V;w46#^fqc*m7V!D?HM;;v&oi~`^c)D>D8NJL=G z3sY5f*lLFIuo*8rJFtB!Dr$gA(~GB`>|5&1*rcCI{plK+NBYxVQW-8qd5@anK1~so zSAIfKd|(@WusF=_e5uIpesq)*v-F#8sW;uH*Y&ZB>za1TsV_X&4zC9Ea!MFf*{sip ziyJ*8r_B=P;o=hG;=*dF>;j5FEqFN7fJ9MhUlF?*?&iIeJi{l>$%)z6QmWFD6uk6WJUTtZ}^7Wq88oscA$cK$eLB)hSyJqV9D_x((39!You?elUE*kqvhry6^ zRTULpD5S-3b!x*2y#nOKH+$t1d=noX+p83W5VS)m7Hn;23wBQTvX)id9Rxv}?R%PH`MQW4=VTPu5I+7SoU5ewcu8D{n9o%x2c1}zVd^LhLL zXUYwnt}^_Inn%jZ_4U8H3%{$I9TmBUbCF$2KJw5-#B|2|ybXkM?WJGk8rO(zo^#%O zy|NXV#32%%rF}BgIj4z~*}%jf%rxF3qbU;iD^?6c_}CPe?CTaK6j4@Y+=6iUpfTJ{ z)N6yi%lKX-;`QDZ2j~2gMHtts+kcu+>kB^Izw6DB-yuFaM|nFj0Own{jDAvWYz&k) z9f-S?wOW+~>s$d$592bD%4p={!MrQ+1+^H4)I|(oHYz(|cJs&k#Srt8HMHPW?7KRB z$18!-rlxB%#JCNcU)K=FMX*w%L7o(6?AK?E{LUS<1Q8g_KkdqRDk|fyvXJ}XsCgu2F=NNU_o^EW~t}lX~Bi7^J&BtqO~T}%Eo(ZYY~S(kD4AVpH$gZzQR&+ z4}0Mx;jE(dE?eW3R&QSD5aHmSuBbxcN1<*t(VuadS@p(lgbd$_cZ}BLj5jiTaM+x$=T{j^|oDLy&B*hq}X^l24yLWedJ>JaXI9=C-e+3(j6yVO+pu3!BVs zEYc4rBez$~&CUPhY95Gdals5srb$}N%feYq4PsKF_Ce4&uM6aBo8iI~G3SR>mDQ2` ztF~PH^GQOZCp|k;oXJfiw?6L_LuE+lt!+)$wpU(ocZ;Uv<}x9)%65y5U>zN`GB71@ zs+Y_1d3ZYKbiD!LD7Y6buDA*ETAbqc)NuUVm6!X=8N~l^<2#tfQ>~(Md2WgC*NkgC(D;EEfFX^PYg1|bCREgwN7tx z)OtTv5MhU_Bpkf2F1ojT5k-?2)wy66v$}(1WdYXLVUJGsDqXLv9v)Z8*4v5FOpJTe z7}WNchtY}iTbwr(6j;o#5;sv7pY9-je9jevxeD^|?2H%9<_^wQtJk1r8s5)a4|e8w zXh(QVI34($a|9?RLxi^ZGt*e=45@!#usE_7Fdf znU%UbJAp4Rjgt%x4jc-Y8`~KCeI?vkfM>!xo&UiXhyK0h_Rbv8J?DVfLL+F+&Rpa7 zMb3iyCL06=1XnlqifMd81XyB1J{yyrQX2NT2=A9nmXyu)50-{;7P(v=Z@6iGp4`wqb;)*q^S5~y18$qDNwUd&P zV$kDNgbUsqtv>Z%pYgJtPb41D&eZ$}4Hotp)3)p7)#U)j!0G*Zrk?3$yp-_R?kNxJ zCc>m2BpM2?Q=>%oD@_ZLq!M-!4;ZARrA-a3$Q_UF#Usi`&<&cH`f5kRpE~{R1~i2J zqG0%Ixe>`De=(?yM z{{H3^QX()MCpeq&8TuI!#>nnx4zHL_Z^0z+y~hV-3a>jmwxu-!0$>JaSl2JE{``qU zd;YRL$Z;RP`MU{AXKVeroPHRE z&k?dD<2~|f5l&y+@-#!gdxQHafaYH3KwQ+vXyeZyu3j{KU^v3=WHo332Q__7zW;6` zbPXgCMFQCQ8Kk4Vn}Jb{a`UKUbH4bLm%TORI#)c+#tn%kI)NOC#NjEq8e0H+d&Ndj zLiWB-72M*f%GB5JVs4C8V6PvR9yI=?%s^|~z`NgW%uBn?qkhg4`-L`&t~%5mBSEG) z(!C8-oU_|-9Kh)|lo3e_1XJWsPwB&~7;b}MH;Rp;RF-smm^{#2=<>dyFk z`{o9z$D{Hdo+J;Ispy^2bhR=ixB^sf<9=eFY^D>(T^1!~uN7A>HIz{U?oQjnmCmGnQ%+8aPShu9 z$2&?QaGEap*A zyL8ZY!n};Q$3wL{Y;jQ~v!NlkiCR8a7Y@}k4@d6?UTN=+x_JdCq9x1G0hzMVvL@yB z(cgwA1s^Ha){eG58TpgnrOIg*6?zfli&7tBUJ;O z(~g<^9*jV>WlIdz_QfeYl%MQct8kaK=$PXgoQ}pe{OleztnCrp75Xb1{<-*e}KdJh#;@6lr!*Cj~KqgVm=nn{zr?F0f>l<0^z z!t?ms4(T24%Lr^D2%GS}R^wlHb}sHiC6sSeC(bIr+kkMs$CfrI(&XvFnHmqL6L-#) zN4>@bgkp^0UCB`2sHkX7Hcb%2AW^-f_c`x`Eo^e4Fxv^CXCrga%jzDCSqYZBs+Q%U zBcj(8N|%ck*p@x@v*7bZ&GioqG=dUcg6z06e%f5?E0lEi$5O+G{vs5qLRFLAyz|wq zG~H>mR#V$Qf9MV~$2pK7@EHtS7T_uDG-waW8h{j@Lo zd0LA7X6Sdi&yPlFv&Mi(?UaWkyl{yrDCh-IEw2A7pHT2vBtBiM^rq+jfb$nnypohr z?z1|BOqE!|>Q^1TJAhn{Qs&!n8KH%jvv0vlyg#9);vE&h6q2}=z`D~Rv5p&WLzhCh z&+o~h=B`7K&)%MykJVhisvpNlGhrcmKvyGUR|ei&boj(EixN-**QvX#?0e*dHFznw{=?mTt`R81u-WE z(-~3|tDtW~7t#SV1Opr5|*Ej%~I+kL-X zcP&ZP4h6 z@{y4Eslmj@&u^cvmXq6H1<(i7x3giRs(%(56-+HHEzPXd$Kll`2f5y!uH5@0XE_0N ziPPZ=Txq4n|T`>GMZwxFtfS;^64gE={F&Se_FhG_dnrq$%Oj%`zgFAm|_wto)f$%6Y_fkJ{{LOQAzECTq-oC`IBs~@6UjB+Z9c^;k3H#h zHQA8oJjX&wS}wl&>fH)vVSTyU%@1keL63h>H-%I#x&DY`qsIGkYbZkj)g+NA;;=PW z_oXPjy=|P2x449ZeRjIae0wt2-I!1Pei)Y@>H5`DA2ecyWIk!+Pm$r~bPVZZ23GfV ztTAO7JTNRske{b?ZF>ajDO2WA{oHZV@@TbHuiFc=BkGkBkzj3XquEm9*+Q4P%|2@h zjfhHe=|b)6;Mf_IAM^6@UEa)c?3dZPK$o=in?LCt*()a$UePfrV?~a$4&v2e zY0bY-8wJdZs<}*Ep^aN%4@dg@#+LjkB1s<~%66Q|!!Ee4&wD=jsNTD^UM9}=IT?bj zJea8a^1~6Emz}GaP+-IX0)#!DgJpkl*7I~C9k-ppYV#9n zX`m>vTg2%i9;7mqIa<0N=K;Y!fj@#K8wWL5mhMNF!f&SuxtMoP4f>ISBj17IyUW>b z&jxt$`NUHWc{dbB`fsxOP_KrWNo#c^h?1-l83nC~}Zo{TUHMskHNUs*fT1o)- z&Mvu{ahXT57FVy8t%r11Dmr{;4njtIU+}crcH-i4%@LQS!q#$buh5l@LY_6m6z#Qzg2U2-HsTad;BaI1_p0cua*AXQbm&)Juy+0mB(#8*K9Yu7@bu0 zBw9}I)D3F4xvU1CZX4iuEM(fhBWZTQg%aK>ve{2cPI9tZC^Wfm$K1^({r$V3?@p;c z^vvG;G^i2_7pn3o#q2dablaRMu~_~NJkDsiI$a_#;m;|j=N+oPRl6|y`Hlu3Yp=Gx z76F3!Tl00e(_dIoL;bt5exF^c?56TxS4tB%>(wb`s!-_LvBaKBp(H(cwqP(NPtyyfp$qUK5bDWedwAyGf;!;)$4C z{WJ`xwGG?!j7A}1F%A+n=e^spc|iH(=f~T`t4CH1B&db)=}|*I?Gh~l{U-am8A^soz299(ee-aRNv^xj=s_0A#F&*`=$$^3omlGRqCK_N5V zyHeKr-+Y_kt%e3wl=>EwfnlMu&bv}>pANHC@rA1`(qdBfhllX5XqR>~GwJChpxt0J zGth<{B|WW7jh&ZXV<#6m%#o_5Oia#Jb;Rx^$;XHloJp+Q*vYmODR?JAd0l}9z)5p(2h4KI!`dkbcIN0RyD@;!a47s}Kx^F1`X0~kJliDBQEmsB@irc0eY zM2!~gYK|OW4}YM5H%VwEP?pVKPEO^*Ls-=Z-;^Rf9o<9J^ybXq-&VcfnaYn1ii9ak z)*$oena^YjTXiw3^m2H3Y+8>*C_ifw?w$XIRoQ%N11p=s%{XKivS9{-;?}=)yU}?) z7&rLYA)mvq7RR!ZoQ|b2TTR{Gp^|>%sg~%WhZ%yC|tMM!`GRy-=li& z0CxyRXX&CxxcB1u1$tqMHKy;FGufQ-1ps&exJfV_ovn$g!*B|2$UUl(#CtU9pEo+D ziv+(lUwqD86QjyetKOY8vb(+j;~*`2D};bBvN7!hH@cSg6dKC{?HZw`O* zlWX*TZKRojWStj|XJJc|HkEc;eS9qJ?40o_BNTFM$cl8Qp}G%sK4D;xik_Pp)a~k$ zmLN3iW|JUH)Hol?R;Bl9-bTkFnok#m9qL}ceRrCk>@ui)%@#3 zLSCI^&yL>yP(7f`)F2X3yFRtWy`gdyR7l8_ahu+c<7p-w%J%fFkis74z_y_xh&#yH z!Rb*3(I?9Q6pr+ct^d@U9VajQk*Wn%X^wvf2w1qa^8QiujRnstOH3TV>EoTTp^np* zuR=qo26TdrkRqn}8#itcgrRL`nsK|mj@N7Kxb$y5^wAW?bVqcRw;KKHweUnn#a@Li^O{ugoxBQtwUM#Xk)os&sdkVE!_X%R3`a;aSgc2_Ll8L3AJslv?wEb+*yP?H!Je zzwqVi!{l;ejI=7zJ{HK{@_ z`MEUA+Te+Vk7=vRtY-m38I0+xnNsh*ZE)V74zZc%_xPNS^3gLhXMvb4u@oG(Ec$a3 z%-6di*P^ah1~;ac+uxKR3}ADqXBW~cZ{Pr!D-n7B4MTALE>k$&;dz9Z@dEajd_&wz z)bnxSDH1w=inZH&EneRH8@~s=8!I#l#RZTrmR--H8QB{VFgme}ywF&a)=3)iVqM7J zgfanAigj(BhpIgGNtA%zVBzk6$EuTwOqKa!`iI`=teW9dHjUhrt7{ zvqA?jC=Z<(_4f5e$>nfOrkW|xl^Dzy8J55T8f!jv(!0`&SgLeO3OBjNyNC)`k2cc% zDeAX3YLUrQzIAaJ29g7E{`EnjBXGF&A#53NQ7IAMc zMtLFzKfO>(lrlYTD8R_D)=BYxj*?L=OFYehw^`J&TnmMm;Y_aCSEH%6xq-2^Mi)iO z!MX28i08#d%nhrlC&l`5{0nfFUA+xKcVlY!Z(n!ajKTOrq4=ARc^1}kEv^)@!0m^CP zwAkzyEqy;D-#!JH(BNvlsy3*TsWE@MzQOh&P56W!Deh5kcM`^ZxJzEwHs`(30n&(5 z-XUUiv`58oC8>olNB8fgZMpX(Z(g=~P~~=Y6BuB>7-wW(Iy0sdPx0+AhmT1X8joJ;|01VGz z{shFcu+Qlila^9%TxE_SU=5S`TRJdFPa`9-N{2qkC8(eG%XChZtme)xmF;Z5E65;t zAOgE%MYi?|K^u9G?DJ~ZFvJ@0qGRmgo@VMLnk%Of|M}&t>>$@CgHoL2nyT?vo}~~= zy!8fT*HZlM6Z!*kiIv6zFy>RA)%sc29mwIlM!cnxhhz6`q~|K=(P|*6ud$oX{@@MQ z0lBxoy+m5fwUxqDJoGlPCR?mr&KNOy5=Uk)Id@f{oZOJHK>}ZPO+)Z;bxC#XnlfFz zZ}rTKM*UqUsFY9an_@P_5NA7So5f}1_Iq*kI3rfO)avSoyLZy1*zQVvDmdcK8d5W} zc$`0f*vK|;m5Pc=%h6$Drqm>0N-@PNU+Slu4 zj8acR-^=C55dz%XJJNGm7dT10a(h)s^jXpkwV=6=rnaVR4%Sd}qr7KEHDLjWaHF9H zAB56KQvk@_xczl;kge#2i_@hZ_nJ)|>_~rI`1zTj6mPh?9Z;;7;csPM`mB8;T|x#u z;m0_CcW__Wq1_R4^Wyp|r(oS^%baFv>ulD&d0%?28Qpr)#@O^KAtB~sGTiE?@e;*= z4=CtJ4h?%exR+bUd|+>0Yb!6r#jw{?a3B%*K|QU`&`zH+SCgwHbFl_8 z8o~f&$I)UJBHf!oy%L1#_c7Sn^^#}RSs`&6gGPR8oV#zE0!gl4b{9QNLc86VzhwwD zIr*kDeoMUmOP2R)NXP(+a%d2dzLJ$U4A@g6pGAz7v)O&)Cj}Z?kk$H~KljdxtgWJl z`Kp188f&_;Ju~=-L5f^>wlS{kqZ#_yQw}j6*;(V;9$M0b(1~1d9dLQF%l*UvsPchIJp*F%3q+PZXEjv zm0HVf4Q4ce)c3=YK=a`ju#O|u{$opWQY|OUGSS>DlL2#VRa7HlI8EEm?_~@XKCtyrK;`J4kQy(gCwK zsx59Qt)b1PS-y8OzHEU)LQ1-OjgK9M8Tf}p9!YL6d09ohGccJv-V zs(`@(h3CZ8w^NxJ$&jzss|z0It4^vCmU|t2_au=dyJh0_aSS~1uu-EaOT4f~0f!&J zsNrL88G3&)(0!>OuiWB@8-=!D$?wPi&E!kzf)j)T;>6{5YCWV^m;0Kb1L#fA+XtG6 zR=yaf0)=Y__k8DhJ!=g`S&#ERTfEI6RW@WJSfc;a0zp{|BrAp3Ja3})?EDimOc@JV z9{_BW4~G4T`e=>+FcIfZO5_`j>u<1{6leDZZ5N;>!6bV8BtEnrU1Vt6V_b<4OPFlPSTtUIpR5OC<;tnT>(M_+`$qY{t9H=kN1RclP zRA+GW3GtrUBt#N9*;0-2 zm;Va=W(G-u!*qdbXL&bYd9*<5QlzSRaCmi)*ez*2gd7;LzTju-?WTqrac}I>;9t-B z&c}WaCi03*cSBD7vH`)|^i&eDYC zJ+iedaqM~btSeS{C&|hPZmEF*inQ~hx;djTV*_vjxwi^J9_MZ1!gn^5iZ^wey2n)w ze|$cDdapMIM!=2za1^P&&aX{tm8VQSy0GMLU~bTnggZk7v|SiSLP|Cpdn!2a6MLew3a*A5Kv86%2`qyqx z$bf>G4BEc^+yZwa9bvla;H#xb1KR4k7@wM&va#vtR9Q@&B4wX&4h&Y1q-o65xEtfw z0YH8!D%##R)>v)^FPB7He&_1J_4j>JL!DPwYUltzPfWaJ#U3o3=DhGwyfGm4xtTN@K9^83l4 zug}(WrXuKMHd6`1WFLyt@{SyJJx0t6m)7Dw}Sd85HXlj|16c?8k$q+49^z!B5#)N4f zLK4)YI$wHeZd-ddB66~G85uzru^gXzG$;%(hf-&|ihlY^o;yD*E+l}hEi>YDr3B`< zBsrBpdWch;!M5JKh{3~aWV=_btH|1Do|M-+2st4huBVHUe)DSsy#S{AQhAeIe@|+(u0w zwtxAg&O8qeR@QI!Xpa{!K0jYR`J{IK2CQ9t*=NOtWLRx-L$oTMLr|eScaB}IXxkQ_ z8XJ3kT*{a?VArsZMVe2vb$d=tl!rEL$vXDCt#cWcY?&_<6|(hxy$kAjYx z;&=GvW)_yR{b9oT83DTsP+|cxH8eCtSI}yrrfxX%z8Sl&JypyyipjeMs$G)E61)54 zas_3i#f%vamHxh8?Bz+;Z)VQcy-?!WJ*CQ14#=w9aF5%)yVdTn*Yqe4&D4i4jxJBx zkdsvuMnH1H0`%UXmsaZnM&LO!ev$avCA0hMxwI02AT8xJiupOxf`h-5f>wTdZM&Wx zjgSX_ihg>Z?9WNop7a#()6_KcxgXA~2K!v>OX?xy4k*0SV^Z3vNWfXI5Kw2m()rs- z{5HkLBZribOK^BWj)SooEm*iSC<1t@cY910XELaYyyw-QXm<3A5Le4`tfZ>3$ffa~Yh>ZY`faQu# z`~e*}D?cGDO-$UnZ+m3-Ekg%5TL2~Y+-B}qoMI!p_173AT^m??pgN$mTDX(Ut4CPA zUHKB!Tm49%+&FyB$3I1X9mye6=ORuKxq3R{5izla-RxS&@L2w8WH$E~70{ams}5PAkiG?lz09 zbFRL}+r?@NjwAFmC6=|)@LCt=cHOHC|2esTZLBm)z9CE^FxW~fu4kXk|0B-R&V-3s zN3$f2bpNg04iY{KE)dhZDjJXZw5hL2xHwnk?>J8JkTB-noaE)ykij?Yngbi}Zp>x7 z!?C^&L&O(mzN9}Lnrjr=Bm7}9;_3}) z=@}Stx5D%1Q1cfajiisCuhcbcEej3 z)e<$Uw{C{h_pUfdIyy4=1%;NFCQjtP{c-0X`eu;be0S8Jh-#*g2$`!{n99<^62Z2o z4v?*$Gf%9sUi-Wt5b zXHn=MT}isMKQo!wD;p2PaQWr0Ea}>{s@x*TxgvVGdoBM}8`rrV)q&9i`S5VP`Bp{3 z(Y@tin%lIR7pq4P^faj}Y!9Ya<;!ezQ7z|`@_B`!-O3i{T~2ictld|0zDkv7so&yW z_!rm|&Em;F!6os`pS7^*^Iq;J{6D{Iuk6c}H$c+8jg56a-G?hOm!&oyKawx05fHq2 zx~K@hKF7o{=-f%{_`N1yofD?>JCjEqnkohne#$Aixb5j*G{bz!%lJ8f70Ri!6`@qRD;4pB`Rr ziFtJGOOzAu?(Z5tK||EUvzMIX;xC@BUp<)`7;DyI(JV173z1bImq>Wfdm!iQOJVa? zc37X^_?4t!chG{3-mjgW<$OQ)_8B6E=6Q$u$mHc8f<{HwCdaVRej__E=RKuASv{u~ zP03bT3r_ncaI zXt`VZr>&jBNzXSHKOt6^A`rYpf^^DC{fADVEoIfz#+J86{s}|JmM`}A2Ofz)RPG`5 zY))iwQ*~nb;)LN~;f&5L2CbbFSgBZ@O!Q5-SD|(dFP}ilRD-(-J0C-b817LR8CW3& zVc5!kc(TWyL>XN#7`YNhA3esawBYv0FMst@QR33#Cw4!TqE76W_u+Z>aYf@MybMwj zavt9$ES?VMdQTC2!H={rD?YbfeyGHvML{g38n=fPk_t%x-NZl#YmjO)v4t zN=Fc8vg6lT2+sImVq2W3);n;zS^S{MAM;IuPO9T3!yK)RL+x0q0vPYa{=z`ThPh+}BuGkaxg1Fn%IrV7pH@33DRsnO(nmV`p0(2ZU)*umFP{C&+e>Y06gKA9R8h8u>*iE#x9hnSk(+NTU1jUSMnr*0dz*@daAP5hNy zxYH@Y`<^}7W1-BYVLJ6X<9md7EK1ZI1NxU~VFWrX&cgLlk&q|b2EQmz`|H}uj(Juc z^ce%A0UslFGdM}v?vX}OrziW#b+JHF39o0_@a!E$2Y5_Q!Dzx&>vz(L%$cVIx$E<58xOt%%#6 z(kBTyIghn4M?n&U`X_$o24&w(L7rFn&69r{FXk!k?+7?(W@0pGjIz^(n_sBvQJ+X&%*2$=WSkE!uSgzw(ijeK+xsFc;=fp{j#u{HZE=P7} z+kq|~J&|(hSUHKnI>}j#{l?Ndgq|o|UVuN!v;L3k$=(B44v!3gEkcH;mNzKe$k!m1 zQiSEZm}xQq=C3y6hkJNJc;$TS6qch@!L(-Jx#x(DJ?awzVE=`ZMLwqyFIRykki0+Vpa-gOcc;Bt0}yRJkFfI(T*i9>D5_$Kpdhka&}oB8Kl0C%7X(YHn)5 zb>BfHtC zfqBfws6)sg$ZvPNb~}8hX@E9`?xDEDM*H--&%5)D!=Tp?+p!QKe7r}*S3$~TiC)Q_ zruEj5dvi<#UXpx{3{prjOcfDq3w|=1s4nPgeY0VMp8E6eI$6hCqHY? zJ~L=AJ{lhxk&zgcMq1|nVcC#xeLq%z?m2fOpM z+DoL#_!W%R>$*MJtoXkqXunk(QA-k?mesj^69Vys0tMyD!o{&22vMU<*eJ)s7W^d) zs{=VgRR=LJ+~|_l+~Uulg?xrzk$gdVIqwdPszsAAZT*6{WJZZVf7Ed9C04xk&G>y$ z___D;GdSqqPW=35P7MUIzW*VeG0>W$%YwORn%Aqq8gJRHT7 z58`%a%i_wK5B`=1=sEurb`~H8V$NRuY3pwA4~LR|1S>izUsbdvK90pApq?(a?9sh1 zw!o8SzaD$^h;2yJFK@_5!^8wMFvLjSxg5Y#;;ZEBq-igfjmE1zG?@wRyaY#EMDSjD zew(c=zVYPQ3g{x}%_6XGys}cY@#(3*%Hy@_A_)+7V3ciR+C(Wwf-(Jif>s(cE*u;w zrYyTZFn6V>Vm?*TX$ODAQ>nj3vYm~dGDfIUzkk(v%MUaopS0zyi5W6=b>-MBq&z&e z659QNf7@SW=}NMX1YQn-x9_X3eK=9SE82FTR`o39Hs08y>qNj|UMZiU7)0#5hMOyF znsj#_TH~378QRh45CNNjMKbZl$WgvE?x|*40h1cikJDz3Dd~nk5))#)Sa+u~#0;L^ zyGcOvYC7~={rw>S|HPF9;i5{X?RX#K8OM8z56z3~WiJNs`d(8{q{=g01Gl}TF%D@r z4lJi4ZeF@<3?VDvRwj>$w-i4-Q~qyw<&zF3UCqPCn`E*eWx09@bz>5;nBjc zxtZ<$S>R?;O8+mKz0)Y_A8(Ghp?c%g=;q;3!(55Ce2!Xynn`R=)IYNO<#`})hJUy# z5xpHl2iCaxth=#|TPMt(gaMf8t?#SrZysIBW((4P7{rPsyP=u)`&#ab!ZUPmHM1IV&U{a6NR zo9mwHLUg1=pgXc~Lur`qjp?3h)9NPkH4rRSZBMey28@cmg{}Ln89FM4*Io!~u7kqV8qJ+CO(1ythoQD`z%jj5cDz`CNQ!SUe2vr(v5?7$KEMNVf!-{aO7p~ zjyZUqFueas0(ki)pJrLNh)^H|=DMT`t&;hw-d|I7K?H+ywAv*pfbD3z5 zT2aZM;#D6vbvvqRU27qCXYf5BoYD@y-)+O}PHaX;2FrVB$RxH8NRpaA#;+aiC>xe( zH~mQmH+sygX{mwzF{Ww7#q}(qoAn4 zw%`!$nx&mL@^#0|llc8xe*V*cv-17wTIdIyrvUw`DkGQEEShFFzNhX_$|UV@hoVn~ z`sw|vAm)4M<3oOG0G@o@uZI!xsO!y+(x?36bG!yFyn3S|eSUj3<9kzL`G;Vk$vDLw z%@ty{m_n$}d~8gdYEy+K7VYwPiB^&N@N>nHgZy$IGLYgcv-aQT8q=rt6U7nzrO(!TtorW$kQw^g`4FHa1Fzu6Oe7vqVL)E?Zp3HEJxS!`(yQjdlDi+id&Oa$7gnVH78D@q@*J67tco$ zI@aO3<_b#$R(C+ThJG&BBUbK=yKzy<<>c?5A+YZJ=^;@J{D1g@RN|+nIJ?8FAh;KY zLP(dZ4Ts-bg~2~w=XJQ7bspj(tejaEC53=_4?%kWC)R9t=USf!^~BBtDB0!QKO+4( z5FyK$R4P*UbzrVTh46!wMPVX2Rk~W0&G~izT4nt$fe)J%HqA+_-ePA~9oorG@zd+{ zWbr9rm&aJm*VsfJf~%{Kh}S9u{sjy0;21SgYEME2GC55L-CR-$zU@5Hw)SZ)!oB`^ zB0zrY72i>lQ5*ct^$;j43*&{aWPnedDq4W+<)BTQ8bimraec_vUrxXLPo|A$?RVv&=cVEA|P|1#xvqZ@-ph(U+NzS0;AVC>GkRaJ1=RD*b6+v>&86@YN!`vSI_P6)` zoqf)|>z=jlA9t1ub%uH0?s~hcs;i!QD!T3wIE%@w_Sv6VbV=_!E4=?e%<$#N3@_Y` zeGx5LD?@lwVEMhJJK-f0Y7Ztv|JSp)PUK7KXX4&2?$P=lsX$@9{Rq_IM|3+lDncOF zH;^>@4*xI3DffIJ=}7afBxyHcM#jbXL;LqeyD%`B_7Z|qcQWQ#={n0R~1m!e9RsH!k zog6*|HmETIy9Fpdr6zW@HxHlU{rv_?6Rwlm8$#mp!)U<6le5U|4j`gOGaH$K5WhR~#^&wgt{)UPHckGP>F@%3GP2>*TKFX+Nl zH|g|$D>@*Ly$)C@=?B|5^i{H(?YB`OH(-3!6yPk&*f44hj-(-sFX+0A$|}Qi9Zp>F zwT^q^-V6YBF-bA&$5m!GLf){~H~zylH*jON?5)&@R>cyho>~3?S0~|kiVO!tRNwrk z@4V`fR;9b(QyIj=p?_Wu{r_iOYZak+~PEl_Fech zXIh#`e$4ZvxWoyDU?K#gLSM) z>n35i^v9=>8&Nn~wW;FXWHRMCO!s2^!6NQsVnI0V3*OTWka!LDaT+G4?wOrE;1|61#s`iFGX>s~?jBe)i{JjCGARly1cuDZa2rvY zBQzrF-PunNg+^qK$>LofGA|N*QaR|+HJh$$YJCT1;z-s^fF#{TF!kQHU|-I6|KSAM z<84#w3_riA6Vu_b5uOCoO_vm9@0yP2+Aw!;;J^7*K|bvOFATlIh$<}+NIHrD=h^(4 z^ra46s8U&pE5apySDQFipzbC+GtkK0)2+ztYW$T7Z8j8G^KWm-Bt+hQDXMBV(fS&on{wX>J^H%&_ zu-Vw3b8*J?6&Mv{l*L;z1Z&mvx*cAnYsw~8I(iOF&Yj{M6#uQ%6Yo7vz5nk~k6_%V z9M9NlFqyjQrFv#p1KP~saSn^h3v=$PdJrf^=CvQCcx(nK>}I{a!})Ex6AO`fn6Q`0 zN;%i_bp)5zofgkFc{sEmbtYs~cuAy>yUd>Wvv~l-1;|dmy1SuV5XDPPDb$g12^Z46 zs@@G?O-{;1NtWtT<6@+0q~hh2;t09*Zrv6+i7SLA4d^cyL}a#-)nOe3#=;R6vbKWY zs`wL7d7#L4PrI7!_}H6RR;v#0VTsw+Tg7K0UDvUcRgUhY)A2^fJg&dko;IH`+g$&Q z=SwwJ4S{8G94ofed!GycLOrp^no^L{nMSws)Xs?KLIuS0}7d}*M-JBo)t;_}W>BSmdzI-xaB*$)^qcWuu;!WE)x`% zd}!Ntwj#0XU6wvI3}9c(0K9i_4~>zyeIe4YF|OsK6o#Bh6!f@po}JlTDLc&tQgjLBmpmli=+Ci7bd;!gG#7 zr^bGH;-z`I0!FASatq-@PNY&)F}!;C0NSL>NA9rkrM2A}POa16>hRN0*}Q$gnIg>F zn(FpcpnzlVdCQDZhH+0sWb6LZsjqC*1wAg3@sX6U*En~oE*D#lrq1zA1o4>8XSj=X zOTG_^;e34PnIa*c&!eveInLa9z;xSGM{V%%efP$tl(@h%dP#U(g!88DudrvfXptVaif%O&ugH&HmVYIiuwSsKBLWOz zZ!rWrDopKb&$a79@uz#z=kJz*og#nrGt-#HF`=evei9(A#F#67a>VaxEdF~m(A5sO zIJ>jWyrG!1w5xF5?kPU|)nMEdAidwwKpZ~@u!V061itV%^w6>$C--B*g8X#a?dxP* zObI48-xTlnxGfI&qL+tmZaQgLB2;!-Mm;# zU@XnWYH{dw$*mg4_U{V!swS9lj28={Kt3&7{Z1O=-T6LF$9@tk4w;|J5TeZd^5M*# zXhVhU8jHh`+sD~!3qlS!y~W?@J7EWS*9YOuaL>0Zz06!!BPufg8WeTzA)Iu)+@%Lr z9LryZC5)IoOHZ&;+8biEwp=2-IY32qY#L*?HJQA5^}>F4zahk@g))|?4V;&1&Yc6- z`_FbTJ7ixs`-tjBB+)wE36AVq7Lg| zxKY@(9P?wN$7?q76(HHCAd>j$Mh%C!vb8ep;#K?}3C2Kwe@No4#Y}OV)176y$Q+RH z_9$Lhwh@dq(6fG(4n%$pjw^I4n=RJ)#f&B!eL+?Jz2SX7r&g20?7A4~%^@kLcClwX z9_&whMteu6vWsdI1&H&shqgv4&Lf+{*d5x)0!57Vb6k>D=nkTu+ab5dM>vk|sjYRBGt zGl|P7t1WoYEQPQ{i^6m0I6hmCkhMC>=ZT~~)eM!wvX-S$f?53s`BZiy9j+q7Jf_OBZ8Vpr= z^lYs~WgE2SU8yV_Gb7_F5&ptP0y9eub5e!4`rp50{Lma`xCj8ix^TsTg}se!Q$1^A zWAg_9lQ?4h`9k>R!-qPJsnfbHB*)W&k9=^0awp!iQ8ZZK1kVJYymv|x{iHmvQ90Ug zq(K*Paw?~H>mQN2_Hz7$PBP7qHBwsX+(85S9>1GeY`a*7bj7^if8pT`^(5TL&NhpA zgy}E-lna{bU*kG3Ah1w)vfW{4uUJlLsf8|cDJiE)WW>qB@^-PwS!r)x zZeeRZ&i-6Jf-t{~m7E`A;v{qH-eX>6j%<}@o4Y^2l0FyoR5fUfU7bq~87W}uKE|G@ z>nfKP1M2JP(dM8%jzmyk9?=;FBrx$+WvjPaUTlR-ohS7|V#9$-w5IX4oyLlRvV^Tb zw6}(fkuGDLeaT8sQZ%2tPP!;nI6T%QJzYSf;nn>|nw{R(8b5|geJ%QK%#ABnG)%he z!bK>fl}k9bl^WKX?l~|9E*4yr9_8p-jC{bu@=*xK3=dNTn5GT-=oDhdc#<5F20|XI zE^&w`SC9Ejh?!z1wl5#)x3=EbTMJ*76}t&?@&vD5FvC_Rlgx9^@~cb21_(4Sw@7OU zi7qH05XOO~-fzQ`ty9xE(u^(BLFQ_YJ6_@hg$*s}&cDA>Y>DzdL5m*E~p}6K=)8$jfPN*F_8#PP=T+mZBIUS7nVdMB|4DvAew)0aP-YzC%o z9}|+_>~@hfH1e_3A})T1z-_E|OFSr)gCwMu$V5x;539~3Bn=b(0PeT_jXRPM(Man$ z@&et|?)Cry2li^s&sxVVgUwUlCbi&kaU}28?MEI$F`{X0Mjju(>c(L$Ei1`dhTG3; zTu^$3Mn?90<344vLzu0tCAcE3Ul}%?ewg~Ln1V>-%bpG6t8bp(k=zUGv-hEU!8bSZ zY{c*KEeoyoZoBOI7O4g?VY@0~ChSWD331|1Mt(1 zChw@uxpSmm(_FOu#z4%(UyF%@le0j5!l9iv$U{4GQ<=@~I19OWyhP@Z_VZ^3ll9>v zcaxF&&(n^Unq~W1#f>R4{6NQ#W}9y!-7`opUmBZhz(`BX2TQ+m3Brf+C__7>b84Jx zk?+d-@UR*L(U+DS1avHywvy}W*~w*Y_+mj=4N`tFpAsM*Q?kOB`x7^L?4K8wva)a$ z^p{01hY>t5!xZK2IGcVjmHRsJbGct(wq|7608ei5;MMW7nTtz9b6*9FT}Q4Su9~RM zhLuyTHbPLXWpJfmw;1it*>bv?ND2M%Y{YupWxtyrz_r0K9wFnx$oZ>9B%1uyL=17_NmheY%q3T(U!S;lgW#9A zd|asWNTFM1Y^D>orvDdMbEX!K|CGz__gHPBlS&mG`!JcvM@Bllt{A%J9lqgJO7WfC zB2TOoL6jRyX5vk^!&baa!`(}#I^2}AO4@wIY1#!&2lDg$HZx}7sHX!qO%|(q$uZ00 z(dD*GMQ#{I3gUK7=LypsR+?!>+B34?1ZN{NY8l8Fh;mLE64$S+rdUq-t`zt4;j&+&7S}jYd}<$v-6J zw&%2<;@D+ni^?=q=b7>+H8y$Xo?yNBWprUCc#4mhge>$6UATlco#w$ozCebR)u!Jo zinL3Qy`^kT*~VWAfs5!mLsl~@S_MUtix&;?8W{e#KcdKM&uH9s(!EE>M%hXFQd|(Ji=D&g{&o zHy`Dh*(eW8=Ol*Empw2Yn{C!xL0PF*F=I=q-jt9yzmTFVW!3md0{34#;hRV-2u#U* z1|ds3;{w*E)mjagKmw`5uea@&yvF$)l0#dO2T825=foV`&sZoNFABuM9hk`L$R4=y zQyeuR3M27&S29934)Ro}(~Wdoaotyi?M(q*Y#RGe_Lk8wuZ3)UB_S%#IOz^d6E|0w zBn`7^iA&WPjd5eU0WIP9WluFBkrNyH*0-S+Pnk_aV~U3RM8HQ&^*58f&8v_F1Wk}on{awk4*!VRugd&}O_#C}!W6tQw`F8)Py^rnqHg)$+pF5E0 z6)jI=oPzg9nLDvvEw$3i>}ETMoMjnKhf=S*oNX=%rCBsx4u00~*DYJ~0cJyl1$XD> z#OT;K!;1LDjL9~BC9%^$=UObawQG&0PITbIjFtLv2l;UHmQuu}$_V-#rkTy9OU^fD z0-|XCJt+c7yv?SSv;%t|iT&!})Uu`K>t z@9&?T23l*=BkSlcjQ4GsS$C}sQ*axbdfzy1zw$*_pjZ4+QR5<`Z3o48N5PWji8nI4 za}xs7qPldnx445R0)aVU^KKJ4-rDk&U=&wfB$T+LXdj{MnJ8LCG7*YsWcM8AiuEF~rLr2W$~^9*>`3ryTqh{=5k*v>tA+R?pS9ZZa75BP$U| zW08D;Yfji<)XljRzs4=Qr8d&ov~v`G^8TTWj4BqScyR4q14);Kcymb`1*0M@jD;(V z>mB2p%8$$AmOA=jC5fWyWBpCNPk3v&)lCG8ndxZxd1&3PxzQ_16TQ!$YHdFkhxdt_ z-Fm1!IWSei!PQ;mcG@d&U$lf2oA-`^L6sDq`ZkQ(l*f5u*s=Ds>(dQY@}7%zc0q#P zJm6m^je~4br;l+&U)+FIXHeOPwWf=!s^A4%Hg=it=naNIRp~9YfoTb<$kJ2$SkAyz z^Eo}wIT=HRYJISWJv~zmNU7Ir9apP3V;@aqo|8v@jmqxqhvgS$j%bdnKb}xGWX28s z5)%;xXTy7F%tcYv+}Ma8e>h;IA2wjOr6X|FQRZ!Ir*eK@JjBc=z@eS&iW^zF-2a11g}mV{ zmjVXF?d;(0jqJfjwS=hb3lHQ_t-hvH&syR=bGpz^Cxd~z4P-S0QOgFq2^J-{wmda5 zY}*nf{mtMZDd9kKX22(^Jd{V;qsZ>5Krwvi)~6imot{4EC&;k~F)%?0DxORgvUUZB zg5^;r(u$w4k~0ybTEg)NwPYoabz~$uYVkq~mzogQ8=^xjjxe%5K#HmdW>&Y1j42*L zCqpFSYR^uZZ_BS@DOW=Y9>jKT-(Htq*+jxJkxpxWAlQ$|R`C{F^S2xqUbYPuFQ+8EJ7Nf}7S+a{JwO)9x zfN?Aro?Ay%!csG5ZDXXzAS_w>gHuf5E7z7f(^^|%thKQ_exp|ti93BDIE+@R-htd zIWtre6PcatQ&j7eO4J}_nurXicvZgV+#vf5K-)#BadcfPb7gw2s&)odxG4!Ghu!LRKT8ptU`4eG85^08<_FI+<4rvB z;9>||)D1FK?4k{h;#x+2E? z$gdfzmV@TMRE&Byrne9GtTz;Mc7u6*;iv-%aKNF zHWa$AB*kBsdwh^D!wnuly9(6;NIiiC7xoh5`eQg~B;j8)fB!DX-mqK9B^yBg(hWC- zO3)3VtPr`;L`+mKh~K%HcF!*HGE9&_03i3{$isLZ}UGZs5 zJa5l&7g73%k)*P$KtSGF6y5_{tarbw6c-cD{#p;9dc8QY&2W3-F{{#X2jkL&<zECYR5ucV+TOs0%Z~4Pahmbt?xE?j2`kQ~SEC zbX1OW74)En2BC4ZT1yz-gRXVzK3^W7q6LIdYKAiM+#mK*eL{}&vJHschQcN8VDP%# zi)ZI}?&`i{0qB81Bpno&zgrkQtvnG08MEFDVV4eA6I9ObdavE4@!Lxtvhm3k<%%c` z#JySqDUjD2XQ#1&%k?T_n9HE`1bhn4wlqMvTWGf_P=&t~ZFvClWt`X`9mhc}mjGG@ z_&{l+1=-p&k|&?Av4sOf0GN;rE6ApOjKLGi2bAYp)=6?S4%EhU5x%~@VAM&{-nImH z+F(gbH@|=&ft8YF+>w>m`xJ@v;))Uo$N*Xq0j!C+Oj6u_-uk;smA8O(b7^xZR``K` znn!%h?BIxFXw7IT@?T z%MNv2r8uc9Co=PCTTU=I8qhf4zhW`^G>eN7T^CL>C;N%<;nUi0%=ITh0piXzeo%XH zALCl>20FGCQACbZ7WYeTYGIZ zcu0~kkRKdZDFM};M9BO1-udSxZK)_8{(Z%ykH66!e?IL^N=AKC==aFL-T%j%fN)N) zmlOblL{fY0ee-&Xx{-6fRMxTPi@qG;stfL{;^KR{^);&tW#J=ahS@{IOH|hcYvFc9 z?pMMCsvG^a)0b`QhdHWk7T&S$1xWXccJEIorBW{=C6-9}v@;K_jdk0!*qlH#lN`aH zSJB>jPv=gHJ>y$zb!5r)KWg9u*ybHG+rJEFy{$~FYp1jIK#t$@t!aJ-Ux38d^YibG zmYWrqSGrfg@7mozsF&u8YhjuwK59+J8`&a1)35!o(XbvlNn&}AnJ97O)n1||)tW4j zVLEnnAPSD~vJH0YN};_PWHB3RhYC`oW2NZY9Ip8&91i!?fTTabMbo{ts9~Ob0DUp8 zW`Dd}SjO&-a3O^>{b)S_AXtK&tD7j)3`kN_G2qmX+CSY97Tq6LzFkIpq8aFm|tI zIfrtU=sM}*Yp0g5EgDP*cDbUqd!@`awo0Afb8TK<`svGdR<E_p`v)=N4Aj#T#08F z+Xfl0d?y5n98Oy*P$-LCYqc5Kk!H$Fvxh4qDO-Ua2$0+D3dBAy&zdNBsSSaF*tSp7 zm+Su=yqDC8!s&m6f7mi*D;XhpWtbDR-xglYZ&!{-12of}^J z!w;b(b++5%-k+KV1BY&3hv5QHv~`e$c@+F;z8kxxKp-$swo7h<&&jT(Di-<)tgj#o z9)cIq>=F}kPzd=TA z-HH)_7IG?}nZy{}9oR~6_XVYLWK95UrkEeM5pmdd$J}$Iecvb2>nKQob@?PwDQTL@T0*PJ<1X+dghBOI1on6MDj+ z2WU#>eGe_hTL)U@ww$=%cKKR?lno%s!w@57Hv%^@go2@Z(aeAzIu;vasfxs!7W9A# z0MN@EZV#<~-Sz>kCW3iP+kuEA#Hl(@6&Ls9fn1#7cj>xS9S}=-D!Hlpzw#IuCEA#L ze8g&HmnHhN zsHBXEzh+@UE`1}?0GB|5keB@A2PZXKq^22j{v?^$eZE4xhT(oMx(IW)IpLy?OTD4} zY0e=iYowr}q|Qtu+pjBh-|oAIh*6>AnDR!GCPW*bp%MuJ()va(U+#wS17%f_+g1C3 zWbq-C;oLH~M1%k$IzvWUvBkZ&u8X9@$2xB>H}D0q>~d~D5_TiBh;g-67Pj9ae1Gqr z@&+eDD{N#EP_9h7&pQM^|1FJHSQ6GPFRfiU0q`un`Xe#-kuSaJ@KGiVh?RgI@4=mH z=(Rr!O-`5%00qAM{poL^qx76KVbX;O)S5O8>lRAy#;^+g2Kt}`EPR*d`tK_zV%~XA z(wq_=im!CPG+{#g7_t2;b(0n!jQ|rMJk4;0##&*g7Z>8*lqjGl)^EcM-i5cJfId&Y z;lr(ulc=yC6rxNt04YVlrD$xm4n93i`yL0pCNNwH8-QTCPQZ<~bkP=$`5BJxP;(nK z@Fb)a4@=0%@wZee80W+XMqL9g`%>(e3;0FCqys+p@ z(#-fP@QMK16Se4f4+~7Ctrp`$lN2bDr0-e>MHf8+zBo{eOQIg**MnkWh23ky3kSO$%m;8Zu7M zV=&AIAwG)&03CBJ2?Hf*)F=T+YdiyhY()xn8G2C@WP4gU7mD4# za~qo6nxjHr((=OSG71cgMjaF)fxii803zttVj{Y91KjfMq2XDjY*@o2f|H$dy0po3 zVy`xei?4Rtd2_k={KIPMRdns!8WvVIzN>=O;%sO-K9mCS;&{eWFit^JO;Ec_ZQnDM zKUYenJJMHB%6)re@YpllcZx%OH(?U_WQ2u&E!$KrpSlSDSXjE!$&MRQ} zDEOOqz9m92(w)v3#6}pFV(%Gh9N)T+)`VD~T&*QJrKnS{08J7v6`y4x7jR z$Q+7}T9XkK?604p3z>DpP;v#>fUGWt5^A_{!xx|BX_2pE4iR74Dc;>jSCy3rn_u(* z$#97RM|GL49!u`MPzD+yjO6lCW%l(@Qe?uCkhsFez@v9#9b$ZGPf`01EI+Sb#*NFB z=`7DS&3-`KUhtG{tXA5uW>@7WA6SMzUm=A=#$2$ryqCc-dtKPx#pZs%SJ%8SzV-c{ z2RVnQsvS2MLl!N!SQk?rpWS=%HtlrVH zj9HHga?E2nJJu`s4M!FBl9a#a5FL$KLSk+>GZh{4m7kXK+^79JM|4n5gs&eL z1|%W@q0F(CEzQ7M6NnFeZSVv80>>;mOqVaG%`ZU3eK6x;kL7ovgfRoH_K(uyf^ia8 zt_>-dvm4kogQ-t%c|B;dL+x@jR=sN>G)l2*H0$gfl`g_(3kk*nQf!r3qA8Yx!Weqa zZxDUK1kg_rr|TtglRg@ZgX%r+eKioL4HS}xx=;A(q%k0A3NM%M-zrsQA_J3~ldc4T`q& z-JHb-zx5y*mIukppGkz~&+x`lHPJtqP|#xT%>$4WoYc>VyL$2A>_Wg#Z5S^4$a^Ck z8;Sm~yCD~hFgap4P2GeEmvAF_>z39-C=Ib=XR$;7gxB1zCw2&uUi?!O#m~o^S6#E? zW@YKpb*;SMVPfF}3YVRU$StNJ6~=RWK+G3o*8 zz){bGgC*_Te@<&aa0}`jX1N6n)W?IzsK-EKa(%7;LpoZTBXMSCc9QJn z#D~K{vsVh31hih)H@WTm2b)$7zZVb^6ILnN5e*l#06vrXUwC#*Re<7;vx~UGe7qNw zeCPXVHYCJ5lDq(96!IVh033TT$_rKD#o7;YZl|VE;cnxD1$NrcOU}P`ZW$z4>9?}E zNq25=JTK^Jf4pb?^NZ7dOfF|x>*Ev0pWiBwxvwdKZu7DIbCg#wa~z3QozgHbS+0GtM+?J@hN~4cZ>xG6B4Y)Bsm{Yl3BaL?m_4t#NAd`2xKK{<0avI< zgN0om?F%OGGj*=}ivsRPK)x~M564PRPS)%8Id|M8p)bz?Qyed z^eOizu(#PuTEBxXUJvi0EktA54cjs|%I~oCE8@9`GYdaB_@ULdOP!eC=C~y{ ztXy6O2gCtjKPG?)GuLa6^|!c9?p#$dttuBdYC(V9PGoem*m^W#Go#MprU43o2oRN^ z{tG-qAw9M^$;$LEYxqLDL-vTP)7v(SJh_UoOMtmzq?VI4B7|rVqz0!6s`9{z#Izpm z+kSK7dpe;UMU)2)*3T0>g?N;)MtK=9bLh;0iC94s}HRLaMe@^I~O z9(Z6TrS|ii)1F52&W>v;qQeK*Oz~mvWSw+$4+}8M-w#V@$!+k9H}?5D>R$ov>EdB( zpS4^4l3f=qsTCMOv#*3;g3giHHUVhqU;37|DOYK8)*K?SHFCT1@JT_A8KsikZ3PRI z?_R%m2QoN()b2|<_;5HNL)!_kg8Uu zaRRO$R~5AV@2`2i*VpSFx7Dek15L_!Lw`$N$eOhuG*BDkA6?7jx)yLSSgWGoTHf~f zAoz=fvYH6o}QZVxh4H-Teb`ynWFBr85WQ+WyHk{ta9j7-P0O7^)M1@ zaY>aS%dca47{JvEfMb>^r&~{hax>w4c@!)U4?!ds56&Z^I1q_=20x;hbxz7`|FLj| zV%;o~y~+=A>Fj*9!Wh8GK;+2FTIq+58KvPuiHNOdJ!zst3JVf0Z+yZYsr@olW%;}= zy*ZXK(D5<{u3v5%xLPp)e5{>X@bK8PxL!JmgSss-a2L`tf1hF*Meo>0z?C)1w)@e=xZ9A3tdIRer{8oQ)~in3ERQ z2%8IS@WBQ4Io-((0&8op+EbiJlZS)_vRBPq1Aic}VquHOSd$OD;5gv5oKok&QUE5! zh1A3bO#S#^bq5v4O)Adl8kO#=&d)r0jlZfMnQL@6K61?Ctw3&C$SF-nvhT$&`2izE zABkkn3Kd{B``q-Sg1m8`rXpka!YA&^T1JWbVTwFcE*lSsa>r*p2gifo)!`BbR8p~L z4mx&hH6d*q0!w3d>Iq{O`o;zRRW=BT!@%!stXR8MaQ+sPBN9|!AT*(upd>|T3rPQW zb7fSm2Kd*X6OP;$P{qQ*yhkpk#H(7vhhjJa_er{FrN`Ur!~{An3N2m1;p*2A=x!47 zm1lW?QgFFU&EE2&(rF2$rukT7MDC%Kv1xU@2X{&07Jl{ZYqLXi6RCk`|6Z23SPhyK z>g=BX1iYevQT1&t2myOVM`4!XJnqaDL>;6Gw8pj|mj8?}^z${e0&4GhyzsNbmDca* z;JND9QZejlft4fQohb5{h^*>eOa19~{e9x-Z{J_(rJ#-v5`1{(HCX;1fg}6gMM+&< z5OG>#7rRt8X&!f{vi;9h^cZe}GoWscmD@dP%FMglGvPl18`FeS{>WvqDg09w!Y!mT z7S%4faB#~|?SMf8txsZsU#u`+)0B;|&4hku86wsnGU>v1otj~IaJ(eaAE+8qF1 zN{1}{7WsnNN(Ddxf!6y);U>za00J6CAeOr!feFt&i;gxy(WDPfh z{Yg3|Nt{>!HRzn8eB<}BND{_@pnfwpWdbxDL9z9J*n4I|)ai-hf&%BHb{4DSH&@xR zynhz^Zvq7H30ODaC(0(G{sLDD%+m^go1ydo&KQ?_%DM%efHms@BizMK;Qv&OOQ%Hy+B+Ayil17@cO_1 zC_M^A&jLimR|Oa($32};O(@Lu`^#PC@857GeB5fQy?1c>x#k`jAY8XHx72DVKmYWh zZqt%ZIji^&Kbp`fiqr8w9|+EL>+1oD*B_LmPj^!-hXG<{i-{C*&=4Xp@Gkv?M#FRC*HV@jt9PRR zYy~;0zx?F&)q!V#+AHie#o_v3XH}Z0jT|W9U3hHf` zAU)@;u(DQclVCeuSw+PL?_r1FLrblPR|dI}ccQPev9f}|rpi^>Skq`$sL2$TvesQX z)J{vf$g;Fx1FaygE=-owoF09Rd0b&N$M#;RM=P4-c~T}kyX~iyTXcbjjlSiAM(P^K z-DQPMG^hVroixPe)hfsf?+V-ap={}W(6Piqr^B%LS{)D0%23tQXabO+05|#fMI7N~ zbWR3MS0H$Mc=lYkfqukhBzv{`%@O-l6><62SP}9oE|e;KY)r{~s#G4S6CZSUkpx9N zhDNjhp%*EUR4}t>y{QS9mm}#UZMIrJ)QR2O07qY%h>GoLMLk`qI7l^f3xrJPuEeoy^!pnyNl_tl^ z0OCgK@24i@f&u&2aIvwBOT58d)D^436n?9t|HoSC|D7)T-+b%;G%~1R|NlY1|8Ha> z)JF(9xB^1sYpMn0`-e5QmyYo<`JflTKs@n7va;jWHbBG+x&x8WO*m|iRs-(+2mY{B zK;wZHVesF?vAUZDe$&#Dtw>)ST&{vPE|@SQZ)Bv9ZbQe^c!d&Z{j3}{J(=Ecq2AVi zQOEW3#puK>t*WA5Awj$T{@$)mi;qwWB0?3=Dc)(&9xIX_aABkJ6Z`r8av%mQXTS`y zthHCdZL?Z`X@0uKi$msCsazn}to-kJrhcvZ*Aq)Yf^po<>D47gq4Mj-LF1gq-1fg1 z8c_B#iDbfVB`w>(Z{1s$jg!?jdkORd2zi#--O^0G)S&W*&Sau&C$!D*W}|J2XOB#^ zFyP8hh)jKSAADWN<*xwH_>Z?2Ql6av+kgi2JD9GiK_NWkZPfwJ9RwxHWk zMsZNt5yF8|5-#=dK4~mcgSz`<#2%qD)b>G0?Vgd5(PgEp2%xBL`lfRM4KM?zNBrxz zB;np4)4$QG$t~equBNHv5@4lkU3JRQ`r3MfxX=6H>OZt8Sfl)E^4I5V*2}f(2-z*h z)mrErm9UiP@o8yX0wVNpzm#ZGOk0c;eHz+JO>s5qTrpsl%OG5-E;q^19B{szY@mw*pI^`I0_+pOV|Ars+$ zv2X>8fDRv!DTB5YQh`C3u*EJADLf|QJ_L&ng&Nrr8fEF4t?~MLw^(;=ka=Ab~1l`hM$|4``7Qe0e{+0-;-%_!^)9H&Ul#*U3 z0H~Cfo|8+!zgg(P|1?FPb8~d2%E~TQLM5gOpjVI%n++&8c;kyZ|0b9pC}mk6vZcn2 z;{P`;7pR+k#Wrt_O0K8I{xKkt9A?^I-eV373OtDicR^cJFw%?#2Z-HeL0;;*ns0+m zl~#8W=We(zmA%zh*h~MW>3wDodNB}`R8p9i#e}1yzC`x~Mdbos|C-WKj%o=gtzJoekjmg*oE#-vl5Z#(Y*kP%wsHj^xmm%%-9&@a`8>aB$|Gs+FF z(^@+_AY@|)}MOZ+CKbEAo&;d|sjyW`YZ z^^i?}mQkj-O*h~6Be}2qiTiVz?tXzw7`D@E@$l-B+L$tOHnTI(igTQx!g6LR{rE}6 znRIQele67YTyG)u;18OC<@f$^J1_GCvi)}33FU4&3rk5$Z)qO`7-d?9^-P%)=phNZ zaXs_B*IDJ2f9V&DdJiPR)VObb$a#?r#-lNJLi@NH9tJCxR z`0)y^v7lH|K+~!y7 z_~E{HG6LGSPHAgTCvu~@u_w*e4E~sfWzwO8tf5ephIS|8MTwDE_)lzY3nk|9HMR^{ zacSv^vSlyFcF-`f=8#mQ&M~z*NVDg1WgoQ2>N;56_FyqxI4Ou`v>7Ujw%|`pxb)Kn zy{p;iXZs^TH%%93PMhAe&4>zP&={8St;HwOjV5_CPb$p@XAGyb_=EJxWPfyKcW znyr+rluBK{s@{-4lkHtEc66{gW$+^d;WI|`VorAj6M6XP&epW&>`;WP8+}>{=&|3yRS}j5gq-8Z^-AM&-4pS;uH_;*h;{G1OfvQmD&TpY5wZk z{Ut+-b%`>*$%ZTa?+uzx0XjPG%RYCmRMU&&%N)`P2QopQVCS7#c?AVqEv@&< z!B+Ij#o)CuP_llzWSXgSSnj**$UwCL$PNzw@X6)`N@B0)^jF-DQZr5JVRrIMQSbFpG-vDPW(1LXNXmIW6BJL7VEM z)ggpY3DxF!sXgdt{ZA>lHO78@?6ftSZ`a}o{lqxoBn$LJ**e$Z>KG#vQ@vUEtMO(O zNwZ3>&8?|-w)V4GkVH1c0qciQrScN1m7zELzC**qkucnb%d3$ToBH!bKO3G>D^lV| z$8C>vk5{-hb`rWchlhKVaUNxir;+`X8Zv zg(O1q&|F65Ud+!T+{T~dAE`8oa<{^{^fY5GqO&qK3P09XQ^-Gw7S)Sy)nz546B{r4dY+Bv* zF8DsynDxB|PyUOmLkkaTq()te^`=J5s^C?zYpYj%UA5D54vy@#jnadTp2h^2%~a8{ z%lcfSQY^{@Q1dD8A}4D0D`WZvz2-BmUz$}I($W%225>^4hFuTRpFV@o(smdUc8o^1 zTDRh0r`}I)t7j{xFeEmx+NiOA#<+HMRmtYGHJPue&%l;=ynfZkyFTn30xXwY^0(72!%HEiH|Z@lQOtj-QoFA8#OXkiV>V?X|1c3R{xPUU~C81jK5#@ZDb-`Sb~P5r<;_^T7107 zP1o^gz2BEe% zAs9HJX#mfO@F-J5GC+=kF(4| zy{e`xj-7w4f02INsEn9E1wLy&_)BVhJezw@!T z!Djp8$3{Z=JJ4sZBAIO_5q5shpZyrqr73$JdLJGAY#?>XPc*QfG1I`ppzhsNdj)@5 z<{6(>^#Ikf-urUW1jXm|a^`pSQQn{zlFIi_AmkbkZtY5? zFmtJR0!BB!6Aqs*-_`Xa`lB;#yzCV{Jg$H#a=U#m-`I#=la?9T6?;okKc;+5 zEj8a;EM%HmCc#ycbQ&MODNkdd+TfVTfi!Q8Yi(_c+U%UkM?yn`w>^`byE98TL>>^-eE)I()XZ9P4Sd5>kI$LevyP_^8UDxpM1{2$n;k1@{ z@VQw)C*k80e0==77=(^%2VJN;wl#9WW<*B&dB9w z4vAO0#+@&0R{28iSuf^sz6wlGdnMUXVB+M1z;fK=|5E87=G9>1 zq2qBC%k>iPE*a0>c+kNuy9}@4~in3?2*n!YaZzP^QeezOB=4@z<>LZW?NQ(!a(3<+*cd756v5b)PH zad5Ks4>L_nrtg+ZugARL_2{IF9Ozr!oHqjN$KbeL9}id&>&4?clT#X>L?!B#Y(Nj` z<%AnW&p*eJ5Sg2qsgEG)Xk2~!-|MS@%D0`q(XTJbNGBcaOR=%Bub-Vr!&zKc1BS$c zDd{P?yFE6^AKR|kk7raWcLyJRd{Sz?s8IwxJ%Ndy8UmHkgKu)`>W204jiKRaw)gol zy~@C%5NbJk^)!Ons{UWaU1wC2&AP=83!;FEf`Bv;6cAAn5HKLp1Pq8slaBOWq=lw5 z6$J$Zqy_2H5=!Vz;G;&0v`_+3dO|Qr52W1ToO{>0cbz}?*E#=^ti11wo7}&ODG|ylc$9kk2(4wfmAZeNg;&n729J%>PLe1c@t85xQw8AQ z*0@z_F^xbpqgVdudwY9pSu{U*y8u!884XoZn36@^L|VA^P4Lf~o2T{QaO_9h!0;t% zT?jL-wFu3>D~_%ueJjCuTiZ4i=_)J?vvC|D8NEMK`poGPOJ z4mn}j)RQ;=>g77SakYq!%!--3KxikS+Jl*SR%%*vB-b^=4A%4AFJJJVW{ z{Dk!0OLBLo2Q&BW_T1)!`h^IkZltCrM9P`+DS^^x(&~lo)J_S3y$Bg(Hlu9IYMYQo32ANYa>J3j1U%*Y1f73JJ%Ha-P{PGrI^v7ucnRTsi;$x(jK$tw^lCft^Y{O zk}ahc8$!u{w^+ba=Snk!cC9R@FxKd{eAMsJ6ihMO`qTGNLQW_ej0DKm8N0nwc$clTV%|IzFL3(L?I z48bM{+#^}3GBK)xeugkpcnf^?b)VVg`0p8w<@yIQDk{f801GuJSt2@tLQpylAGX`AH|Un$Op#MLEV9!lu`+U90onhDnH{iw(rZ^Gc~n1}fA0m3p~6vR8{rN_JNW zg4$MAUy#VcAZiy>o{x!1=@w&1a~mTg!1);SM)>%*6dsmjm^cH&6C1B&I2=Jdl_0=0 zS6a%JM4k!STt#a|j3wzzn$JtF@Nqr)l_PV_Re{8ge<8mxnk10QvPbJKE>xwM#@D0nLlaGL%>Z$GMZ@~7t8XYO!vu?+p*$y2WUM>7v0>}$ zZh`I-lq!?5<6!QJH{dfj^gI`gy*WMT0F#|3?}kAHw)SIRkjoC^`Ds?93(3ZyHC{LC z)v~176Laiduhe0cuz|h=k#d(gk*u-lDeLJ02!PSgm*0G?Ct<*63 z8EqF0OpUn?Twa2y9kkCBwd!oq-J*dEW#dy6)2t_@CB-Zy*l+LkjvTfg<;N}MMTna@ z5HmFv55AP8$T;nXC#z{hWT`ybyC;;3`;whJ9_I(e(l`%t+gX!vr;39D0F7X7>==<1 z5V-cJ<})fJC3x_<{AUwiD4!w>D{l2s4RQkX9&Ln(56g|oF}S44ux|GQgF*MfL;;)W zDQ`%{-*0Hp`tPMsv| zPS>+biNv~07bI!s$9VC`^jf0tdEcAFVcsz=ggm^G@F&sXINM5kaNK{| z)%8Q_DjGK#^f|OB3^yxG@xRQ{JG;zStapc8K60cv;L0jLmtcvEdyLDtG%RALW5tFh=OH6a4Mo}6ZEvz8c!}Jq zl24k@g@WhCXnoFIpS)UWe%o_6k|-FAllt~*{lG7o47X5(JWWAd=FhhuFgI>A>@Q;P zep+nY*50(gp$5Rzhf~s0bVlB(D?>~~c=;>(2LE3l=7;R}U1y!L~pbB;Kx z(Kk1r3*}B(nn3xK2$W7s1ET}5SRl;}Q_|%i1rh%yw9Yu=0(mWOQvd4HB3B!03&^$d zK@dLkBc8;#(GwgDTzw!RJ6Lp8RaIgHAm%793d|3eQ3u@#n=52VuSO@Y%~p>rV_#{9 zu9Ea;t_1HB13w`626R~WVX*9oeIr?^q^xLDYss&(cEzA`!#!>8wyIx&0S|Cqb6#B( zOOmJ&nVo5rSD4~iStTF9wzmi}4zz^bEEi^m7J+_qZl%GVI`V-8-sZ6tKcE^R`==-B zghhqs)pT?Q0N(OJU|Dy`-9LF#LIM@jn=I!v@K}{$`{)?m^T}Q1ON&GS%j}9a@`U5 zqE=OKMA=%wj}%0ezbd6q>1fyk^ZO{<%5;v&NDh8x<{99lP3A#6fcYnUI2qCmL0UAP zt+lR7ao_|FFgUjN1r+nVLBrI!)o_wnHs z&{G}`nTIvAmC=C}4i51T7l!NNw>&_3EEN?-Qyv#UWtF_kOZAM614uMRH*{t3bnAi% z9y6{#V-_y(wWvt+p7&4EYPBam#!X>&XJrZevkuTP)8)8^#Xhg{CN^-(DGHLFSzUU4 z@x*cuDayt_l^K-&el$e}xdLWB$|BvJovdj((oQp#^{y3AaZKGF)c^OacrS5*VVyMe zt*c?j$Pn-?RyLLdxQvmc<;7ZSAL7(@ZZ|ZZ3HVr2yWwOONpjQpeXs zBoul$^5p=R>Mt&c5@3psKAs*HlX=t5UIjTZ%YD79|?AgN+?>4vfV z-Fr`96#+D2(Z)!mm?NgZ;CsT#m!V(QLZh7hnSv~IGjndEvL|F(#OqMC+0Urw{?p?z zx)=UB8p)qTBnoUU;HxOI9?4y$&$z3!LYs-i%^B2J2~+>;aD|TEA3C~5yT5X~K?GgU zkOXsJwu){wuO3%i=yXGdKmT^831ER2`gqXDQsZpOqPS7MDenCFYnf+P&EEyZi-wvi zGzG+`rlc5U2M3b3nGMAHdWP=PijC!XGXV=kOEQNG48V$V{)zVIj<$WfZ>g=V9a`d4 zT`Z9<8#`TIq#^sELqiwaNdJZ-&Bt<5nf%waj8)&wb1GS7gG3q|XItFsy*I2sQ!K9e zrHIBd(pONDw>LL7`smc=#n|^+7VIGq^U0)R%4hO2DZe^2JWeqQlc+w%4C`FXEX)~= zo=%RQVfd5!SI%w!+5A1fTz048;gz(T_J^jZt&6Ra-ZeXekvZ#qVY2kv%*<3HC%tsL zPOqaomfIqxjALo+g4RDE|JQ8wDxz6{%y%7P~<-5BuKL*{Y z&U2Io7Fx*i6b|0-)W}H5C%83%^i1-0?b-zUZijcnr5Kx@mBKt^q4>GCEnG6H)-Mhs z(!+w4fZC8M|6sXlhyd{B?Ch+@?>3AXi8XB|+Vr_wZI6zPWkKlE3q877 zkVFg9+e4U&xC(#j9#R#JL1oNj(Z?Hst2VBTOT58Lp97nUggu3hallkuj zyAFN)rA2B_y2+nER^?(W(VR|>Lh~9| zICj@+Z0$=yfjw?$=L;4|#Zst-z}Vr>rfSEh$1K5kz!HXv8JJZDFM{}k^HV-g#~%3# zOEHiw@gvb9h(yeA=ay-$2UTt+1LeRc?`^g-3yN{iK58}C7#2SFmNIC(rWEiuc*{a+ z#JQ8BY)jy;my(~-(x%{fERVh#ek##U48jb4;Nqewd-Ev40>01Hqr`xQ(7=W81a2;o z1kdIo%v)0v6McZU1)?|(nl3>hmye{?v2cb_CG;~cH0~5?c+{)?Bs#W+_Q!N+=p2m_ zdwWtFuB#W{@!G7#+pYbRi7s5R5zMvqq4+dzZk6N5v}$!ONo};{YA9!S0JazgTsaW4 zgw%Df*VH{TOqP?A14!R9$WfNV+wztPDiO4sZvLm`uf(^w@i>g!WK3P z=V04Z7b-+;bkZY@N}+jpJg^iWErJQ4DrS@dmh^KPk*|BAKNsJ4hF1n=%wy(zDtm7a zjRXZJl~IcDOz=K9&eiuFRorO$?&pys!(~;4&`SzlFt%0(Y2R&kKHh>y7n|Ld`SQtk z~-c`^h7W8H+E&yu8<>k^_U`^VptUM}tOBhCi-rfvkx zbd9L{I5ujSKg+YciuVAjTG3hU@4vY^3Gws{03=Pzjvb#z7Mp6z*UVjfxAZ;6qRDZx zuc6vS4Vr$}&Ty;SiSD1@P(NFcqPrXTF2xr=YGFyrrQLa4l!ZOGqI;aCAN$2S#h}P_ zgW4`PVy}jzI^~)(rmv(Xs>fws2|{Wos-rp7`{A{=Z3!1w8>$_2)9?DtjaJ&Bw2V;= z4O5>^K95$5(^mBD>KU?fT_{H`if|a2qlP1Z74qf$D9wfa!Xx_PAEo#;V~o}dkxv5^ z?4l6Z_H>J!&i42FSxGj}+a)h^Q_3r=ET^S?SEr{nU?!^7z=1ed~Rfb1X4cm6tDfQ}?IGGBPqsFz%76s!NND+l0vo3l;{z zj!ttOQ{?iX(|WJ!^Gu9eo{?UASJHHcpkHUOm(_6VE3yzXIw^YD8w++D?nvL^CA!LX zmI3=hy&HIrmS>{hhh7AC0CwYgr*+S@L*Mbt|FRgt&UEDAZ!g0N>Dav6haVY4|KVXV zF#JDV%Ky0q!(WHT9sXUH{llgLC$t3q#WnrUwCBHj|IL!d7I{fzE5;3XL7xF$qawxT@kaeZSc)q;Jrzl`Gx5JgVce>z1XpiAZ|hKx{Ma zrVYiDm(LC&d9@U;F?{~~>GS)$fmlrwmrw;0!SwBBqi?G?Wwq-_%Jt(Ws}@V{l*WFf zXG-K6Vr^S8%Cw9gco50^O=Yzk7r&5%(q4Q%y?gcI2QB%fiyv1NER%C&^`wi_ibPf687Y9Ccf7~14Q*&J0ngSn$uQwClg$?Ch9^RSM@`rrci}Zcyau<3_~T-o_O=>V zu&t#g*%4-CwA!6dTo`L4V1?fef5f>z)EdF{Km#c@xj5DPd$Zta%(LXyZsZY zOY(kLm*C(t-Js64xMb()>YB@JC|xX}aN)VvmDi4@v%fjJKT>a6jWEatu;iwbM$mFb zQt@js1;HUH-xKGG^{`FjNe&K04h{=9;@5=nXLy1scGdDhE1_5mbMsW^DMXhGntvaf zFGKy0tpDm3PQGk&yb3|npKNUWy>&XwvTIL6Rr3r*mgO>2X;x>u!WrpmawNPXOh#Dr=U0Xcl-BgsOc~u6$#fl{`-!GLCSGictY0T z~64c0@pIVTLiG4yV6Iko=y_yawvcE9N9^jv>lX1e+ce`w+QcPt2B_T@o?huAyx zkZcl8)cAN|Ev4LCP-_mq-EGw$yt`*G69?GAU0OMwvZnMEShcW`GHT`hQ)K$}8X*~^ z>N;S^dmF9Yb>%9u;oMzaOD|}wn)e#j?c(HM!86!_g|}8g&Q8`~r`j5{MADm;7auKS)j-$cgaWRr2eJ#J&v{^pYgJh>b30-kRVhAL<#oumx?x}HCFuUwxk&qlj zt;Q1<8YCh1XZBFq7b{=mtO_H`Qrd7(z27zn*-lhlIzK*)v?kO6P8P!AwX)Y>U1Ai1 z-Wa{_#xLX%(Vr4h%D4Pg+Gv<%X1u~T&Rm2=%Hy;%a}y#oyw7k}yER=V4Y9)sL20ld zUWC0I3u3MPY7t?S<^d@1m^Yt=K{(f+4wR<~0R02V}*Zwp=*hc7a$A~-&9%cM>C`chJ~lz?9Y+5zVG69mn& zS}gU2Qr0#tf0nXSBRimB0-dFGE&pNWm9z9WA zT_$~$aU{v^cV_Es9uyD|5W__8jh@(?%Rz=$a)Mb{GnFh7Z+*P|IhZVtMSjJT>+VB- zDQ-#0nTdgkd3bH688Q&>MZ_C?MjiF07dg7u1|>+4!QN@QKo0v;wKcf38s=)ZFzj4( zoc2id#-w8nHer-=#p9%2r@htu{P`Q*pS1UO@Ph{j*9Y^+>D1nq2S%DQ0P-^}(^6O0 zR#%Vv^10=YufyOh)%;+y;aUCMOmF4RwFEs~+hL1$%hRfu#GOw)vIH8;49I>#+AaCu zL%aE?QaRMl<*kXP*33OzaK}ifdds<1ok6qrT4tS97DRe@ql&@9yEKMLu3mR5Bp~3$ zpK;%2A7@lS{)670o*wUE7o=1T|59)E20I-54Si3WLowQx4HhMIUsOO`NK54SB_o3{ zL#I4Fs0?**l7iyq;z4VDiH(YhjXFGR-D@8%vC0y2IZv;km(+0O+swrPY5I6$Z2h}2 zGHQ4CAK+izhYz?d`MKQGwH6Ffrjg>I%)LE`gJ-oJu$r#QThKW06IqwN@L_q=J8U;3 zU5?UWNhf*T9C2)s&dAC&LWMlLoXt~rYQ-Y-kvXEk#K8SL`H?p;YATSKiRrv$55b0% z8W*}mB4d|y{QIDxD@mthlHp9)$&&0EN>fE;qRDl2rlY6l2gKKN&%47sBOcs9p94?a=a#WYOl|4;H9wz z)tQYr_9i^Mv6qhzHXcb+lnB0g5thUt1yy?Gd_!7V*8~l^I{QO3qCo=lCdZ?V(Q>^> zq~3gVpWO^k9|vxKXd71nBpWA3bUd2#^?>}o$!C4-dwi;xLzh#-Id$$X!3qrml4X>h z-I}z(SA`ru&)>PaRv*kVV6vPdPJ07Wo-Stbp$|DZt;Y_gyvDO5B2HGlTN6kxkz8&2 zR#{gy=*{{kf`=w+B>3`cbn?8yr&_5E9fCtST1imQX`T8pUxKQc=za#SYfBJ4l_6FA zUM)v*Y7d~Nwg#DWD1UbhJJ!G&i7CB|X53CXOn-eB-opnMR_er0{ zZkQldFY5lzVbOeeJkMaQSHyDMxfqE zxW}%=M-M#CQgE|P9&`*Tf#fD7GE--!CZ=@N`hYoGKf6sdK0WG2H8_5k6e&0+u=oRc zDf!UJaSD&UI^AGL9Bw_+iS-_xr6QjH#0w+&4(H56EGjibMn-3r2=vVapZF=$l*-FZ z0T?_cTe`pH_*m+v zB36v+teSn!95WY6DQUt1gt26q>Ag2#f}tY|Yy=a_fmr>8{vQD78^3|ooo;Z;B6Hq| z#>J0dv?fB*oD@^x^MhPBqmEosVlg-u;u^Tl5;?DRl3bzo%Vhb!1rxK-2@ z{w2jbMb>B0Yaw0AXW~aECCc5y>!GM9i@dF3`JE*~zFU<_NJ@Fr$q)gihd2-gd8Tce7CI)`iGM1;x$;sK?(hQ?voW^b>t%aOApTIE+ zpJFkHty5pj8T0ui2CY-r8xvWHsSm3fTB?i^{;Lg?Ewg;(-sr9QpXINEebYEe-ZLmw z=(+``!v)WgwR(i3%^Eb2PJm24Yr`9_Vj3vHpdgBKbbR$H+n+XqxI&Vo#+A@3gqNM9 z=*9gL4#0RE+moAs8?-COC45U59~ap>Lt~eq*vozB!ML6?)nT#Qd{JIX_>^7VAQKR55Bw4KikqGd8sJ2W9$0>Ar>Z z*ZVQ91Pa80pOi5_gXnHFy0w-|KVSNIC|%9D2KO6mQg)a^AH6%^cF`MALVhn#0BGEby1u}Snwf~NVR2KvMv^^yt*urY^`h`4zpzEa z>Zb0RyGo(0TCi^&lbAp_?SC(h(n!Z+((98p~wKTu;TXOPwYiHJ5KDr8xbp6ew`&>Mr!@^~b~X z#EP+`r{$=4+Iv%XA1Qnq7%2Ua%)hj_B80MzM`H~&nZ$Ymqig|Bbh%5XkRjGylvC*q;xCh|GT})i-y4J=# zWN9-AWe_Q93=4Kc)h2E4^F+`he6}`bi_Xer$>`J|KDqk*tdgaRvCy#=my%C}DKWv- z@bC_xrmx+et8(A>K|a!NH9k5kTnH-_pk%i7IT?!+3l?y!krI+S-%E1J3O>PC;8@^x#q!6Fo_3p!xRtQLi<+&V4)9hqtIu^C^GJgTtw_lG<(S`(AFGO@9LUrl`V!LTTBf;Rw?brtDF3 z2Plt0Hdr`8;*@jSDy|-M);}yKaQ=dcakYi2=-3F%m)IwJ1NYm16{MlfZ|XBXUm}dwat*k_;~ldgN8Td z%J<@p%}frr=HDXIl3?SLj)$u*jSrBF-_VOs<5AZ_;+}hGJIwuf)*xW{qEm*x)Boc$ zp@#B(feKjRMBg#wvB?dhR&Q2pildHkRFWIga``zQMK8pfd(dxZ=TR9zXodbL^{`mm z@a6k6cTFB+hICRh^OqQx%leaxg3QI`gQSczk0gm+w|<=WVOyMM>vMcgVG3#HE4-3i zZx<71U^rC@j@8zaO<$|v0;9<+5uKyp;Y0i1i4zkBPH?=e|I@caGTL#}6P;-=$UV-iH{BBoHdPKX*M9Pg_BCC!|9o?>vsWAZF%928! z9_HJ9Ti!Xb|KkdNQ#tN=@zYM-7VpN;(2!lba)WUb-10whpIF3w9;ea*?3_SJZCSIY z7`_JQ@I-1FEoU(Bw?@n;`Ef5Vw+6?!dbJH2_Ey6=W7{mQAjVU6+eE@gjuWWppDSyR zv4YLZp}6*+T>EQ7d_#+evWU&t^Hw`zj+&P-Q0Xwa3VAoD2X1Kfikap`Rqno*6SjR1 zZ;Kn6Ru$O`-c0M_1Hb8&tn()A6D|Nqqz0xIHOSAgvs0?;*YSrTn;cXKpY6%ucjh+w z2RFdb=6^UoQf*sVMyI5-0VWK+;V0^1K7tacCR)h(mYR!~diSq3ICh##h#I~S{^9ED z^@HGx#KP*Hd{kDXX2<>KbrT9SnE)>ARXNX8vhx)@l*OOq#P# zmgzM<&J+~ZX2~On3&_0L+>!taoc_K(7yy0pH!3zsWg0Rq%b8h$)M7%nZ|sGgRmJuC zO;*Mz_1kWc1QJN^ zuxz{TX1Txspvv-sCc5wH@@`Zp-NT~Ht;?2Y4_z4<1IdBPt(jYq@WKF$&oW?c}>xe0-Nn$<|4~H^854W{7`Nj#I zge-3@8!wMnKac~0T~^E4&n+|~z{PE3^1i$waI$=6PQ%t?l$D8`@58!Bzcp`~Tal#) zQfWRyBsD(1a&c`^mh6lDNU22`ez0lQi&fSg{}UTBmj8S2aId4Xf$bhUU65~AiJOG$ zXrBWw+06XnBWyD zPPyQ+81;~;&J`C5<~w&vEza`7nk`zK&M{E8&*L8yl^zzqcl{OLPAVI@dt2RZABOTo zo_!zIHsWU$uP6StASPEDJu8Pw1a|eKLJl2PWII-EZPTZSavt`fWcu3N>5|@Kn^~>n ztg7_@eWM(~61}1>f4)~mBRDXa3EB<8F9ga9Sn0lH%d~v9c_hS&sIWXK0}3FCTOJr* zl0Ok6C6@_*kaOzV+r+4Cz{jx(S!Ql=8Mm0MP3fbDa-H z_T6vX7%~ud>sCVk1h8dFEoL?lV6lxAZPYwIM@0B*+Cj?!7Mn@D2Vk*h1TFFSy8GR4_QD?l zXle$aoA|OkFHMyzcXt#NG$v$X!mHV4phO?>mj_9>^db_Q~^hZSDjD z`%c3#mXP^?2nGLxgOmMpBlbuo8yj0w)9;8`;~JNRnMzH{Th@6DE&gj~6(I!ltglu2 zY+IGPl9^ej{%_o)N0M)+LYGp?k)NV8ABsONUDrPK;sel=C$&q0CA$oLlfk34oo9Wa`~}^G)PfjLs5Ug1oa@R85f|*LpIn!g*alB^wyFd z&ilHf%&{QXk|GbnQeC-n_RV!H4OENX=v9M(nGi>8Gu8X=#XG?}yUoFCzZtN@Vvj|G ze3bJ0hC%{&r?@OIkP?YaxR2!%-;WWgeDaZYzQ=tlIDo+MUpvo}#<6#Hbj7huq{Eni z4Zek{+?V{Vr|z06em3AVt#+~l`{iv%$2S>Yzq8zC_MImdzwSlApDqZQ!{}L=jfNpd z2mf$7O3gqa5O!%#_odjgz_^UDeAZO`SvaCZT^b9k0S{_u+_QdDw=Ee2DE!=7ZkD^E zs8Rdb$Ti;kM!v9RkAnG+!w1tt>Kwre zIRf~U@|Oh#kh&7H5*XySd2+JP&$<`!41Wj8X7l;QZv5cvE^asm!@ug(-j!d zXl`bTW1H)4`qTAR!@LcZ?n`FgDi}{*!A({0h1u{pnK!AJl`IzA>?g@BG(X+b1F7Br z+Ja<;H9F6_ze2QdDtr5Wg$;$f3I>FaA-bkGLTx+0ph0%wKgX zyO9zpUCS4QqU5hsH*yO)SxP(cv~-;;b#RY$aIb9@1%;GIV6wJCMx_|NBb>3z#sJFx%-?3vp6gg~^Ou2wX1Ac&G3{>m%$lZtJ zpBCxIFpia2s0(w1HV z#l=1+_zWMeK^2CMJ~+O&Ga|_5wHpnfwB}KHLX@6WIcG4dMn)FD2Tw$0`XIHQyXeiI z6D6G$iEVkgI2I{JhOG5hZX!(}ZKRWZ7BT1Q1*;YS1XBtQOD0JbL|c!MX+Rg5x>2}t zr~P~v*L^lQ|Lx6~Y~gv32#VQAcYKTibZD(%uAtT4FMLO<>W%&G-h9Kqhw z&X#Yng3F?jvB`C1?89m+k#`xLb&JjQ$>CUsbuWk;mb*si`JJXT;hor*Sx<>U(R-H#kY>av;~p z+Tu#}m+OJdXrHn)s*Q0?v#ATcZnv_ks%~yRhL?1fCdOt+L1)-FI)EU61(`u#@;6pH znZSgv0V+JcCreG0(to(wILQ}O3+N(JxpTs^NY;DqM_TMr&W-q1x9JcdykZEkS}c+e zv8U2BEYVTw_Om0wd*6Q_iNkSh5vNDEe@VC8fikb5~ov{>#*WDd(nV*Zg$F%Uj_z% zpfcr#Ybny@TmVF_45t^Kh7T zI(-^nEj+o3JI3<7-wNb5BkR8W3makvtq=xT&t5G$XuT9#adjT%g<1XCxv$;~2E0>& zoJ^V-d5sn0Tuy$2A22*qzIY0!+=ClM`v^(E^MY9Z6~3-%Mm$wN)UWlB${a z_wHlHg4}oaYJ#rhSpA{`09T!z=GcLln1^rhep1T(iJw1*)oWYFH?Fp`*_IX@;h`-} zCJ7%0(?kVq&VdZOHuSwP7(G*euAWCKXm+xOB#TQE_nOke&Q~60&s(UQ3WOMyr>EU% zm38`N>m2p*(0o2%1>(;xD|Wdfl`=&@Z~*1+{aJ&5(s}=vr3Kr^@GKl<&(IK@e&c;6 znBa{q^p|Y{4d0S$iBo7-um0H7_;MgvIK|qu=P1S0UQu6PV-CdgI^czhvCWGYw$Iw{ zbKmEd^`k4c&r3vatfZ5I@YYLtBRie@YxYcdrZO|XgY7Xu@L7a71@_1nO(J%mYYeC` z&<#e$lJj{P@41KlOskOjp@C9?R1NL~S+Tn@p)D<|d21$$Z{yKdAFR%ZWgD$!f)S3E z+IJr9lwxmNbGA=GW2UYTkGB|9aYBdC)F9pE8Ot%IY?Yf)`DJxRS*{o~+{2JJNg`~! z9Ptx3bfyd>W5zisLPRzQa*o-O4DtZd08H=W0me34SUAMOqAQ^0dy;M)br~tHSJDoT z+X7Oj=b!rF!9Rg%q3`<7US0;6p`ednsHn!r#=VJX$YVlyghYQKDmeK5+jhm2U<3e+JOm;O~#Ea+EE7IAK(2YCoKf&zF}TYdP6 z_YkPPwwbS}h`mWxUD^yu;Zh; zu10iW)3WUE&z?SF?@CI7U0fV;PSQVYzbXUZlKlIM>YQ@hhnW1u5)c>(^>QCB-l6^6$`L+f;^FyV1zejq*&40n-M+v z#!!tdCNtppvvMAuES8|y*^vx;Om-2k+a^4Gt~cFg@xr~1_dQkbJGF>F{hAs$F=Qolp z%5Rl^w@cZM$~lQ&h1F@^3peF?a@)T)rJ9u2z#mU859;rJyZDzr_mZl?>fca$LgP@O$8ggv&;+4GP(mUY2G?M{>OW zN40mB8q$%!OcvMfTqd}GCNVz*$8U#q1Bh)_t4%p)AxgCu9#Q-4aAj|mEha5c^rrU5 zKThl0KyjeQAOA__c^iwg=kK<;t(c8l=8n?Lj7%yJ-PZSLSlDa2`#}1du__6qc|Ioc zxEdVJu@DbD4>qW%9Ebq+J+ShD6%%C{x)nJ+!NT)$$y?u(QxYc*KAf9>1vXIJW`SLJ zHOOPm{i2fNdwJF=LBw>0qEq|dC9&El(t?Yc?92^4ps9)XLjQ2@J{N4HiDMgs9``mp zL2MY-!GSuALCAtx?kcUa@!SSHsw`K(}LSM6ApQT1&Q%VDnWERyLjd)T8L|1tVCxNG_}m2iaXWh z^U4Ea6Yk-C5p{Q>&icxqdmtq&U zJf1m?uV5zcbBSRZt)?Qvm2GYLK5WY!yBcg`uyb(f$5K8s!yr;*;P>Rr_|()xUeTbn z&?R|dP0{xVt7I{{7HsEC_cyljJzZ!)9_v!*wh%j;xd!4oQ1r#dXnRdDYApfw6rEg7s}S__PfgHxHtm~9lo?P> zTcwMK7!%omU{Q;a&TalM3ZHcxFMbOn2JfDmb0`woz~W`51rBJ21~AZ3FmL%d)tS(e zw|07&j^eG2*OqVVIXD%vJ^?4B(vMad!mJEUhK|~t-jB`HTNB~Uk?bGC-`d!6b3rq3 zt;PNwAL{H9blK(VU^QDVWfjQlmWR02kxxLU;5{jESHwIgS=`Bb|H{0jrlsG1a>tlE zJIaDg`#v`m*qPpv_Q3D-dKPeZr9rJ^qM!dm@)YvYB#?Z?qxo8e=9xz)mgad|%bB-J zxpttBE7bo{z8jA6+$d~Y_7t^-a*(g7s}ZuFZU@*Cf_W2AZ1|mbOJ1p*DBoDOw|Zx0 zC%BydGq-UR951eKK^SE?IUe}~+HYX02busAl&Rn?$Xcw5u7{egM_&mx zxIo&BU^pmxB^U7bU7#F%eHJL{VysR6Xde;}#7EQ0fOezkwba{K{>6Jhf%n?lz}k<< zmhQ_wb`}^K2G->3dc?nB1UjCKtHYk+0!PgWo2Jl$ZO_N*(23U4$6!nZoCK2aJuzr& zpcCZFHUTyshWMa(t~4hC$>aXFRN)wNd^=WI+iJwzyA&)JJomE7&L=eBzoR%)F(+6E znhmKQ%@f0xG46J9oto<}3D7?e$44uGh>+6=%)N>= zZ{1A5IF|vGm@%{1CJ?3HW6lH?nWu*`ae9IQ?ioUJQ(h=`H7qV z90R4dkpLD+&a`ja7lk~ln>EoP1kx8dmIVlYra9`gzBxpahg z^{yGtmUtu{VBOUb*Mi?jk(X!&fG)808xtQE0C2JL?@l!oZN31Vdz5i6mOkQytr_X} zOS0m1i9h>1Y8foABn}-fTmgg&@Jmc=j9@SrWABE5Bb;7U2*IYz03;yhvVMNjCgHOs z@3VA-Tu#&n8ZH97ooKd@oq6$rN`)o;#ufBcx|;x9w}dYT$S5G?=J*TP9hwH%5@cfxjeKS;=?hs<8!*6v{>?aMT$v6U$$9G#^OHH-1=I7RQLFY(_Ebrg50RHYYf1B#{^>ja9O!z<9YX5gV z{|jyX|F=&j&}Cgv)ezkk&>%q#}7R@&jzb0-0Yyi|Oa+}pm{r=6%uOSNzwChWO5Z@FZ zHHa>xZ|H0z3H=}h^$_5lfic}Y_sfoh+Zp6b&o|ra-s<9&fO8t}i;H%?WILS*gC-wJ z1QU`}sgK0ZFG(1lH$39tbLe<)IayrK?@LBWowtEWg_fFz3RBvFg!1&AvDlRw=^0YvBm<2qHjm~tI6Wq~UTZINjlVe2erOan$7ZooN!oCF|6#?;h>_)#+-_PrxV z^YPwLJ$7Km`*Og}Jwq;0AuUP}u|_Lw3P4UPh-GgHK*C$aF?0$SG{!zeCj)(Pe=neJ zCms_%4hVGCFPxPiT7U^{fE+7keqdr4Z%vgO?=3N4&CKA&^~(gn8;ATZW|_8@Ap#C< zN1A35fQr5N-&q!9vA&LopWgcg3gl*2&H$D@4ZlieCuF%j)x1QBs#UTQzNIkW?{HII z9=OHbiX>3QZH1ttp;NmvC6L>E@LnkvQN z2FxnVj0HD5v|T+N`NYf70nD!3l9VB=Ji?d;7$7be&B=xX6_F={;A9J{h-VA!V!Af{ z6Y1%HIJ&6Nc5j@k-3cwXOQrzqT0c?LuwG`gM8833`->A~u z+LEOI_xR56BxjahosSFd4nfyYO>LU1)gu-4vmD(atGsu%K9<#bB0x|eA0;+sflaID zBt7>gshrp|B_-XbUKg=xx;Od#8=5ASkwZw*Afh_(nO@KKqt%gi6nU%f)(>k>EF#eN9}e?I+)tkh?S|05r;+sEcvz3aJy&74 z4Y$od2Y)(7+p)v)#P&U9JFVDbr-ZL6;k4e3UfAQIws9-L8A8{CG$ocMQ!k<)pHd{n z2lW>heyi*F@SRG!q-WXO+@Yq%Ce59>SHEaE6ak^UBu1w(4u#+*iy&EjF)_++bC34t zk^-6uN>|NiMQz``P>_L;{XGKp(tDy1UF;vv!p0rmW5_Q|Y%nDUxVOe&J?$$Z`>P31 zN{(;s3Smdsoce#ycJb2^CtFMHH(|L&PR{RM^3iMiz0=-r%Qa->2$zyJ1DAyaBxlq) zCVd+t{d**=u;XYD%zBLd*r7~w+ri;kpf7U15vV2VJSDnq1;c3{2M6O?&LU`W+APtq z5J3^ijXkSYN{;2Ou5x6cUO*xF9eq+j){`p0Yn3Iua6;MM@2IffzbXF8LS>DS?z|-f5acr*3(yPi47^U zYkj@CJW%;m=`P@TiEaCJ>*%)%pN8jCwRM1}k%)e`q7!2Ln-2d^oD>6|@IUT__GPgt zT|AeryZ1=b8|5R(tdg4jGcWSuP#v#3(||9?^8`u~3H?&-Bo zPx4FIGDSbD;>;TwHzsQIrr2ysCX)+ZoXZc4f`5B&?gcT-{dnbt_*k=d#`{Fn=j2F) z!8J19j^CU`pxgH_ZMrN~@m@IX>q^!1;j3&-UTYzWO_iZ@oxeTQD(J76ij4@RCnh&6 zDE^v{uEQ#Kl}zT=f|ZDdRJMmsTvG||KS;8_!!oD@Y?kk2z90wWc(=Rg7a3xB-mfUI zD)B&|_0LF?$efvoHo-JUzb31dUz|^q{I$(pVP2|O*yRf{t zyju*h8~4JG9nq<)<*^UeuK%ogctV&?s;JFq@#5Li(Qn(j__& z1-Ut0(#ag?at=# zCJI&%(*RU-m*g>vBKX>jSwv%PSwvLMIj|^aL5*-)!r6x>x+E`IU>pzKjyS1uDY60e zT7Cye#FvkKAB8kX@~3Z>0$NvN^?+?Y^9t58Nyb{dAv(m1WqO6d)!jVfYH zrGSLWWHDKm8B%g+#2oMik?@9aPms#ryj#o`Kz`{@XFdB3&^FwfBfKo4_p49L9(Ya5 zIKve1HTr^&X?$WD*Rz7?(w%Hk*d>gxeb9eRYQKCEv z9S_B7z$Jaz1KP{KqAM3{v~Yix*#-|O?<{f&iadb+OkO=b)57*<@NJmDJYR4X44rvr ztsXWluzJf=M5%5z{gJ>R9)V@aY8+Hft~`A-e)qzJ6RQLuj_Jcx07nxVq##l8!B#}P zr=eCar*FtVN8iD6-SQ|k6Y~B1?C8_64M#`YBMW>$Bf31lAm*f5Z2#NXV;ZZL{dx9C z@ywu~$gw9#DBec#9ui=#=4AtJTXOH5lz1eP{u_cKPMO{=reTTZo%_ zBBFXXGn)!=%cKf5rcKR`pg&aer%kI{>@5qs8MThM z2W8&rAB0j{QqU`W!7QGrwHii%Dwb92vNhDsXh`y(T{!G}8aF4DbUk?}m`^Ksg$#QhyEj*i64`RyUN!Y7cnm6`W1Bl6J}cZ{~xX1wq2sFkyf z*M^q<8%M_%@S%MPKs21;>m&8Cw(cY(4z!ajINHlk{Q+ZTJ?ngdK^!M~O%?S=&}M0E zO6E8XHGi0eY7VR;ihFL07dliOp6#NXasHl@^9KTNEeK4}-M(29c@3-Mt+D3trYC_* z@q`1n8g*7Hsj^rLDvpl8JZYl;@feiI68A~_k@(lZi5lz@=G*F;E2RKor}GhFyt zCG6k4KN;bQyZfKv*-{061qi_5aZwfA-G5K?I?(<*@?YVS{6}vYWdnc@;BWx4C!7`_ zc9*HbVK)=LT`5OxPvs?Oa4-EG8YxPituhG!)NuwOAb>Y|+jnx10q~C7;p_IZY?XjX z8+q-_!;&@K#)5<^ku1OV6U%gw^6txBa7~)w3|%`rX84Z`_pa6*rN!;5S!R_o6SxnL zIh$XwrWE9L2gZVV@885+m+*D|XYN$Ht(~mz!0VQDqZAlcn<0nj7_uafPtBmKcX+<@ zyZIlQ?UG5yXww=bp#MiM$r?gy#uxf6EC>Y}RK5bls0erLAHOR-PQ68icE-J=_?}=N zKB?<)nwS(Mm5PxK!@2j^PGFbS@K>7x$gpLBheO)oID$vsD(JWkGqB)hZ#yFYbN&^A zf1gBefRw$(5{ zW>NEaZ6~e8^Vg@0clkeghTuK?BjJM&WQtD_~93lqTmmps9E; zbV_R0l46rzyeSG7fUHZI9<9AO%Omp6QuXP3&fCT=LB#vmDW z5uHqsK>>W&7iLP%aa}Xh?pHw;?Jv(4sJk!e4L~m$f!#i!mxxh|^yxM2Fa>gljw?Ap zBADP_l}z~JY1wmc+sV@AFzs&sW7VMi`ANNRiB}u2930_eRi3p8X64RBpg-EGx*1

M+kOT|Rcn#gveE#$akqDS_nOV=OkHuBfRr{-cvnw{(pgxCQ?{`wI)=u@uz&lSM* zdz66hJorVK4|-QNlTmByIjh#bT^uuzNM-g9NB#m%G^?)-OUV(gnCm=mK+ZAYm@^dO z=wD;H4^pk=Z=ICixSgnrxVR4xu%31%OTH(+J4ZM$d}C6H{B_gm?dJCh-#YH7zDPS=eAZk74W}*s zSmr@b_MRUIAc^r8m~uFF%k};4o>8?MS5XwORtk>CU4k#Ym@<9Ooq6i2eGQqn@RK`n zoi~+U)Vh=njPC#RD!(@P{oq;cbJi1xbr)*jCV#~2gzijJ;Ocva8AEE|sK)Dx4R(~x zj}tQog=%0i28}sCO{Pj+t)I}66kkVfKl>ZU-&1I=tR^+>1#yVj9Hu5t+Ub^Se3wI) zd?<18HzYckoIzKJ-sNl?E1ahJN7ve+J%uIi7#3h^H5UkFG=b?g-FM1vFeG zszKC}EQCC7{{5y20M3Q}mzYY&w*Oh)?six9V<6=Kf}W)Ndz1f()S$Bgu)D7Z22OHr z=+H%w4s%?5t%U=vfilpfG^eCq;|8=I3Wp!BCR`h$_JL$3{6*w4M)}VYW5d9M1qS^T z`0BuJ@r%Zzp^V9^DPMr^p2$`h!LZ$@y8Jii>355g|7lqGrbAL#Pwx}5$@Y;8T7QM@ zX^l#l7tO_xyA-SP@y6sA?%DLNmzP98R@8Vv`_je#kJmY_zW4)e0;I%T;5k?9$o>zX>h3l%HHnR+*Z8%zA`EdEQyWugf)NZpp^$KAWfPd8$ zx77>?CnawZ#=>b|ka|!9yyIJCrmOtqLX1igBmWi3pA7w2nj5rObL%^_ksoB7tB!4s z$_W7~otD(m)`gX4 z3&_40y;P{^@KxUV(rpUa@WQKOk8hi(CDGoK5o!RHkTSG@Cp+6$vOzmPhT)N2+(}+8 zIhCr|a!KBBKRcW5SsyWqEdTc2jP2jOvT0~^?-P2z0*8lg{YCvVPm#nu7Yhf5tALRK z-OPHS_7H&)fqE~W%Cw9a~%lDq#I z?>z*&P*@qXCLeld@$W8e?*nl7hPHgrH zbpQJ2{rdmL+gpc4)vbTySO_SBARwS3NFyNKHFS5EN)IKS11c#gEe+D$AT8Y>Eg%d^ z=Kw?JZ;d|BdEaxs=bY>J$9sL};sR#&?6vn^>t1)?pWFCBOk3f#qU54K=Aoj((D?$; z-F<_7mR1#>46TwPe7jE2{V>Q+UF#>y%jR+CPjXr)miUC?!mooJNGs@$p-g^!D zQ;TwIl-hLPy5m}FMcDh#l>#1aXPCGC8k`e=^<+OtY5wh%1YStbpzE&62B7w#j{#e7 z!zos$SeFd793=e&gn&_f4VLo}lnsxOwePEsyoYdF*;D?ToqVwy)e;-RkuS#tH$GJ*dBQMEkI{&-}%uuP9R_g=!NcYbffET`v z%xd{%%=mjW6g-gH68rJzC!AWbzwbcC_s`RSb(5)k2A(Dj_qR5WvgC4AEJ{Ca(coY2 zb}FMqotl=PCBjPn_E)Az-EEZhm6p)41NZizT&hha-)ac0X?9T3GTJry7h5i z&JR%Jaj^k@Jx7!jHv%0MQu_t`SDZ5V6R20O*@jrk?ZP|JFE zBwdI0%)jBk1MTuT#cLZqUhYn%Rq?~q8F9^X?#s?na^`X)BXo@%7H`@kvGa(v3s&w~ zF0XTM)o%q)K88I?{_;kSXixwbLplCA4}my&bg*CY7Ok&L9D0E$69u>t5oPt_2V2W) z(*vXG7-i6A+mqsx`7FK!_SJ3UE$vAWHaiJ;>@=|{jyZBAqHp*^9&oL&zuTz@r2#U9 zU;hz%B73bFlA)m0TWFChUsctG#&S@Y-}{Fs#>i7Y%#oMx;y>zQT^o6dwLA0yZ6J6J zrbFmBkOLsCAfHIqfWvc?ZRm&VQ{UVx$}C8q>EtQy+upNa_WgoSm;;o4U%$L`(-abF9_DS|4#BHN!I_c_3YPSb#Ord2 zsxH%yydJxCoF?7%Tu~?YwVpHxjq}t!?uYvQnq!7zl3Ve*aM|VMBzqbC)a|J$Zr*{B zK z$2-L($?!oecbtNSLzGRiHHzqfz9*0sGtLunaqP5yQSYz7VX62D3il2)b>G<=YOB6{ z=KIEbMHFVaal5%#%EPgFn8t9%c_aBg;pr-3e79+({Bs7SJ8OoHleNcw9?vpWUei0% zDbhReAVE7VjN!#TGAmvuB1NX|O)9Zy)yL$^mZrgc-peid1bHq#2i9Bbaw!(8dJOYb zbP$ZbSZe`qE&ZS^cE>jjN64Br9I}VJyyK6q$`|G%{A|Nr?cc2~5idj)8;LwUQ!K@%NuNBf4m~h-I?e zJE=RFYz|@QJlKz^QVKa&3lv*j&I`0PY+l)2#UUzQ8$F+?ENOmOmQvf%#vv4d_>q(x zV=`B%-B@#Fwy0M?fAe1)F(8-y)ByARFzVJlXHG~UFGsRLGH;kWned0tPk}mQ0XVpl zdISLdI->Z+!My%`{9D7Z3A_XNp9vw4jveh`W~89jg{Byb+*qY&Y;Ml4L}_oaRW)7> zYGd^+-7cAJk_;XA3#p+tkE5l;;Dp-*&2J)$^_!4ecvUKNzEbtxM-ygM(>Z)^B)V@);v@Ur1Kzwmw4YQV0T8pQS`$VKT`gQm*pAr%x zs*vXcZE*+;a$ff0g>5TFUsrEAO`{ub^KISunuxBPOyGr##C$JRu&3cmU>iKLxoedxKPWtG!dU$-2xmYPrhK%wd4U ziuZ^%lhQKOTboTcuHs#?g2CRUWpGMf9xjFBhET$40mIxC1gX9;gAg!TRa3=?csNf6 z+f)jMb5+m4etex%#)vrF@!AWugapo{HW|j;j&a-J?ype309o{>dJWH`WM=iC&FuCs zL3%yZ8?8gyN35lljlAZ;b#*#y4QsEK^o-{3$A-E*RI^U&_T=q3QSJ(>>s2p}a}H>( zgNVr^yeRXhla@MNUdCP78N8fy#(sZ%#q@|AMCkVU%1_zUO9y@m!H)03+sczo;WcM{ z0=njc)=HzM8)pm6Q`^Yp{Khry-nv5$!f=FOo+MI6Ir82Gx^C45sIb`(c={S~k6cW` zRV&5K-~4QQv`|q<;A-dVUE6UzRcX8;d!GKy3FW@3l`1a8XoP=#m)EZ*=|s?Q;yA& zu6b{?@xYz<4;GlR9)UK@xfCovho{HwRus*A15C|~&2!AmjEy@aEML!g_Z^bGo@)nr zTXSQwY$>6zt8~-Q1T1nWJ2H&_i)57A5+Mf=^?AMTUI+**v}?newsgFXraWY_pacHa zWQ=DgGm3*Ivn794;QPqz36MIsVC%nL)|y*QLI_jGn-2;v zi=1*iUlh;WUcd^$X8e()^M!|?ycM0p+6xAgGI=h&P{-qA*BuhV!8|4-UhF3KDU;b$jM-QQFn!l0?z`uMX;<#SDKij zV!fJ%UP$gVPBz!KNiOymp9Ej|`o-r}WpxzxWT>i)0_Wy391RWB8;APUVpW_v@&Qvp zjRCKbX2>-9fzcE4{i4ED@j(X;R*F^!7=6Ub^UqtyBl`@OHBV#2Sd7dt_&m>3uV}*%_3%=`Qk?Ynlmt;pU-6URf(;}?GdSAw@`vlXD>}#ClBdu z*>fsgW@_zeYh=fnS@WolB}l%49!`$vs0E3b2+nTT{In{?eZ9M*VKg(&aQXhQdVkXYz6CLvCvlFzDd51nzEbFJ_FmD^ zvq`BKSW6FA&!h9xy9fOl?@;w-Z(u~o$6#;KEe0=M&J>!EoN{P>K_c#{M~qWv_n2sb zmkFpemWuGp*OOT}_hR%k`lc&Aq7p{yI9b`En|DU1E2J-GZ!6M2kddbbxB&9r;*R?tc{P~H$O2dV>7nSB!2MO{wg{~T&`mPRFnx0WJ z)sbZx(^j9%@p-YabMv#X2=qxA4o#1J9hm!U1nassukhSKmiuE6UzBK4S6G04TR9UQuHW+?MIB`_1LjBK(kiYYt%g(fO_mUyfO z8<6Iom@a><{~*rauk>x4!8^1$vvt3akT)F~U!)Cwm2u%>1%fC!2K%*qb{{`VJwm@5 z`n^Kmh4jaA6X}uP9~UVYzac%Gl<>ljB-Y@$!XN{bFvQTH^o}Jwr9UVQdYvx`>02xa@aZipj>^T!Zo!~SL z6^OLell`(uQ(ke0m^%lTn&}`!|HsHSd>|yK;YC9YZfiqcTzW=@b9;h{W$9|gUV|~a zXgXGy)cL{c0gKQhnL(w7ozXgw?1+B42Lu8&bxu1K<>lUFGsp9O-b$N694~lGK)8{r zFRsyy0LRzerQ0kw;Nfd|E>DP7)Tt;4= zn6@_D>aNNFF2(UAH&tdtmG0x);p~o%vE555qN1bj*m|Z-!{?Ju(*_*)Tu0P~p0?|A zN>Foiy~(PcNQ710et*@Ijl*Pt*dlhz?^^VeT#oru^WLfElHL)-q;ap?o+s2mPY)D9 zYNG6-ea5uYEc~Q&qCKwz{zO1^)Srxr!_HySz)%Mx=T!#q+LsMu#=&GYL-6AJjZTI* z^*z?(bywpPrVrK;WmZ{YI`@EE(~RMY!dJ#LP*G8F6H8WE~>vEW4rU{y4R2JfD-s*904 zhi4|cP;4mxPgZnYOA;c-rRjpg5$R4YwiRbq*YBC}X=4nwSx=oKaC<7~Io!^NwH3lK zR8?ZWq-S%4+nX85X>)kdux{1ps)=ED@@gd=8(`wcP#l^ zdJ|_rA&=X}h5*lvMky*hd{H+652Og=dYFZ-VuImS2_)v?kT><&J90>1io9j9{z+C# zlqJ+@t#CpNx@hs5R&jvSr4FB~vr|SV(q`%7muPqxoS*2?{rFGRQ8F9T2?xl0fjre4 zTX7Q9*;jV!%WJwn!L9h)3k*$9lV%8mjWKa?uUe;4f?q;O6t1eVl8{>-x&QW1%d6a1l6`H6Ok2s^MtRb(Ce9#vvYsX zv!8sNz^Yb*!aZKzg1hGF!JQ^_xdfBQV0%*Q7kpqR(vL029Hp@;o|1C#JbOWD(_7M| z;>bh>me56ja(l6;RDHB=Jv$<4zfglF$4uV~*WpA&&RlQ({MvwhDpv0%*F=h-8=D*k zOMd9gub)MG#gwT^24m1W_6N$FW9nV1Y*jOp zKj^MJc=SExEiYX7vQ%PV=MsZd-OMgFA8k9)rF>|!fx3!3Y>5Qs=ToKF==&arYQT(8 zl#iBLZ!G)zrP#6q?m3eJtKZ+$WvMCpaXo!Ga`-}%%wmqS!04IpF}xE=M~o#}yEzlb zTj}5;HK68IHtIdJ6kz5<;d}}wo9k7_V5yRA%hM-;BIg=&+ogrbVksW4{79zYTCk~;#vjd;FF9gbjT{EE5?9Y`F*2X2 zfNiib&Q8H{QPr$KRI8f4%kyjqRF7KFbW)lb)2by*Vn1f&U{-+NORYU^_rCICw?8$3 z66HVph_2i(_o?PQLPhRWbU9dp8L z<~UuCv&5SXCXg2&aaqZPO=QdX%V0cnKgmE+we39ID_f-l=LU#kX+G~y#I%S6uDG-Z z&vk2RtY(`+80=Dyn$8-I==$yN`Re%F$5+Hy9CZ$|)p4-0jWj<_QR?mo!UGhy>0InF zcVQ){?mDtc52;|0%V94v>IXga$Pg=Fl(Lc;jH0z32;D!hZ%vFc8GLs3Ig0Idq=RW` z&5Mc57CdVS=N84@5NQSG^3`9dRw-3^ujU3Ad=Ec7O*kBtnQ$nbBsSyKjbT)^RwNKR1t( zshobVB{X%pnhA1as;M3Q1Zw9-I{UwMJ~#94Zm(S2FKhtzZX$UPGa+Hl>hsio{KUuz z7bMoMVYd7!v-5gS<}3*x0>_ooSIc2<;#(|JM?*egZl=XYR&xC7t&fdl-%8Z*Kd7#;fveG5TQ5Q^ALyEi-1tJa9KO(>ZT{Ms|w{&^gF%>(~KOg z*cPKM=5Z_NzSwi_A-=}(f>OcdloZA{ob2Yb(azsgtU05~G-*2DC=)Vuwp~n86nZ7CsrzwA zvL^m;X17vq32yfOQV>5gd9HiP-t?#w5}0K)V@M9?_TS zZ?7$yL*j7eZsS@NmzS=GG5meXezM^*ikIU}6&u&LDD8(EE;g&jeU(fd@VjiHCvgkY z<1U5d-lup26u}A@!xF{->=&eda+Q4{5y7glN;_ltf6RTvsgkK;qcXtFGMNvm|~ zrni!X=C>{l7}uv0NZ0x@@fh%ZPQy4(-NrBNt?I#)@A;rsf0#1=DcBpsj{*PuiX>MS zac>YQkAOS^aTnjr%FgKW=1Vel`lR_;eb{UDI{pcQF_W5?S3HHrs*Q>uah zgvkhUTJ&Hb<=H8-ga%gM9Ebbs&Aib7k`ZlM)HUTx7H-<%4J8o}T^UyuDw)1$fU4e0 z1G*XMo!Z}rc3uyQV!S=XH87@rzQ=tXd>Iw~m=*;gr!g;7F#KZWN5T_86nqmdBI-JBMF^uz}x{b*++(H zA9+iIE`(hbtlWOy@1ng2@-l%a`N@hR2&()W(mrV~M4p0Bm6&?4pIbOz8E)>h)6;AI z6dlMesth!9%=BNT5GJx)Ktg-8Xw+Cr#>m86jc;_pJVBnAjn$;jg`Yk%4hrqKH=0;AnDN@Pj1~zsdcRdl_ByZUkC)v z9Ibws!Hew+bS)3K$e5G(7LvTbl9ASE*8HNty`CFA@KgsfPLFRdOAZ2RK_x{slb}vt z*OfC?CK!Xy11FW0lhxH?@u}zyACx!#?wZH`zTdGgB`R>XZ;H#94t;nBnL^MGLB)AH zsuEk|N$M;eGR!hg0h;BG#tY#3Hb+m#Glqz0lIMc2h#1Qi7R6*87MWl>#LE!^Gz73X z2_hgYrRF27o~=9Pu2FdJ{|dz6TXMy3V--Mqb1qZVF~Y~p#OEw0@G=KB{cGWpL5nqF z0Q-;}5r6W6(rR?@h5&C>S^g{KA}q0RhwH+AGCu4+Nfr1%8pY-M=nb^K~9jy)VA1g4_Ggxmb zsQ^E9q6Qge|48Arr-iL>ey*)2sltcaJ02*h%qVt)=YhLB98vED?*B}s&z;zweN zm|&oiL)hOcU9gfX(%p<9NA+v@h6bq$^K%n$iw@ppU`I2edA^|AuXB1W9yd)86~9H(5z0M9h2oU2DavA9Lw(r-*Y(wwtqom!a6Voh&NXmf zYP-%j7Pv7K_*ivVyFW&sqQ+1wL49TnQDRV`x}K}FPkSA$(nBek?NzaZIx?1xa4ZNd z9}{f3Z`;?(N&_7prfK}X+_V57CvXhRrEvXibte2hF)Ss9>$(14KqB_mllkVTSY50t zwW%q|;vch)>(hCbIOXF|S=DD(WXp@m*8iO1vW2DdlQ4DTDhb0CH~wQvR%yYj>G3v` z!?zQM9C93~#X|RYTBe`X5)#A5krmR7LFdsr_fc1-SIP)*#>rQq9!vT;z7s7-aQyOY zij>*OHWnnB6Fo{m@P~iKho3Dd*-aeITx({&6rl4xJ{fOeoK2?ju5j8K?xQK06*@n8 zATZ@U$xzFy^;nxVD=p^1B^{8hGVefT2uP*+tT+#KJu&A}oHR5b9RX2P>X=17A4s12 z*2-+pT6~o?W02lVN^p%YZkUECaJv@`gn~3}kPPxL2KV?yIj>sCpJ~b$C+R{*;yjP{9wXcb>j&WvAp1~*lBSB#Z7ZCNU!+5)2J{hQ zm1PgdvVTS!FgoHJ{Y+J1`AtKiPv)mgAMni))LaP~%}jYm=l z7Im-S2vMLBJo>#=u2b2JY9jk2OXk9*6o#zSo0k`#Fb36jYw9X^pAOj>-MDy6Cii7; z!SX_9qS=wlYNu=Cm+A-y+g*zp^X-p^+X@rZ{--`OoFpUF5xA|)-21PUq#{x^`s*Ts z?*F0fTnZGmBlobe>=LVgfcgvD>NE4~yXUJ2J42gZz8sa9#?!P^+|TuK1v$Y!wPh_J zm&gwnWa@Y@%?-LN72k_ZAOgi0R2P-+$3oGR=umlKz;64Y1#%=vPleKl-+bq!cWD-#Pmmr$psgBjYBo(WPj;@4onQ@0JcNR zAzduXz2o`()IA>eSB1;6W*wuUJ$+EyQONU6D^2Z@m*eK!Z&)>lqE4nRn53?c<6C@$ z#zW)L9ZY^p+ikS02*e9EJwcg+Kb+C!Ei?i1R?Pzyf#1|O@c6A9M!w{4_EvjC?Lw|w z6)W$_wY13m9~mL^|1)XPNk{--pR5R_y7O;G1DWi+@m2Y@%-BSF?TrU)gnQAT`UHif z!_fRzm<-O)SYN41ACrHcV#P2ys#{}E7eEYj9cD`WAo^1=ET|N?u)4%jlX2z$C4y@Y z30$eIA1Ck$OS8R-XnwBt(+C^I)75FPRj7s6?6226&^VEaB*7DoN!P4$QOr}3_x7;U z(1>kJmLF2E_HeS0G^7A>Jw*|#bHvCLaPEH17yR$pK zU5UjEJ|BK3gNJ7Dv~j(nfT=2J9O~p~)!}$7q~SY~+||`(wq|=K%ueTh zZ;@xMEnnWjap=y4*I*&^)FZ4Y7`Nm_lZW5yuE#Cf#=dgF+UaY!P`%m zTYBV{4)yvdhn6xDGDTce#X+LEXAfLf55nf>^v_KOj`tawXz zJom+{k8p>@(zkEdQf@GjQDI*fzNJ>{@h5Tl6j6&bG94s>cp`hQo89IkZwJ@{)ms9m z8}LK|EI$Cz1~6mlU7&5`x6PN(m|x(t_TN3A1S{GX`j6aC1^JvB1Gf)Gcfa*sqgSKu zKa-`zS5XAHRjr~YXgWM-26E5)50Hy{%Ing<$tvasN=Sam`sj{_5)xosAl(#E)F*tx zZU^2H@Zc|doX}FKUaR2!Edra^r5JAKhV1(2Vi~@)mKMN7iEOOw7EeA3G_~(2f$J$! zMTMru;Qrv6lKjY1lj~~da$2!D512s84RI_a;HYk+=|J1P@Q&a0G~V<8TtIZW!tte! zH4h@uL1viBC)AAqxo`01dH-2G6D1+u%U%|2mCV0Brq`o>aLbVZPrp|HTcz~x59O)F zYIA-o0=dS@AOTFR2enzG|7Z_lB>gjinObl2AKXD+u7B|RqcP0C@)713K({6j}H__@JQgd+Ns>9qqr1?72pE}L=JwZ(RZ~rSOjQkdHxv!{8pz&DF}D6by@#> z6vKxViGJ-PrfeMgHE>dzeNM}-N}xHUqO!TDWKc=AU~+RQEIxM0qS4YWg3vyYpCHGR zK+z`}^KEN}0>_Z=KSnnO_xAfRP*8+9x_wU!1Z7ATf1ujPfnB7e|N7P3TQ%70KSO}; zpv9L4W{Q#A1S|C}i1ZeC{_ej98)v~@PJn=O>g2RwHgq5zSU7J@`KGX+ypI^rW5fMw zTO*gP*C?Xc&r@284NY3&D7A^FoU1B$u~i+PkW)s!Uye(KGYB>6lN@M8#zg41NlazJ zG*Ww@cSy;7K|1;gQ{>^YjQRy(NBq0L=L?N$&^G~BePo^zyV>mg;I6c(&grHJeO$JL zP^0x1i+Fa<=MKlDCxa#4#*Ho21dzb}cv*EDgHDF!_0;c$E*E;g3o)48mQJor)#>Hn zWY49;h#(j+(Sb?|Gg@;qo?mE6-W7z%4h28^%3P3 zx2rz*ecmu9_q`nU21)YBWW%AWKwOBmMTymfy0vMZIY@LQ54{&5(3I>k_%Y61tM>qfJ%$yiJ%D>@?`zG>HNfRQU_#0 zYf1f}V-4CT=RNYKn;0g{=)rndL_()YmiBkugs4;&D4Tep|4owwuh9OW`_q>flUDZD z2*XKbh+>M1z{wyWqqlplX{^G2f$zb1Lu~l5uAhi+WF!&MXDwev(lGm`+g-|T>g9o= zgXwPei%_!iEhusE83eJ%vXI4`bAe#4`p^+ImisXkPUEYzK}yFP-p#DB%yt7%Y{d2t z?@DK6F#sS62@T<`vS6{%@)i)#ZgNQ3*`4*+R|7y6ezg|i3f2LK8%%VrJB#!mouN7V`AjKpD%=~%CzUk7n1=!rb!K#*`>njdy8y}#^Ig~Z?;1mF0Xl8 z{YXJ1fnuohYOlUhF9EN!jlqSGcqy`pXA|qGQ7CgF=zYHleVMRQ(5>LH&U)nn++0P3>zg^6=@aLMkwvG> zh~z0IUtE>E&s|?UYhFy=1ykKocC#ipCOCl6<9Ht;@9;3;-4+&V}}@r+pOJ6iOHny zRkXr`7(0x=(r<=fGI8Ti|n>CV@xiFR6q*HXSfgs6}Ge(AhG;ZG9wjlYJv2B48 ztt~qI87?Y~_$j_1)l3iLR`R$^xPrg;bIuF6u(YYf@~RuSH^R#45NR^wdr9flm0Xvm zv1@!VW0xDz1NAwL6}&UOB5ZBm86Epc9}y&uW@(o8FRgl4m8Mhvy$JP;tTKQjYIt)ba6ri4*N^)psA0W+4fB1!t(6xL=KDZ zKQey$NWKF2ZJ_dJnahj-D3qvea$b=AS}^M)IK2aOaQi2-e~1q`FR0&zLssfG*#b;x zgDRW8=rU11+)fGq@%(mBH<4ng>OXP!EmjJq?70{gU{zS1bDg}rOa^D1g5=U!DpNCB zOUXbVySF;!2EZs2YD8SfFvtN^T(L%n_@C=jnrXBN>)EYJuBqb-aP3(>* ziU5Q@={9$9Cf8quB^k+~^|IE$M&48O;WxN0Ox5d4@ za=vG5$X8O|vy*STJ3@`r+aSm_01n;RpF~D-pUQWq5$_A4d#!yjdU34s@mYAXptr?x zrtogED*PC1Oear?B=TP-@8jybrEH4 zr%QY}W((*5L&N65c|27r^^`D{4<0pf?Bq&1}U( zz;0~>*&$%eopS&XLr-gQEa0Qx-20wcx||j8Ff%8T>{Wiq1K!oB^K*lfwXKoduE@fW zO%l#t>aB`XISt+K803pNHN=-l_2LtPWPg(okE1+z+mew)Gvio;g8moIdee&$j@Ra0 zXu0|EJ5#eNpjZRk&yFDp`=z@-B^x8`kTphDH6wM7^GBr|327|PdFXq>;eRSRWL_X} zI8d~jsP|$zXbXkJI;5D*OiJrFRl48XGtYDBLe4eQ}|(9wrsVP&Qfd0s~N~=NP!TyY(8DrSj^_ zDNNd3raIJtn3Sc`2uDiPg)BN@O{;&KZK9fyURpIK0Okk^i^_q9(1-pvmF=?A6Rx~fmI~-nBni^ zoYVElOv+8xd+J@an^rZw>TDPHgJ(-ULr23ZIg*LS1ecljTX+IdT=NAAibEND-IV6v#1jyl`8q@d3fC?Zor`)85(?#|YK z7m0qKoyfM5f;3>pT+F>gh!{}GH(N^~{UI(hCwHBrZ=E*?WvX`jx-m#tQ}+|xazg%; zrdL7BMFTY*1PY)AZonxqvNAJCI**50?H&p_n241!dJG4YLxG1udtMndUw^Y&Y1uC?(9l2axMJO@__ZRR#o=CKS zc&`KOCYtNuD!G_=G|zvmJv?Df`>(Z!mFohiy}F7kZCdx^C=LAJLZ&KxddD+9CAOn3 z%pG6a&Hl+UXgdk*?MgVH6R{};8w*4s=l6qxpJc&QK_ON%I`J7RCQiiK-4w)8g5A!! z&zc?kj}x^aSxSjI^ocG*k6thW$%GNX;}~{6=6n&QOUd#I0vA>$cD_@n=| zr>#aao%TgG&??i$Dr@c$dhz6P;xlbpB9vF*X zb^88mEHX*8GrDKJpnMc%Z(dx#4)ZJl7_tg^YWZqYCxaOp1)7&-0RL}K*_^wJ6c{RE zK)K9#|3qMjjRr04eJ()o0hQy+0|`G`E6UV#=EF9H+^Naw+@#<(kF3Ds%%6PQ+V4xe zaLoB8fPu!WLC_rVf}H}oFn9qVz^Ir6BYdz?_cA1x3w!!XWnH#bu! z&POwh5>JbrxJQ6G@fA7bn{^4M!&40uqHlSGgSN$CkEg@i?t&2-zRKlp(NkgpM!&;@ z86GlMjaY519mT%8&#)=sFTTV4`c}|;W`1mY!L;6aV~M7+k8H&A#^~>MPCk5)1rwI) zg)G=^uMLi^-Z9_b-Wa9tmH^3O>P?Z@`4j{)w>%rfviQm9tDasuEgCP&p_{SNu+gw# z_^L-sK0A^xh_!&H95tVAl~^`kZ3+N9n>tfODQ zQtiwclyX7>Z3VUGaAnL#%pC^Y5`$em#nrnbXpdU28253&m45$FJ6 z2Tfm(HHbb1{mv=Mp@oJ9VoSO17N`hM9hnmkX)Y}NB=GkE3{bzdH2Sib4kQ=@E)Kp5 zD&^7|EJ}CQ!2c*iv%PZG$1g{4E5{EGMGXbK*r+4GXM=hH&o_9XT;kvbY(^qzPaf^S zoh>9fGV4N(4}~HH4(5lwXMOck8wINbJnbl=b+o(qomJO#6j_L88zI~T) zj0`0L0pu0nEA%m#vKMsVjS@EM-6zZ{eLpeO37{};{ox(dT|5Z;jVJUkkTk&hN4QGfpT5B+0mQ1|;EX6-*7QquxH5OCVt{6&D4Y{`YFJ9^Rn9d!LpBm@on zCP$-|_*MXRv@;p+{U-!QR_5O`-_3madBds@s#Z|``8NiS@c8}5q(@UV1wYm&bS?(E z?yL-QuaNC@b#9#ut^s`RAUKz!1sn?caRjdn-3t#o`4uppziIVn^3-JhW4tW0!6BvD zu_`+Q29PT6V&8qVWi5%l{uqz&J_(`pZ*q7m^KZ=nMZ!oaM|eI5eiq)}7x^Cw`y9`i z;|4Bh?SWWb1K8+0-Kzi_*3E;Xcx_RsC@5IO1T~jx_W`hTx>E0wV#gy`t$~Jy#$5dD z`a}Y+*YUcO?$vZ9z-j`$G5X^q)NAnm01^ok{2RR+V2uHP;bCXU2TRT8C%eH%MU0lu z_4Z))GW36_o1O|F)USSV1wbUelrb_Uwni~7gZ~Ff4ITXyAgPzgkpSc7&GIjkzn=<# zGnT-y2EF$70}Hr(DDkAD{15g9>3TSFLv$GTLfA$BnkA$E7>omv^6&9?fJYA}G!PHs zXg+(s@74_Z1F*4G0PI!Gi^h)bZ)vrX8Ble-yYv9311)X6w5nQXe}cjLqTcdUH#ju( zE??Eb6TU*jCWkQ6iJ|uI<_m`~28|LfR`xe}stzDz_kGF5j_$AL3_!KnfE`tN3a!tT z$sdJTvzLHIdZm{F{s#!)~~fD&~~ z-uNScCxeF8t<0DrnyZ$-&F|_se;hB;Shu>?!@Z6=q{MjrXFCsqE*hl&^qI>0Pp;+yLuHS5Hd58-fe1w%CeQDo$^h-(!HngEkzKZo5KP*_ z53KZ>-&<*Ei9N`9%1!pQ=k6Qo8%h13v%7;#mI|tAH~k=JTMcA|{oT)ypPsq&-0?i| zJW(z2{XP9XATyGyL@sHKK4+ z=5HSJ-x>RVWKsWB9Q?mJ>wlkZ|NoXA|6UIMzk2ZhB0ZGc-DBK0{!MyCU8=vh7=q7z z9dt19%{0EPn^pGLy(WdqJ4V%Jq2P25eNZ_@`V?Wg1P*?wxp|L(uHr`sxM)CP|^Qncr zgy}Nq1%Vtq`4@yC;GR&g{pW4%>6{599B3d?VPkCuyjK7&*#)`)J6`1RB1cgyLG^!h ze9?R60Vs5UO3GJr5ZhyXvz7j}dGmIi$gr?sN^%YnN&^vAx8E{-6Xm}>{dE3s=ak;D zN)EH0iiw%a#<%=B0!=4kHEIsVizGu~Q5Vg>E(%gREGQ_u#N{d)HXz@q9NK1yLTv8s_*}p`sH>z9B;fK~MoS?}E|*)ugK@G*FnrCEhOb-j<*z-Yd{@ z!6@K>!F)M$F}m+v4S>?7i;&7Uvy&c!a?pToyWk|z|6N?*&-TZgeYTJE63j0PX1ePw zQ`5bRXSBZEO}O$8g=u4?E72Zq3h}>Mw_0&eNuWbv%OkvX#+!7n>OvcChkbgp{oeK` z>HO++Ws}*=_Jx1r)o>(eKmW#j=Unml{otWJ|4%kHw&p7f^_4jv{lrSEx#F{Z9$SN* z1Lnb4X4k8QmJDk9n>@A-$?dNuO!x(?EKHko9#<^dfm<$iPgy}-8dY%Ks0O2$S(!MZ-q1%E7LG6A}N z?i;hNzfzlXXwp+~&3~Ko{fbw<%MZxj${<+0mpV79*7_#`RG zKT~P7+f^cTcXb9CdU6N!tmdZ8eNA%? znsj%C6VGMnn&BH6ky;iA3e;P}k7?v_qwmD=Rsqm9Z8SSm-eZ41W8Zvw?#No1_r-Yl3O2K&ZWmC2Oq*v9C#)PV^ z!{HS}6H`-rM?wG=BSquTtc53)J z1H0xU-ADW6sTm%(aaQYJjaL6E#7^27+V{O$M|QX%gm$cEO;$hkkTIX=9BH({uXL_D z$`SD7qtgKfGfeFhL;<5||g&DRd9>0yNSFjUTq>yh}IF zVj4Xht(BS$=+=Ky6vzW=D#lOAe|QrzPX`tx;jCgdq0-%<<#P{y7oZ9TI5GRxV?MX+ zZl~}Mo_%X0R!0=#X=?_8iusvtA z9CL9w0}qf`v*o_eEym_hlrWgec$o?spk|qb)|L$34G;9(nOC;Am+|z3gxe&84NzM8 zg6wYXT38N;c>c?klJiHGp#%7wl`AAlaPoPceR5Yi|fRG zJO&?Y&xFlr+=06}EG%!m`g#8m>jPxJ=g!XV?Nl!dYm@PMNlkJw(B)HB=wyC03;u<} zVm$o}?SokPW^j1$Rp_+)7w&QDgBK}RJchBTNr`#j(>t#{JeQgr6GPy-Fi~LJ#OJa7 zUR{W3h);Bvn3xzeuq+Wg`}Xy#FvjkdW-90Ib|q}G$s&s8ftg*EPI6Kt)W($F7XuSJ zI4Epwys_YLblKnuYaaVfXVG*O6pLJMVMiM*vTW#dzXXwHDU!^^9= zEuEUpA7#lIJ}|opf2db$+%GHUHB4J$k}JMtv#PAys8&@?0aIw-Mod4l9FObyS_K zNVn$G42MC7)Ri@!A)aIe&yd^prdx1u6zS?tYI&Q7vjel`Yfr(YaH8U|v9U$wt|>{{ zWhV!n^_a>rhB$?*a?_4T-YTffB(qLk>r94hm$S8_W90z7W&#^*uT9b-vD(TTj(lUM zkeC#(tM1~e&~dD@}vGVzUMM6a?J7)meOUQZGFk;ExWIvPo<6b`3}+ zXngV7^^+9~AD?d0iHv4Vh{cJ$goJ<2V3I9jnSR!BSuZFdL1*iA^X@HLB~4A>=05 z6p|=>=m(Z@LkX+Z5+{Fn#cGCy;GLrM!HOA9i@Y81G;j!vKXsW1n+<`zfRmU|4Bo zGSt*UN5s6cxcCEfDCM<^&P@9{IU|bedtVg2NC!U-?4yFok|@RG_$M6j$r&FJ8CsU` z!N*)y3e#mCh8h!zXE;y79`PyH$i+_-B{U=4zO?7a54Ih@=Ht(K?V05Ya9x9}xSO*5 z{qrnNCKwaLACgcA89?&J_*@bzH=pW8FTZhwa~7z<@_Gg% zTGEG(m)bj6^H}JN#S~0d%35;Sn?J09T6}Q0@%Yln>3EBDvh(r*87up*mJD{#kL;9l zLK;(uqqK+y1JeA~;N;}iXvT7Ye>f`(Nc*a71B_tGkhYBF{!g%qqo0@ke8Wc=GmI>a z8eeS%xin_xobv0Hk5aF^@+^CtqRP^w0=@ehrK-NYqV`9@;-R5kLzyFJZ{PlLg8tK9kioeSCB3*YvUPph08dI!C_E*6MVNkw;?=@DFjE6Z+GBq;Z{GMl0TuVz!)-$y=n)7MB8r9`7 zdwX+D$<5r^76oD8%k>_Iymov0_3JyFLPDlX>l2UbHH(aF)k~B$5p&WrE_7-mBhw;+ z7isBmjRH&W)$UzEkyit=yhMX+zv`u(Hj|Khd(?`lY%i=RFR*1UR+>I}O*1QAez5AQ zi_T#K&(aWwofz5vx-2{{;3>w*$;pMNf3l#5SL^o(Yn@v2xCucACX>DWDVOaz!|FGO z_4c$G8BwvPl4CAN4QK`D_LVN{WTWWJQPGF6u){3|v&{RZza}0(lTdt~m9;ea|221| z(QLKtx~)o-&yPYIX}*yv)1{Qm6f%3_TJBZU-xxg&-3g%2ldPN9BPk8X9{>}HX3T_ zYDt(xhTgowA+^;}GJH^lz#|L%7L19wr3z3QSG;!c%noDyF$p>(6y@PEC@wEA??(Ds ze%;3KzInz2<7a${sj13*ANDsFhe*bf-tOhS-QAbi9_*X~+UDne@4cGd|- zhKADOb(tE{$RT?}t~Jg9()W+Te_Z7KN35O2JmG7YJn7fzSBFYsy$tSyx-SaxG-D)n9&;RUqb{FGLq*DtXF- zPws72P5^_1gak>chE*GEk^;s=@G^6$a&T4+GzF)A{aTxqrJhNE$K2GGG;Jn!RoI5M znVLe688q}WLE~&lT=!4?Yat^g<~{MU;b5IOaIHr_{fb*&84?%}0I<@P>S*QuEEZSK z{`&?5()8+mXdNY3?#XAjO-jN&T*ezMrzf@?b>r{hJ7-J|QC5!!zQx$sr2_3+pIi-`P+JVd2JS%@WuGg^47r@STCT?p(zc}GiQtwgby9VWK-wB?eveO?!r;#@IVO##VhY&gVwae2AtC9suY zqzLF(kzt`+Gt)$w?M-C=@aWn#_<1pt&hGQ+dKcOxt!kEC;PeY`l^xCEqH=n}OrY!8 zxt08@Tj;=wM%vKrt)CK`Mc{Nr`FH-dZuIHm$?}7(_#)K| z3$^kyiGj4f%9Qqew^*kMxl)^zsVeA34N0C+SPvZNOCfDiD7XLlm-1qITNK zkAkeZxycnRSi=*k)fvlX^YLGbagtX0;@CKe<&_mH6ujzFyj`v?%dM9qg^u`^+6?F6 z-@xAxiNp+xX}e2Tt^j1_1C-DJ>SQ=QXaDwlnY_c}%)BUWo6+JN*^OQ8;Jbv&r#{BL zPAfnl#88iKoK;UtKB$mFpUlq1BKoJyAw459#ZbYQCTJkXb%#j59rP#<2u)t_&gD#I zZdEx42hF(>9PcP=UxeWWmo*%ae{Vs_1|w~C*5`)ms|1Funy1pLb2i$u_%7-S8wPAG zI*wO%RJOm&t+q~-P;eLbot?cN+L>|y^W##5&2Ppcg815?&z(o}a${n${LffeOkZ&m z;$}V@Vif|^r3gSZOSJ{@nKyZ;A!OUlW1c$(eFB*;Rl!V@ohR!NrkvhiCcL;+j_+I47dHID zWx>FiDpbmCYsa-&Ty1anko`IJG4I*;U%V_N-c9=Mk&i^y+}wHsW{O|QB+3*4wdULv zHuLETH%yzTvKOVupwDcse{)skJCN{Jo-+G)+d=?7b##>rIN?BfIO)VX@a0(6mY=#O zGc&iG?qA+M`+KW5ZwW!3Pc-W_+q3duiW`I+qceBN(&c6rets)*y~f1}C*LEojkq*V zZFmljj`m{w@6x8by2>N!#{v2iak3c{bS55hH;6CfX=JR*NRhK$Ds`{qm)iQT>gsAT z$JkXe?>B&;pm>B?4e|Gu&-}&@No=2AWv|k#E&d>=PPG-CM&u z$=|Xp{NOU~zr?(~$D_c-mB)4y_F?r$R#r^8;;MNMAdp@LG(EZ{<-bdem)Spi@uC_U zQh+9{5;@p@Th+r#45=APN)vvyLgqTEV})AM&=dOYlVfa}rK)A%RB=Pb4dlDk1z|9a z;eu$u!TZfhmR7-z{xqq-#%5=KLB3>y=Nty)`MPQ|b_=Y$DC4#K%F3P7MWexC(LW%P z7WA5&uuQ&-&q$*`(jKvKaIvA}EIr)-j1Obx$uC0)gWl8IPCFXko0x0s9%6VB=Dj|! zejHBou-C>$T8i8!E+=RTB+JOYYx}0Y63zSGz5M8H%PbH~z$^)Jy)#Jzb(2>G_Q-2~ zSfCde=m_Wg;hIvww{PE@(H06118{YYZ&@T*$NzZIk&+SN9Whx~`?~%EeLW!TJkx=k)w(D^gBf54eyU{4&D%WZvG$)mAsClMaqxyBkC~Kf*k7rqkt9%>sStoM;^| zns&s!YHjcaI-1ZI0He;7D~hVn?7Y18Vr;tCGn(|%j%_x+dfY^4JpIb6m~gxd^4cnp zmR`wE`KoSVK|_7s5S3nWovfD^)DXvFo~wH1QCj%5RQWNzy~=@exJs5H628R66hE&) zZ)qXGCnjDeyU|AriC70CgU_AEaKd`9sniS+S`5(H0M-v z#qG?}PfMi8N~96`YT8c{Fw-^mfjavl2T22?lM9DY5I2&%ukVNJ;hI~#iXQj1EG(J| zVi~xZBTk6R83NnG8S&ga0Wg2P1!nCNwN5FTmQAI z>pJd)mh`P|_h<0avZ~4-Vw9<-(U!evy|v@xAaxDFt8{l6c%S>~?LiSETUFhq#lgx5EnJNErnY^-tywS>w-tr&ZI zU>WIDUCfR*(AXMCEeNL*2-K(MUS4cwV6ZVeC+O0PE1X2{o34)H45!V5`0@)1L>t{_ zDWsY=-CF=5g#kKUMC5hix3LnBr_)k)3a?c|bndu2(nn9QK%i~~dn{6)#Gsy^3)ghN zzcjy5j^^otyn0ww{C3k))^WPB-njxKmRPqRMO{}D!IBqGZBPtmIvRi=E}8UZW@7p$ zL(W73at#SLB&T@hbZY6I4G*oahmcq+?O|{N57Llwg-7+UFO>`%scAjG%Uj|;GLWH~ z(b1u;XKnl`icNyX(ek|o3WU2+%4;Q;NWP0H92|LhOAw^9sr8vibKT1Y~< zi#1n~YZ}>C2$NO6c7=&aj>e_T@2)MFSUZthlg_nvW5Gm>ySjk?2CbSH#!}(5g0Yfv za0uCXyV+=JxQMu9G*AjEN@^ce&uVSe&X-J+w&0-GE6A6=s`v%?5%5seq-q=Q$&m;s z%bBwwnQ>$?-}cV(64d?ekyv0+brw6mLGD=-17KE8n>w2Os zebVD=>`#7nL~*?W519iaZueQ5@$%)K6t>zZg0X|Ll<#d zHex{BeGT)*%JbU9}m|Ood!j4?~ zPZ0shF{!CeTh#s(Hg*=Y&>^g$QkgM`Pe<9;Y+}LtT*yDnbS3dq`)5MWpTnA#-;t@y z+zAoA5e_@@rS5GTr@X#%1H>k6{VVBurEF!HNL*#1rG&(8#Bs`=Yw6l_1bh@Wl|Q$j z%_RAASpJbuhvTbo27l(?vQ)KwSA z3~CdJypPA@`?+Vz7R^X@9ktwk^Kf(RM2)|$u+!3Zc0Xtd3?f_4i$%HyemIIIEwq&= zGx#0VxGrUvArN_a#`}1o0F~)FBRz125@f&K+E;5iC+_d&jE088?c0H=T!1mNg7ecm ze+c6ch{3T}5ReKiYi}GJzUSud9v^%`kIsb>uQ1N^8^XfMy7naD##(ZTJDOJehE0BZ z3K{-~RAuT&sa|=hVbNa!oL|2%|2T9lJ+>5aPh4FkHpA2JqIbJv5jd;3JyvQT6Sy#tW#9q&-7b)pN7EMwDIssESHiN^n%W#pY2c3tTtD@yskbo6 z1irEH!gz#p)f&)|-t;tA{RwGiZpHgY+pp;ry9%$ldV{N@8U(Y=1kyix1&7@|6Gpi zR3p%tG=Svw4%3650jJ+RKBycb*6jkaO+q7$9ksmDN%7H~U3`Od-31Iq{j;I>!Y=_V z?XT;t7p$7RY@RG)5d9f5^?qtTT{<7O$7rqb70ld5!Mmy}PSiGJi z%xY>5R6sJBY$L4S-OpN*&E3qztm3)nKRl%c&=crXBaIekDmUV(s(zICk+}R)SL4ps z_ImU33KIuM_WJacqJZi{JPMPehd4$;5!jSeerzLb?Ql4GaKzmsFp!f623RJFjBp)5 zT5cN?sfIa(BKs6JPUu+m(s1(jJO%?mEJP_zgmcvilkq#1@qU1(ybIgDT3=iW`_1>S;9>6BCD!SWl)zE9` zxAhcxb%rOOB&@h3PF!(*OF7@Qs`A__^z8c~3#;k$w-N;n?_U+EsqXn`~#5RDT#@){~!ej%`ARrvc z&ie5F!H%M!MsUUT*5;Vh*XEvsy37KA2Y18Zs0{S#!CIp5iDs>hMbA*OfmEk{+7>Xs zc9f@pmu3if5fOO_9pSLC(9;7ODlc$)EXlj-avd)O`l375xQko0`h5a)x7@bw64_A9 zMWEAIWy`vJtXm3*rYD| z%Ms|bf4~_9PzV%0JX}_~h06!ey_gFm{`}sYVQDDbx$)Ue-j#qG9;k*opdPv!88Lb& z$pJQ0!~@%%g*ROyf855hs~XyX5%)jaj>pQh6`}Y)KcCXs4EeV&dGWs%Z2qx>fK`!C v{{8X4bS{Avi~q-m{eQcjYya_W>05ykzhfAEX>8B`Jg;Y3`kJLr-UR&%@c%7l From 166dd7bccaa054a5e16711c0b5845f7086c9b157 Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Thu, 13 Aug 2026 10:16:50 +0000 Subject: [PATCH 08/15] docs(wan): keep benchmark artifacts local --- examples/wan_video/README.md | 2 -- .../assets/wan21_fp8_sol_h100_benchmark.png | Bin 42973 -> 0 bytes 2 files changed, 2 deletions(-) delete mode 100644 examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 13ceb2f..a7d6daf 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -245,8 +245,6 @@ same generation interval. | tf-kernel FP8 | Dense | 0.879 | 14.855 | | tf-kernel FP8 | Sol-FP8 (H100 CuTe BF16 QKV) | 1.146 | 15.730 | -![Wan FP8-GEMM Sol-Attn H100 benchmark](assets/wan21_fp8_sol_h100_benchmark.png) - The benchmark prompt is: > Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely diff --git a/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png b/examples/wan_video/assets/wan21_fp8_sol_h100_benchmark.png deleted file mode 100644 index 64c49be8b4cfd247ef47d3c8d93a574224112e94..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 42973 zcmc$`cTm$&7cGjSf+B)S5fBiS-UI}cs(>K9gLDCcR4y9i;aTkrI$z zLI}NuB=6w&&7HY--hcPLcV{voCck#h-h1t}_SwfE6(t$sThzDk@bHLb-%6?B;a&TM zhj(>^@GAI*#Ek9~4=*oUR_e8e+sroH)lFR!zk4q_dy~%T6=jxh#q3oRRFjEa1o5~0 zj7R&Af8gIG|D2Iem`_O9H}4rg8K%lH4YKV=-EkGqW?>LaXh2;oo9S5+eWui&ZlQN?toYO#>1QcdOn5 zc|M5{EW18m>n!{(Q^Rt7s#No!<16iO!ypB%hlty0tvUJtwnc6rT+Kb?n*PG3D-9D+ zZ8xZ(h>uZDOn9(Fw$185{d4ePlVGcI4rNoTzXg*N2HZ4%z(;w67bA>iv|<#1zY>Q&%@(z5&lr z4lhk3-Xi~S;^5OU)|nngE%0JJN=7+_?&S_KoK@VX!R<2+W+>%u*yQ3pTiTT8H7t5F zTtPvhs-|>hd*x5L0pc|VUE5ICyfJAsSq==D|M{y5SG?f_Z5dN`Z!KZ!dU-OG+rG zh#g>**&AFpPurv3Xlccj9cNHhibSicVNXWe24`IhCM}U2J^35pC9hqI`LzUFnsDFndC9 zhn_~jv_dY2t$|tcquSWWXfzgHqb@ikdOusU&`1mO(thiiQGABOXAWS4ue&+d zCFtQ$<+djh675cTEM!0WVWo%?XVmZZjRZ5@Vb+2ILEE`LX3IDqnXuFg1Pa2wwpnxn zU%hbyV}vMZXx+P?XS}?$uj`EnGrRiXiQRD0PN_-JwZ~aMTX37ukG3oOldYV>F7fd3 z_w(IG`i)-I&+HKm^bpRyNqH?DaWETSUY$lgKeEVowMaLVS;%4kS}#atnF^_i9>Fb{ ztX4w8?X;3S2!WH!+@`Iy>~y6Ze(3Imj>2t9$~RwBRiu{8f3$+xv9)z6hJ@0%dxp{S zTldDN)6Ja?3)?AiVeh01H;Z3jotto7ArA|of8Yq8mz3jkuig!y#GFR>47}uGExSuG zbvs+JRxs3D6KcwF@Kv7EB}4)24u^AWL4fQyi|9uB;E{@ zQc&AU%w=#66*ExkkfO-LFEap1>)#!Zb2H&E9L8 zG27r4uj9VVi`H+H<}^%GPZrH+7wA(rXOx5vxTV#AaEU`{L)8L}Sy$v_>R7e9rxijjr{%MPLI}-7Q5Fp$Cnx8RUy+uY zsdn>s#*|pIpqZwhK5YxTNxAm5`ZV#|E42?z;E{~0kv1wK~6sVRMAmXyVlejoIx!rYEb$Ou0a!mKaYdagY%KkM{ zbyQYXc8y}~IF=pe08~uD|y4G`Y^z+<0*sG-=ky;i_IjNyDTwsO`ZWstz!GR${D2V;w46#^fqc*m7V!D?HM;;v&oi~`^c)D>D8NJL=G z3sY5f*lLFIuo*8rJFtB!Dr$gA(~GB`>|5&1*rcCI{plK+NBYxVQW-8qd5@anK1~so zSAIfKd|(@WusF=_e5uIpesq)*v-F#8sW;uH*Y&ZB>za1TsV_X&4zC9Ea!MFf*{sip ziyJ*8r_B=P;o=hG;=*dF>;j5FEqFN7fJ9MhUlF?*?&iIeJi{l>$%)z6QmWFD6uk6WJUTtZ}^7Wq88oscA$cK$eLB)hSyJqV9D_x((39!You?elUE*kqvhry6^ zRTULpD5S-3b!x*2y#nOKH+$t1d=noX+p83W5VS)m7Hn;23wBQTvX)id9Rxv}?R%PH`MQW4=VTPu5I+7SoU5ewcu8D{n9o%x2c1}zVd^LhLL zXUYwnt}^_Inn%jZ_4U8H3%{$I9TmBUbCF$2KJw5-#B|2|ybXkM?WJGk8rO(zo^#%O zy|NXV#32%%rF}BgIj4z~*}%jf%rxF3qbU;iD^?6c_}CPe?CTaK6j4@Y+=6iUpfTJ{ z)N6yi%lKX-;`QDZ2j~2gMHtts+kcu+>kB^Izw6DB-yuFaM|nFj0Own{jDAvWYz&k) z9f-S?wOW+~>s$d$592bD%4p={!MrQ+1+^H4)I|(oHYz(|cJs&k#Srt8HMHPW?7KRB z$18!-rlxB%#JCNcU)K=FMX*w%L7o(6?AK?E{LUS<1Q8g_KkdqRDk|fyvXJ}XsCgu2F=NNU_o^EW~t}lX~Bi7^J&BtqO~T}%Eo(ZYY~S(kD4AVpH$gZzQR&+ z4}0Mx;jE(dE?eW3R&QSD5aHmSuBbxcN1<*t(VuadS@p(lgbd$_cZ}BLj5jiTaM+x$=T{j^|oDLy&B*hq}X^l24yLWedJ>JaXI9=C-e+3(j6yVO+pu3!BVs zEYc4rBez$~&CUPhY95Gdals5srb$}N%feYq4PsKF_Ce4&uM6aBo8iI~G3SR>mDQ2` ztF~PH^GQOZCp|k;oXJfiw?6L_LuE+lt!+)$wpU(ocZ;Uv<}x9)%65y5U>zN`GB71@ zs+Y_1d3ZYKbiD!LD7Y6buDA*ETAbqc)NuUVm6!X=8N~l^<2#tfQ>~(Md2WgC*NkgC(D;EEfFX^PYg1|bCREgwN7tx z)OtTv5MhU_Bpkf2F1ojT5k-?2)wy66v$}(1WdYXLVUJGsDqXLv9v)Z8*4v5FOpJTe z7}WNchtY}iTbwr(6j;o#5;sv7pY9-je9jevxeD^|?2H%9<_^wQtJk1r8s5)a4|e8w zXh(QVI34($a|9?RLxi^ZGt*e=45@!#usE_7Fdf znU%UbJAp4Rjgt%x4jc-Y8`~KCeI?vkfM>!xo&UiXhyK0h_Rbv8J?DVfLL+F+&Rpa7 zMb3iyCL06=1XnlqifMd81XyB1J{yyrQX2NT2=A9nmXyu)50-{;7P(v=Z@6iGp4`wqb;)*q^S5~y18$qDNwUd&P zV$kDNgbUsqtv>Z%pYgJtPb41D&eZ$}4Hotp)3)p7)#U)j!0G*Zrk?3$yp-_R?kNxJ zCc>m2BpM2?Q=>%oD@_ZLq!M-!4;ZARrA-a3$Q_UF#Usi`&<&cH`f5kRpE~{R1~i2J zqG0%Ixe>`De=(?yM z{{H3^QX()MCpeq&8TuI!#>nnx4zHL_Z^0z+y~hV-3a>jmwxu-!0$>JaSl2JE{``qU zd;YRL$Z;RP`MU{AXKVeroPHRE z&k?dD<2~|f5l&y+@-#!gdxQHafaYH3KwQ+vXyeZyu3j{KU^v3=WHo332Q__7zW;6` zbPXgCMFQCQ8Kk4Vn}Jb{a`UKUbH4bLm%TORI#)c+#tn%kI)NOC#NjEq8e0H+d&Ndj zLiWB-72M*f%GB5JVs4C8V6PvR9yI=?%s^|~z`NgW%uBn?qkhg4`-L`&t~%5mBSEG) z(!C8-oU_|-9Kh)|lo3e_1XJWsPwB&~7;b}MH;Rp;RF-smm^{#2=<>dyFk z`{o9z$D{Hdo+J;Ispy^2bhR=ixB^sf<9=eFY^D>(T^1!~uN7A>HIz{U?oQjnmCmGnQ%+8aPShu9 z$2&?QaGEap*A zyL8ZY!n};Q$3wL{Y;jQ~v!NlkiCR8a7Y@}k4@d6?UTN=+x_JdCq9x1G0hzMVvL@yB z(cgwA1s^Ha){eG58TpgnrOIg*6?zfli&7tBUJ;O z(~g<^9*jV>WlIdz_QfeYl%MQct8kaK=$PXgoQ}pe{OleztnCrp75Xb1{<-*e}KdJh#;@6lr!*Cj~KqgVm=nn{zr?F0f>l<0^z z!t?ms4(T24%Lr^D2%GS}R^wlHb}sHiC6sSeC(bIr+kkMs$CfrI(&XvFnHmqL6L-#) zN4>@bgkp^0UCB`2sHkX7Hcb%2AW^-f_c`x`Eo^e4Fxv^CXCrga%jzDCSqYZBs+Q%U zBcj(8N|%ck*p@x@v*7bZ&GioqG=dUcg6z06e%f5?E0lEi$5O+G{vs5qLRFLAyz|wq zG~H>mR#V$Qf9MV~$2pK7@EHtS7T_uDG-waW8h{j@Lo zd0LA7X6Sdi&yPlFv&Mi(?UaWkyl{yrDCh-IEw2A7pHT2vBtBiM^rq+jfb$nnypohr z?z1|BOqE!|>Q^1TJAhn{Qs&!n8KH%jvv0vlyg#9);vE&h6q2}=z`D~Rv5p&WLzhCh z&+o~h=B`7K&)%MykJVhisvpNlGhrcmKvyGUR|ei&boj(EixN-**QvX#?0e*dHFznw{=?mTt`R81u-WE z(-~3|tDtW~7t#SV1Opr5|*Ej%~I+kL-X zcP&ZP4h6 z@{y4Eslmj@&u^cvmXq6H1<(i7x3giRs(%(56-+HHEzPXd$Kll`2f5y!uH5@0XE_0N ziPPZ=Txq4n|T`>GMZwxFtfS;^64gE={F&Se_FhG_dnrq$%Oj%`zgFAm|_wto)f$%6Y_fkJ{{LOQAzECTq-oC`IBs~@6UjB+Z9c^;k3H#h zHQA8oJjX&wS}wl&>fH)vVSTyU%@1keL63h>H-%I#x&DY`qsIGkYbZkj)g+NA;;=PW z_oXPjy=|P2x449ZeRjIae0wt2-I!1Pei)Y@>H5`DA2ecyWIk!+Pm$r~bPVZZ23GfV ztTAO7JTNRske{b?ZF>ajDO2WA{oHZV@@TbHuiFc=BkGkBkzj3XquEm9*+Q4P%|2@h zjfhHe=|b)6;Mf_IAM^6@UEa)c?3dZPK$o=in?LCt*()a$UePfrV?~a$4&v2e zY0bY-8wJdZs<}*Ep^aN%4@dg@#+LjkB1s<~%66Q|!!Ee4&wD=jsNTD^UM9}=IT?bj zJea8a^1~6Emz}GaP+-IX0)#!DgJpkl*7I~C9k-ppYV#9n zX`m>vTg2%i9;7mqIa<0N=K;Y!fj@#K8wWL5mhMNF!f&SuxtMoP4f>ISBj17IyUW>b z&jxt$`NUHWc{dbB`fsxOP_KrWNo#c^h?1-l83nC~}Zo{TUHMskHNUs*fT1o)- z&Mvu{ahXT57FVy8t%r11Dmr{;4njtIU+}crcH-i4%@LQS!q#$buh5l@LY_6m6z#Qzg2U2-HsTad;BaI1_p0cua*AXQbm&)Juy+0mB(#8*K9Yu7@bu0 zBw9}I)D3F4xvU1CZX4iuEM(fhBWZTQg%aK>ve{2cPI9tZC^Wfm$K1^({r$V3?@p;c z^vvG;G^i2_7pn3o#q2dablaRMu~_~NJkDsiI$a_#;m;|j=N+oPRl6|y`Hlu3Yp=Gx z76F3!Tl00e(_dIoL;bt5exF^c?56TxS4tB%>(wb`s!-_LvBaKBp(H(cwqP(NPtyyfp$qUK5bDWedwAyGf;!;)$4C z{WJ`xwGG?!j7A}1F%A+n=e^spc|iH(=f~T`t4CH1B&db)=}|*I?Gh~l{U-am8A^soz299(ee-aRNv^xj=s_0A#F&*`=$$^3omlGRqCK_N5V zyHeKr-+Y_kt%e3wl=>EwfnlMu&bv}>pANHC@rA1`(qdBfhllX5XqR>~GwJChpxt0J zGth<{B|WW7jh&ZXV<#6m%#o_5Oia#Jb;Rx^$;XHloJp+Q*vYmODR?JAd0l}9z)5p(2h4KI!`dkbcIN0RyD@;!a47s}Kx^F1`X0~kJliDBQEmsB@irc0eY zM2!~gYK|OW4}YM5H%VwEP?pVKPEO^*Ls-=Z-;^Rf9o<9J^ybXq-&VcfnaYn1ii9ak z)*$oena^YjTXiw3^m2H3Y+8>*C_ifw?w$XIRoQ%N11p=s%{XKivS9{-;?}=)yU}?) z7&rLYA)mvq7RR!ZoQ|b2TTR{Gp^|>%sg~%WhZ%yC|tMM!`GRy-=li& z0CxyRXX&CxxcB1u1$tqMHKy;FGufQ-1ps&exJfV_ovn$g!*B|2$UUl(#CtU9pEo+D ziv+(lUwqD86QjyetKOY8vb(+j;~*`2D};bBvN7!hH@cSg6dKC{?HZw`O* zlWX*TZKRojWStj|XJJc|HkEc;eS9qJ?40o_BNTFM$cl8Qp}G%sK4D;xik_Pp)a~k$ zmLN3iW|JUH)Hol?R;Bl9-bTkFnok#m9qL}ceRrCk>@ui)%@#3 zLSCI^&yL>yP(7f`)F2X3yFRtWy`gdyR7l8_ahu+c<7p-w%J%fFkis74z_y_xh&#yH z!Rb*3(I?9Q6pr+ct^d@U9VajQk*Wn%X^wvf2w1qa^8QiujRnstOH3TV>EoTTp^np* zuR=qo26TdrkRqn}8#itcgrRL`nsK|mj@N7Kxb$y5^wAW?bVqcRw;KKHweUnn#a@Li^O{ugoxBQtwUM#Xk)os&sdkVE!_X%R3`a;aSgc2_Ll8L3AJslv?wEb+*yP?H!Je zzwqVi!{l;ejI=7zJ{HK{@_ z`MEUA+Te+Vk7=vRtY-m38I0+xnNsh*ZE)V74zZc%_xPNS^3gLhXMvb4u@oG(Ec$a3 z%-6di*P^ah1~;ac+uxKR3}ADqXBW~cZ{Pr!D-n7B4MTALE>k$&;dz9Z@dEajd_&wz z)bnxSDH1w=inZH&EneRH8@~s=8!I#l#RZTrmR--H8QB{VFgme}ywF&a)=3)iVqM7J zgfanAigj(BhpIgGNtA%zVBzk6$EuTwOqKa!`iI`=teW9dHjUhrt7{ zvqA?jC=Z<(_4f5e$>nfOrkW|xl^Dzy8J55T8f!jv(!0`&SgLeO3OBjNyNC)`k2cc% zDeAX3YLUrQzIAaJ29g7E{`EnjBXGF&A#53NQ7IAMc zMtLFzKfO>(lrlYTD8R_D)=BYxj*?L=OFYehw^`J&TnmMm;Y_aCSEH%6xq-2^Mi)iO z!MX28i08#d%nhrlC&l`5{0nfFUA+xKcVlY!Z(n!ajKTOrq4=ARc^1}kEv^)@!0m^CP zwAkzyEqy;D-#!JH(BNvlsy3*TsWE@MzQOh&P56W!Deh5kcM`^ZxJzEwHs`(30n&(5 z-XUUiv`58oC8>olNB8fgZMpX(Z(g=~P~~=Y6BuB>7-wW(Iy0sdPx0+AhmT1X8joJ;|01VGz z{shFcu+Qlila^9%TxE_SU=5S`TRJdFPa`9-N{2qkC8(eG%XChZtme)xmF;Z5E65;t zAOgE%MYi?|K^u9G?DJ~ZFvJ@0qGRmgo@VMLnk%Of|M}&t>>$@CgHoL2nyT?vo}~~= zy!8fT*HZlM6Z!*kiIv6zFy>RA)%sc29mwIlM!cnxhhz6`q~|K=(P|*6ud$oX{@@MQ z0lBxoy+m5fwUxqDJoGlPCR?mr&KNOy5=Uk)Id@f{oZOJHK>}ZPO+)Z;bxC#XnlfFz zZ}rTKM*UqUsFY9an_@P_5NA7So5f}1_Iq*kI3rfO)avSoyLZy1*zQVvDmdcK8d5W} zc$`0f*vK|;m5Pc=%h6$Drqm>0N-@PNU+Slu4 zj8acR-^=C55dz%XJJNGm7dT10a(h)s^jXpkwV=6=rnaVR4%Sd}qr7KEHDLjWaHF9H zAB56KQvk@_xczl;kge#2i_@hZ_nJ)|>_~rI`1zTj6mPh?9Z;;7;csPM`mB8;T|x#u z;m0_CcW__Wq1_R4^Wyp|r(oS^%baFv>ulD&d0%?28Qpr)#@O^KAtB~sGTiE?@e;*= z4=CtJ4h?%exR+bUd|+>0Yb!6r#jw{?a3B%*K|QU`&`zH+SCgwHbFl_8 z8o~f&$I)UJBHf!oy%L1#_c7Sn^^#}RSs`&6gGPR8oV#zE0!gl4b{9QNLc86VzhwwD zIr*kDeoMUmOP2R)NXP(+a%d2dzLJ$U4A@g6pGAz7v)O&)Cj}Z?kk$H~KljdxtgWJl z`Kp188f&_;Ju~=-L5f^>wlS{kqZ#_yQw}j6*;(V;9$M0b(1~1d9dLQF%l*UvsPchIJp*F%3q+PZXEjv zm0HVf4Q4ce)c3=YK=a`ju#O|u{$opWQY|OUGSS>DlL2#VRa7HlI8EEm?_~@XKCtyrK;`J4kQy(gCwK zsx59Qt)b1PS-y8OzHEU)LQ1-OjgK9M8Tf}p9!YL6d09ohGccJv-V zs(`@(h3CZ8w^NxJ$&jzss|z0It4^vCmU|t2_au=dyJh0_aSS~1uu-EaOT4f~0f!&J zsNrL88G3&)(0!>OuiWB@8-=!D$?wPi&E!kzf)j)T;>6{5YCWV^m;0Kb1L#fA+XtG6 zR=yaf0)=Y__k8DhJ!=g`S&#ERTfEI6RW@WJSfc;a0zp{|BrAp3Ja3})?EDimOc@JV z9{_BW4~G4T`e=>+FcIfZO5_`j>u<1{6leDZZ5N;>!6bV8BtEnrU1Vt6V_b<4OPFlPSTtUIpR5OC<;tnT>(M_+`$qY{t9H=kN1RclP zRA+GW3GtrUBt#N9*;0-2 zm;Va=W(G-u!*qdbXL&bYd9*<5QlzSRaCmi)*ez*2gd7;LzTju-?WTqrac}I>;9t-B z&c}WaCi03*cSBD7vH`)|^i&eDYC zJ+iedaqM~btSeS{C&|hPZmEF*inQ~hx;djTV*_vjxwi^J9_MZ1!gn^5iZ^wey2n)w ze|$cDdapMIM!=2za1^P&&aX{tm8VQSy0GMLU~bTnggZk7v|SiSLP|Cpdn!2a6MLew3a*A5Kv86%2`qyqx z$bf>G4BEc^+yZwa9bvla;H#xb1KR4k7@wM&va#vtR9Q@&B4wX&4h&Y1q-o65xEtfw z0YH8!D%##R)>v)^FPB7He&_1J_4j>JL!DPwYUltzPfWaJ#U3o3=DhGwyfGm4xtTN@K9^83l4 zug}(WrXuKMHd6`1WFLyt@{SyJJx0t6m)7Dw}Sd85HXlj|16c?8k$q+49^z!B5#)N4f zLK4)YI$wHeZd-ddB66~G85uzru^gXzG$;%(hf-&|ihlY^o;yD*E+l}hEi>YDr3B`< zBsrBpdWch;!M5JKh{3~aWV=_btH|1Do|M-+2st4huBVHUe)DSsy#S{AQhAeIe@|+(u0w zwtxAg&O8qeR@QI!Xpa{!K0jYR`J{IK2CQ9t*=NOtWLRx-L$oTMLr|eScaB}IXxkQ_ z8XJ3kT*{a?VArsZMVe2vb$d=tl!rEL$vXDCt#cWcY?&_<6|(hxy$kAjYx z;&=GvW)_yR{b9oT83DTsP+|cxH8eCtSI}yrrfxX%z8Sl&JypyyipjeMs$G)E61)54 zas_3i#f%vamHxh8?Bz+;Z)VQcy-?!WJ*CQ14#=w9aF5%)yVdTn*Yqe4&D4i4jxJBx zkdsvuMnH1H0`%UXmsaZnM&LO!ev$avCA0hMxwI02AT8xJiupOxf`h-5f>wTdZM&Wx zjgSX_ihg>Z?9WNop7a#()6_KcxgXA~2K!v>OX?xy4k*0SV^Z3vNWfXI5Kw2m()rs- z{5HkLBZribOK^BWj)SooEm*iSC<1t@cY910XELaYyyw-QXm<3A5Le4`tfZ>3$ffa~Yh>ZY`faQu# z`~e*}D?cGDO-$UnZ+m3-Ekg%5TL2~Y+-B}qoMI!p_173AT^m??pgN$mTDX(Ut4CPA zUHKB!Tm49%+&FyB$3I1X9mye6=ORuKxq3R{5izla-RxS&@L2w8WH$E~70{ams}5PAkiG?lz09 zbFRL}+r?@NjwAFmC6=|)@LCt=cHOHC|2esTZLBm)z9CE^FxW~fu4kXk|0B-R&V-3s zN3$f2bpNg04iY{KE)dhZDjJXZw5hL2xHwnk?>J8JkTB-noaE)ykij?Yngbi}Zp>x7 z!?C^&L&O(mzN9}Lnrjr=Bm7}9;_3}) z=@}Stx5D%1Q1cfajiisCuhcbcEej3 z)e<$Uw{C{h_pUfdIyy4=1%;NFCQjtP{c-0X`eu;be0S8Jh-#*g2$`!{n99<^62Z2o z4v?*$Gf%9sUi-Wt5b zXHn=MT}isMKQo!wD;p2PaQWr0Ea}>{s@x*TxgvVGdoBM}8`rrV)q&9i`S5VP`Bp{3 z(Y@tin%lIR7pq4P^faj}Y!9Ya<;!ezQ7z|`@_B`!-O3i{T~2ictld|0zDkv7so&yW z_!rm|&Em;F!6os`pS7^*^Iq;J{6D{Iuk6c}H$c+8jg56a-G?hOm!&oyKawx05fHq2 zx~K@hKF7o{=-f%{_`N1yofD?>JCjEqnkohne#$Aixb5j*G{bz!%lJ8f70Ri!6`@qRD;4pB`Rr ziFtJGOOzAu?(Z5tK||EUvzMIX;xC@BUp<)`7;DyI(JV173z1bImq>Wfdm!iQOJVa? zc37X^_?4t!chG{3-mjgW<$OQ)_8B6E=6Q$u$mHc8f<{HwCdaVRej__E=RKuASv{u~ zP03bT3r_ncaI zXt`VZr>&jBNzXSHKOt6^A`rYpf^^DC{fADVEoIfz#+J86{s}|JmM`}A2Ofz)RPG`5 zY))iwQ*~nb;)LN~;f&5L2CbbFSgBZ@O!Q5-SD|(dFP}ilRD-(-J0C-b817LR8CW3& zVc5!kc(TWyL>XN#7`YNhA3esawBYv0FMst@QR33#Cw4!TqE76W_u+Z>aYf@MybMwj zavt9$ES?VMdQTC2!H={rD?YbfeyGHvML{g38n=fPk_t%x-NZl#YmjO)v4t zN=Fc8vg6lT2+sImVq2W3);n;zS^S{MAM;IuPO9T3!yK)RL+x0q0vPYa{=z`ThPh+}BuGkaxg1Fn%IrV7pH@33DRsnO(nmV`p0(2ZU)*umFP{C&+e>Y06gKA9R8h8u>*iE#x9hnSk(+NTU1jUSMnr*0dz*@daAP5hNy zxYH@Y`<^}7W1-BYVLJ6X<9md7EK1ZI1NxU~VFWrX&cgLlk&q|b2EQmz`|H}uj(Juc z^ce%A0UslFGdM}v?vX}OrziW#b+JHF39o0_@a!E$2Y5_Q!Dzx&>vz(L%$cVIx$E<58xOt%%#6 z(kBTyIghn4M?n&U`X_$o24&w(L7rFn&69r{FXk!k?+7?(W@0pGjIz^(n_sBvQJ+X&%*2$=WSkE!uSgzw(ijeK+xsFc;=fp{j#u{HZE=P7} z+kq|~J&|(hSUHKnI>}j#{l?Ndgq|o|UVuN!v;L3k$=(B44v!3gEkcH;mNzKe$k!m1 zQiSEZm}xQq=C3y6hkJNJc;$TS6qch@!L(-Jx#x(DJ?awzVE=`ZMLwqyFIRykki0+Vpa-gOcc;Bt0}yRJkFfI(T*i9>D5_$Kpdhka&}oB8Kl0C%7X(YHn)5 zb>BfHtC zfqBfws6)sg$ZvPNb~}8hX@E9`?xDEDM*H--&%5)D!=Tp?+p!QKe7r}*S3$~TiC)Q_ zruEj5dvi<#UXpx{3{prjOcfDq3w|=1s4nPgeY0VMp8E6eI$6hCqHY? zJ~L=AJ{lhxk&zgcMq1|nVcC#xeLq%z?m2fOpM z+DoL#_!W%R>$*MJtoXkqXunk(QA-k?mesj^69Vys0tMyD!o{&22vMU<*eJ)s7W^d) zs{=VgRR=LJ+~|_l+~Uulg?xrzk$gdVIqwdPszsAAZT*6{WJZZVf7Ed9C04xk&G>y$ z___D;GdSqqPW=35P7MUIzW*VeG0>W$%YwORn%Aqq8gJRHT7 z58`%a%i_wK5B`=1=sEurb`~H8V$NRuY3pwA4~LR|1S>izUsbdvK90pApq?(a?9sh1 zw!o8SzaD$^h;2yJFK@_5!^8wMFvLjSxg5Y#;;ZEBq-igfjmE1zG?@wRyaY#EMDSjD zew(c=zVYPQ3g{x}%_6XGys}cY@#(3*%Hy@_A_)+7V3ciR+C(Wwf-(Jif>s(cE*u;w zrYyTZFn6V>Vm?*TX$ODAQ>nj3vYm~dGDfIUzkk(v%MUaopS0zyi5W6=b>-MBq&z&e z659QNf7@SW=}NMX1YQn-x9_X3eK=9SE82FTR`o39Hs08y>qNj|UMZiU7)0#5hMOyF znsj#_TH~378QRh45CNNjMKbZl$WgvE?x|*40h1cikJDz3Dd~nk5))#)Sa+u~#0;L^ zyGcOvYC7~={rw>S|HPF9;i5{X?RX#K8OM8z56z3~WiJNs`d(8{q{=g01Gl}TF%D@r z4lJi4ZeF@<3?VDvRwj>$w-i4-Q~qyw<&zF3UCqPCn`E*eWx09@bz>5;nBjc zxtZ<$S>R?;O8+mKz0)Y_A8(Ghp?c%g=;q;3!(55Ce2!Xynn`R=)IYNO<#`})hJUy# z5xpHl2iCaxth=#|TPMt(gaMf8t?#SrZysIBW((4P7{rPsyP=u)`&#ab!ZUPmHM1IV&U{a6NR zo9mwHLUg1=pgXc~Lur`qjp?3h)9NPkH4rRSZBMey28@cmg{}Ln89FM4*Io!~u7kqV8qJ+CO(1ythoQD`z%jj5cDz`CNQ!SUe2vr(v5?7$KEMNVf!-{aO7p~ zjyZUqFueas0(ki)pJrLNh)^H|=DMT`t&;hw-d|I7K?H+ywAv*pfbD3z5 zT2aZM;#D6vbvvqRU27qCXYf5BoYD@y-)+O}PHaX;2FrVB$RxH8NRpaA#;+aiC>xe( zH~mQmH+sygX{mwzF{Ww7#q}(qoAn4 zw%`!$nx&mL@^#0|llc8xe*V*cv-17wTIdIyrvUw`DkGQEEShFFzNhX_$|UV@hoVn~ z`sw|vAm)4M<3oOG0G@o@uZI!xsO!y+(x?36bG!yFyn3S|eSUj3<9kzL`G;Vk$vDLw z%@ty{m_n$}d~8gdYEy+K7VYwPiB^&N@N>nHgZy$IGLYgcv-aQT8q=rt6U7nzrO(!TtorW$kQw^g`4FHa1Fzu6Oe7vqVL)E?Zp3HEJxS!`(yQjdlDi+id&Oa$7gnVH78D@q@*J67tco$ zI@aO3<_b#$R(C+ThJG&BBUbK=yKzy<<>c?5A+YZJ=^;@J{D1g@RN|+nIJ?8FAh;KY zLP(dZ4Ts-bg~2~w=XJQ7bspj(tejaEC53=_4?%kWC)R9t=USf!^~BBtDB0!QKO+4( z5FyK$R4P*UbzrVTh46!wMPVX2Rk~W0&G~izT4nt$fe)J%HqA+_-ePA~9oorG@zd+{ zWbr9rm&aJm*VsfJf~%{Kh}S9u{sjy0;21SgYEME2GC55L-CR-$zU@5Hw)SZ)!oB`^ zB0zrY72i>lQ5*ct^$;j43*&{aWPnedDq4W+<)BTQ8bimraec_vUrxXLPo|A$?RVv&=cVEA|P|1#xvqZ@-ph(U+NzS0;AVC>GkRaJ1=RD*b6+v>&86@YN!`vSI_P6)` zoqf)|>z=jlA9t1ub%uH0?s~hcs;i!QD!T3wIE%@w_Sv6VbV=_!E4=?e%<$#N3@_Y` zeGx5LD?@lwVEMhJJK-f0Y7Ztv|JSp)PUK7KXX4&2?$P=lsX$@9{Rq_IM|3+lDncOF zH;^>@4*xI3DffIJ=}7afBxyHcM#jbXL;LqeyD%`B_7Z|qcQWQ#={n0R~1m!e9RsH!k zog6*|HmETIy9Fpdr6zW@HxHlU{rv_?6Rwlm8$#mp!)U<6le5U|4j`gOGaH$K5WhR~#^&wgt{)UPHckGP>F@%3GP2>*TKFX+Nl zH|g|$D>@*Ly$)C@=?B|5^i{H(?YB`OH(-3!6yPk&*f44hj-(-sFX+0A$|}Qi9Zp>F zwT^q^-V6YBF-bA&$5m!GLf){~H~zylH*jON?5)&@R>cyho>~3?S0~|kiVO!tRNwrk z@4V`fR;9b(QyIj=p?_Wu{r_iOYZak+~PEl_Fech zXIh#`e$4ZvxWoyDU?K#gLSM) z>n35i^v9=>8&Nn~wW;FXWHRMCO!s2^!6NQsVnI0V3*OTWka!LDaT+G4?wOrE;1|61#s`iFGX>s~?jBe)i{JjCGARly1cuDZa2rvY zBQzrF-PunNg+^qK$>LofGA|N*QaR|+HJh$$YJCT1;z-s^fF#{TF!kQHU|-I6|KSAM z<84#w3_riA6Vu_b5uOCoO_vm9@0yP2+Aw!;;J^7*K|bvOFATlIh$<}+NIHrD=h^(4 z^ra46s8U&pE5apySDQFipzbC+GtkK0)2+ztYW$T7Z8j8G^KWm-Bt+hQDXMBV(fS&on{wX>J^H%&_ zu-Vw3b8*J?6&Mv{l*L;z1Z&mvx*cAnYsw~8I(iOF&Yj{M6#uQ%6Yo7vz5nk~k6_%V z9M9NlFqyjQrFv#p1KP~saSn^h3v=$PdJrf^=CvQCcx(nK>}I{a!})Ex6AO`fn6Q`0 zN;%i_bp)5zofgkFc{sEmbtYs~cuAy>yUd>Wvv~l-1;|dmy1SuV5XDPPDb$g12^Z46 zs@@G?O-{;1NtWtT<6@+0q~hh2;t09*Zrv6+i7SLA4d^cyL}a#-)nOe3#=;R6vbKWY zs`wL7d7#L4PrI7!_}H6RR;v#0VTsw+Tg7K0UDvUcRgUhY)A2^fJg&dko;IH`+g$&Q z=SwwJ4S{8G94ofed!GycLOrp^no^L{nMSws)Xs?KLIuS0}7d}*M-JBo)t;_}W>BSmdzI-xaB*$)^qcWuu;!WE)x`% zd}!Ntwj#0XU6wvI3}9c(0K9i_4~>zyeIe4YF|OsK6o#Bh6!f@po}JlTDLc&tQgjLBmpmli=+Ci7bd;!gG#7 zr^bGH;-z`I0!FASatq-@PNY&)F}!;C0NSL>NA9rkrM2A}POa16>hRN0*}Q$gnIg>F zn(FpcpnzlVdCQDZhH+0sWb6LZsjqC*1wAg3@sX6U*En~oE*D#lrq1zA1o4>8XSj=X zOTG_^;e34PnIa*c&!eveInLa9z;xSGM{V%%efP$tl(@h%dP#U(g!88DudrvfXptVaif%O&ugH&HmVYIiuwSsKBLWOz zZ!rWrDopKb&$a79@uz#z=kJz*og#nrGt-#HF`=evei9(A#F#67a>VaxEdF~m(A5sO zIJ>jWyrG!1w5xF5?kPU|)nMEdAidwwKpZ~@u!V061itV%^w6>$C--B*g8X#a?dxP* zObI48-xTlnxGfI&qL+tmZaQgLB2;!-Mm;# zU@XnWYH{dw$*mg4_U{V!swS9lj28={Kt3&7{Z1O=-T6LF$9@tk4w;|J5TeZd^5M*# zXhVhU8jHh`+sD~!3qlS!y~W?@J7EWS*9YOuaL>0Zz06!!BPufg8WeTzA)Iu)+@%Lr z9LryZC5)IoOHZ&;+8biEwp=2-IY32qY#L*?HJQA5^}>F4zahk@g))|?4V;&1&Yc6- z`_FbTJ7ixs`-tjBB+)wE36AVq7Lg| zxKY@(9P?wN$7?q76(HHCAd>j$Mh%C!vb8ep;#K?}3C2Kwe@No4#Y}OV)176y$Q+RH z_9$Lhwh@dq(6fG(4n%$pjw^I4n=RJ)#f&B!eL+?Jz2SX7r&g20?7A4~%^@kLcClwX z9_&whMteu6vWsdI1&H&shqgv4&Lf+{*d5x)0!57Vb6k>D=nkTu+ab5dM>vk|sjYRBGt zGl|P7t1WoYEQPQ{i^6m0I6hmCkhMC>=ZT~~)eM!wvX-S$f?53s`BZiy9j+q7Jf_OBZ8Vpr= z^lYs~WgE2SU8yV_Gb7_F5&ptP0y9eub5e!4`rp50{Lma`xCj8ix^TsTg}se!Q$1^A zWAg_9lQ?4h`9k>R!-qPJsnfbHB*)W&k9=^0awp!iQ8ZZK1kVJYymv|x{iHmvQ90Ug zq(K*Paw?~H>mQN2_Hz7$PBP7qHBwsX+(85S9>1GeY`a*7bj7^if8pT`^(5TL&NhpA zgy}E-lna{bU*kG3Ah1w)vfW{4uUJlLsf8|cDJiE)WW>qB@^-PwS!r)x zZeeRZ&i-6Jf-t{~m7E`A;v{qH-eX>6j%<}@o4Y^2l0FyoR5fUfU7bq~87W}uKE|G@ z>nfKP1M2JP(dM8%jzmyk9?=;FBrx$+WvjPaUTlR-ohS7|V#9$-w5IX4oyLlRvV^Tb zw6}(fkuGDLeaT8sQZ%2tPP!;nI6T%QJzYSf;nn>|nw{R(8b5|geJ%QK%#ABnG)%he z!bK>fl}k9bl^WKX?l~|9E*4yr9_8p-jC{bu@=*xK3=dNTn5GT-=oDhdc#<5F20|XI zE^&w`SC9Ejh?!z1wl5#)x3=EbTMJ*76}t&?@&vD5FvC_Rlgx9^@~cb21_(4Sw@7OU zi7qH05XOO~-fzQ`ty9xE(u^(BLFQ_YJ6_@hg$*s}&cDA>Y>DzdL5m*E~p}6K=)8$jfPN*F_8#PP=T+mZBIUS7nVdMB|4DvAew)0aP-YzC%o z9}|+_>~@hfH1e_3A})T1z-_E|OFSr)gCwMu$V5x;539~3Bn=b(0PeT_jXRPM(Man$ z@&et|?)Cry2li^s&sxVVgUwUlCbi&kaU}28?MEI$F`{X0Mjju(>c(L$Ei1`dhTG3; zTu^$3Mn?90<344vLzu0tCAcE3Ul}%?ewg~Ln1V>-%bpG6t8bp(k=zUGv-hEU!8bSZ zY{c*KEeoyoZoBOI7O4g?VY@0~ChSWD331|1Mt(1 zChw@uxpSmm(_FOu#z4%(UyF%@le0j5!l9iv$U{4GQ<=@~I19OWyhP@Z_VZ^3ll9>v zcaxF&&(n^Unq~W1#f>R4{6NQ#W}9y!-7`opUmBZhz(`BX2TQ+m3Brf+C__7>b84Jx zk?+d-@UR*L(U+DS1avHywvy}W*~w*Y_+mj=4N`tFpAsM*Q?kOB`x7^L?4K8wva)a$ z^p{01hY>t5!xZK2IGcVjmHRsJbGct(wq|7608ei5;MMW7nTtz9b6*9FT}Q4Su9~RM zhLuyTHbPLXWpJfmw;1it*>bv?ND2M%Y{YupWxtyrz_r0K9wFnx$oZ>9B%1uyL=17_NmheY%q3T(U!S;lgW#9A zd|asWNTFM1Y^D>orvDdMbEX!K|CGz__gHPBlS&mG`!JcvM@Bllt{A%J9lqgJO7WfC zB2TOoL6jRyX5vk^!&baa!`(}#I^2}AO4@wIY1#!&2lDg$HZx}7sHX!qO%|(q$uZ00 z(dD*GMQ#{I3gUK7=LypsR+?!>+B34?1ZN{NY8l8Fh;mLE64$S+rdUq-t`zt4;j&+&7S}jYd}<$v-6J zw&%2<;@D+ni^?=q=b7>+H8y$Xo?yNBWprUCc#4mhge>$6UATlco#w$ozCebR)u!Jo zinL3Qy`^kT*~VWAfs5!mLsl~@S_MUtix&;?8W{e#KcdKM&uH9s(!EE>M%hXFQd|(Ji=D&g{&o zHy`Dh*(eW8=Ol*Empw2Yn{C!xL0PF*F=I=q-jt9yzmTFVW!3md0{34#;hRV-2u#U* z1|ds3;{w*E)mjagKmw`5uea@&yvF$)l0#dO2T825=foV`&sZoNFABuM9hk`L$R4=y zQyeuR3M27&S29934)Ro}(~Wdoaotyi?M(q*Y#RGe_Lk8wuZ3)UB_S%#IOz^d6E|0w zBn`7^iA&WPjd5eU0WIP9WluFBkrNyH*0-S+Pnk_aV~U3RM8HQ&^*58f&8v_F1Wk}on{awk4*!VRugd&}O_#C}!W6tQw`F8)Py^rnqHg)$+pF5E0 z6)jI=oPzg9nLDvvEw$3i>}ETMoMjnKhf=S*oNX=%rCBsx4u00~*DYJ~0cJyl1$XD> z#OT;K!;1LDjL9~BC9%^$=UObawQG&0PITbIjFtLv2l;UHmQuu}$_V-#rkTy9OU^fD z0-|XCJt+c7yv?SSv;%t|iT&!})Uu`K>t z@9&?T23l*=BkSlcjQ4GsS$C}sQ*axbdfzy1zw$*_pjZ4+QR5<`Z3o48N5PWji8nI4 za}xs7qPldnx445R0)aVU^KKJ4-rDk&U=&wfB$T+LXdj{MnJ8LCG7*YsWcM8AiuEF~rLr2W$~^9*>`3ryTqh{=5k*v>tA+R?pS9ZZa75BP$U| zW08D;Yfji<)XljRzs4=Qr8d&ov~v`G^8TTWj4BqScyR4q14);Kcymb`1*0M@jD;(V z>mB2p%8$$AmOA=jC5fWyWBpCNPk3v&)lCG8ndxZxd1&3PxzQ_16TQ!$YHdFkhxdt_ z-Fm1!IWSei!PQ;mcG@d&U$lf2oA-`^L6sDq`ZkQ(l*f5u*s=Ds>(dQY@}7%zc0q#P zJm6m^je~4br;l+&U)+FIXHeOPwWf=!s^A4%Hg=it=naNIRp~9YfoTb<$kJ2$SkAyz z^Eo}wIT=HRYJISWJv~zmNU7Ir9apP3V;@aqo|8v@jmqxqhvgS$j%bdnKb}xGWX28s z5)%;xXTy7F%tcYv+}Ma8e>h;IA2wjOr6X|FQRZ!Ir*eK@JjBc=z@eS&iW^zF-2a11g}mV{ zmjVXF?d;(0jqJfjwS=hb3lHQ_t-hvH&syR=bGpz^Cxd~z4P-S0QOgFq2^J-{wmda5 zY}*nf{mtMZDd9kKX22(^Jd{V;qsZ>5Krwvi)~6imot{4EC&;k~F)%?0DxORgvUUZB zg5^;r(u$w4k~0ybTEg)NwPYoabz~$uYVkq~mzogQ8=^xjjxe%5K#HmdW>&Y1j42*L zCqpFSYR^uZZ_BS@DOW=Y9>jKT-(Htq*+jxJkxpxWAlQ$|R`C{F^S2xqUbYPuFQ+8EJ7Nf}7S+a{JwO)9x zfN?Aro?Ay%!csG5ZDXXzAS_w>gHuf5E7z7f(^^|%thKQ_exp|ti93BDIE+@R-htd zIWtre6PcatQ&j7eO4J}_nurXicvZgV+#vf5K-)#BadcfPb7gw2s&)odxG4!Ghu!LRKT8ptU`4eG85^08<_FI+<4rvB z;9>||)D1FK?4k{h;#x+2E? z$gdfzmV@TMRE&Byrne9GtTz;Mc7u6*;iv-%aKNF zHWa$AB*kBsdwh^D!wnuly9(6;NIiiC7xoh5`eQg~B;j8)fB!DX-mqK9B^yBg(hWC- zO3)3VtPr`;L`+mKh~K%HcF!*HGE9&_03i3{$isLZ}UGZs5 zJa5l&7g73%k)*P$KtSGF6y5_{tarbw6c-cD{#p;9dc8QY&2W3-F{{#X2jkL&<zECYR5ucV+TOs0%Z~4Pahmbt?xE?j2`kQ~SEC zbX1OW74)En2BC4ZT1yz-gRXVzK3^W7q6LIdYKAiM+#mK*eL{}&vJHschQcN8VDP%# zi)ZI}?&`i{0qB81Bpno&zgrkQtvnG08MEFDVV4eA6I9ObdavE4@!Lxtvhm3k<%%c` z#JySqDUjD2XQ#1&%k?T_n9HE`1bhn4wlqMvTWGf_P=&t~ZFvClWt`X`9mhc}mjGG@ z_&{l+1=-p&k|&?Av4sOf0GN;rE6ApOjKLGi2bAYp)=6?S4%EhU5x%~@VAM&{-nImH z+F(gbH@|=&ft8YF+>w>m`xJ@v;))Uo$N*Xq0j!C+Oj6u_-uk;smA8O(b7^xZR``K` znn!%h?BIxFXw7IT@?T z%MNv2r8uc9Co=PCTTU=I8qhf4zhW`^G>eN7T^CL>C;N%<;nUi0%=ITh0piXzeo%XH zALCl>20FGCQACbZ7WYeTYGIZ zcu0~kkRKdZDFM};M9BO1-udSxZK)_8{(Z%ykH66!e?IL^N=AKC==aFL-T%j%fN)N) zmlOblL{fY0ee-&Xx{-6fRMxTPi@qG;stfL{;^KR{^);&tW#J=ahS@{IOH|hcYvFc9 z?pMMCsvG^a)0b`QhdHWk7T&S$1xWXccJEIorBW{=C6-9}v@;K_jdk0!*qlH#lN`aH zSJB>jPv=gHJ>y$zb!5r)KWg9u*ybHG+rJEFy{$~FYp1jIK#t$@t!aJ-Ux38d^YibG zmYWrqSGrfg@7mozsF&u8YhjuwK59+J8`&a1)35!o(XbvlNn&}AnJ97O)n1||)tW4j zVLEnnAPSD~vJH0YN};_PWHB3RhYC`oW2NZY9Ip8&91i!?fTTabMbo{ts9~Ob0DUp8 zW`Dd}SjO&-a3O^>{b)S_AXtK&tD7j)3`kN_G2qmX+CSY97Tq6LzFkIpq8aFm|tI zIfrtU=sM}*Yp0g5EgDP*cDbUqd!@`awo0Afb8TK<`svGdR<E_p`v)=N4Aj#T#08F z+Xfl0d?y5n98Oy*P$-LCYqc5Kk!H$Fvxh4qDO-Ua2$0+D3dBAy&zdNBsSSaF*tSp7 zm+Su=yqDC8!s&m6f7mi*D;XhpWtbDR-xglYZ&!{-12of}^J z!w;b(b++5%-k+KV1BY&3hv5QHv~`e$c@+F;z8kxxKp-$swo7h<&&jT(Di-<)tgj#o z9)cIq>=F}kPzd=TA z-HH)_7IG?}nZy{}9oR~6_XVYLWK95UrkEeM5pmdd$J}$Iecvb2>nKQob@?PwDQTL@T0*PJ<1X+dghBOI1on6MDj+ z2WU#>eGe_hTL)U@ww$=%cKKR?lno%s!w@57Hv%^@go2@Z(aeAzIu;vasfxs!7W9A# z0MN@EZV#<~-Sz>kCW3iP+kuEA#Hl(@6&Ls9fn1#7cj>xS9S}=-D!Hlpzw#IuCEA#L ze8g&HmnHhN zsHBXEzh+@UE`1}?0GB|5keB@A2PZXKq^22j{v?^$eZE4xhT(oMx(IW)IpLy?OTD4} zY0e=iYowr}q|Qtu+pjBh-|oAIh*6>AnDR!GCPW*bp%MuJ()va(U+#wS17%f_+g1C3 zWbq-C;oLH~M1%k$IzvWUvBkZ&u8X9@$2xB>H}D0q>~d~D5_TiBh;g-67Pj9ae1Gqr z@&+eDD{N#EP_9h7&pQM^|1FJHSQ6GPFRfiU0q`un`Xe#-kuSaJ@KGiVh?RgI@4=mH z=(Rr!O-`5%00qAM{poL^qx76KVbX;O)S5O8>lRAy#;^+g2Kt}`EPR*d`tK_zV%~XA z(wq_=im!CPG+{#g7_t2;b(0n!jQ|rMJk4;0##&*g7Z>8*lqjGl)^EcM-i5cJfId&Y z;lr(ulc=yC6rxNt04YVlrD$xm4n93i`yL0pCNNwH8-QTCPQZ<~bkP=$`5BJxP;(nK z@Fb)a4@=0%@wZee80W+XMqL9g`%>(e3;0FCqys+p@ z(#-fP@QMK16Se4f4+~7Ctrp`$lN2bDr0-e>MHf8+zBo{eOQIg**MnkWh23ky3kSO$%m;8Zu7M zV=&AIAwG)&03CBJ2?Hf*)F=T+YdiyhY()xn8G2C@WP4gU7mD4# za~qo6nxjHr((=OSG71cgMjaF)fxii803zttVj{Y91KjfMq2XDjY*@o2f|H$dy0po3 zVy`xei?4Rtd2_k={KIPMRdns!8WvVIzN>=O;%sO-K9mCS;&{eWFit^JO;Ec_ZQnDM zKUYenJJMHB%6)re@YpllcZx%OH(?U_WQ2u&E!$KrpSlSDSXjE!$&MRQ} zDEOOqz9m92(w)v3#6}pFV(%Gh9N)T+)`VD~T&*QJrKnS{08J7v6`y4x7jR z$Q+7}T9XkK?604p3z>DpP;v#>fUGWt5^A_{!xx|BX_2pE4iR74Dc;>jSCy3rn_u(* z$#97RM|GL49!u`MPzD+yjO6lCW%l(@Qe?uCkhsFez@v9#9b$ZGPf`01EI+Sb#*NFB z=`7DS&3-`KUhtG{tXA5uW>@7WA6SMzUm=A=#$2$ryqCc-dtKPx#pZs%SJ%8SzV-c{ z2RVnQsvS2MLl!N!SQk?rpWS=%HtlrVH zj9HHga?E2nJJu`s4M!FBl9a#a5FL$KLSk+>GZh{4m7kXK+^79JM|4n5gs&eL z1|%W@q0F(CEzQ7M6NnFeZSVv80>>;mOqVaG%`ZU3eK6x;kL7ovgfRoH_K(uyf^ia8 zt_>-dvm4kogQ-t%c|B;dL+x@jR=sN>G)l2*H0$gfl`g_(3kk*nQf!r3qA8Yx!Weqa zZxDUK1kg_rr|TtglRg@ZgX%r+eKioL4HS}xx=;A(q%k0A3NM%M-zrsQA_J3~ldc4T`q& z-JHb-zx5y*mIukppGkz~&+x`lHPJtqP|#xT%>$4WoYc>VyL$2A>_Wg#Z5S^4$a^Ck z8;Sm~yCD~hFgap4P2GeEmvAF_>z39-C=Ib=XR$;7gxB1zCw2&uUi?!O#m~o^S6#E? zW@YKpb*;SMVPfF}3YVRU$StNJ6~=RWK+G3o*8 zz){bGgC*_Te@<&aa0}`jX1N6n)W?IzsK-EKa(%7;LpoZTBXMSCc9QJn z#D~K{vsVh31hih)H@WTm2b)$7zZVb^6ILnN5e*l#06vrXUwC#*Re<7;vx~UGe7qNw zeCPXVHYCJ5lDq(96!IVh033TT$_rKD#o7;YZl|VE;cnxD1$NrcOU}P`ZW$z4>9?}E zNq25=JTK^Jf4pb?^NZ7dOfF|x>*Ev0pWiBwxvwdKZu7DIbCg#wa~z3QozgHbS+0GtM+?J@hN~4cZ>xG6B4Y)Bsm{Yl3BaL?m_4t#NAd`2xKK{<0avI< zgN0om?F%OGGj*=}ivsRPK)x~M564PRPS)%8Id|M8p)bz?Qyed z^eOizu(#PuTEBxXUJvi0EktA54cjs|%I~oCE8@9`GYdaB_@ULdOP!eC=C~y{ ztXy6O2gCtjKPG?)GuLa6^|!c9?p#$dttuBdYC(V9PGoem*m^W#Go#MprU43o2oRN^ z{tG-qAw9M^$;$LEYxqLDL-vTP)7v(SJh_UoOMtmzq?VI4B7|rVqz0!6s`9{z#Izpm z+kSK7dpe;UMU)2)*3T0>g?N;)MtK=9bLh;0iC94s}HRLaMe@^I~O z9(Z6TrS|ii)1F52&W>v;qQeK*Oz~mvWSw+$4+}8M-w#V@$!+k9H}?5D>R$ov>EdB( zpS4^4l3f=qsTCMOv#*3;g3giHHUVhqU;37|DOYK8)*K?SHFCT1@JT_A8KsikZ3PRI z?_R%m2QoN()b2|<_;5HNL)!_kg8Uu zaRRO$R~5AV@2`2i*VpSFx7Dek15L_!Lw`$N$eOhuG*BDkA6?7jx)yLSSgWGoTHf~f zAoz=fvYH6o}QZVxh4H-Teb`ynWFBr85WQ+WyHk{ta9j7-P0O7^)M1@ zaY>aS%dca47{JvEfMb>^r&~{hax>w4c@!)U4?!ds56&Z^I1q_=20x;hbxz7`|FLj| zV%;o~y~+=A>Fj*9!Wh8GK;+2FTIq+58KvPuiHNOdJ!zst3JVf0Z+yZYsr@olW%;}= zy*ZXK(D5<{u3v5%xLPp)e5{>X@bK8PxL!JmgSss-a2L`tf1hF*Meo>0z?C)1w)@e=xZ9A3tdIRer{8oQ)~in3ERQ z2%8IS@WBQ4Io-((0&8op+EbiJlZS)_vRBPq1Aic}VquHOSd$OD;5gv5oKok&QUE5! zh1A3bO#S#^bq5v4O)Adl8kO#=&d)r0jlZfMnQL@6K61?Ctw3&C$SF-nvhT$&`2izE zABkkn3Kd{B``q-Sg1m8`rXpka!YA&^T1JWbVTwFcE*lSsa>r*p2gifo)!`BbR8p~L z4mx&hH6d*q0!w3d>Iq{O`o;zRRW=BT!@%!stXR8MaQ+sPBN9|!AT*(upd>|T3rPQW zb7fSm2Kd*X6OP;$P{qQ*yhkpk#H(7vhhjJa_er{FrN`Ur!~{An3N2m1;p*2A=x!47 zm1lW?QgFFU&EE2&(rF2$rukT7MDC%Kv1xU@2X{&07Jl{ZYqLXi6RCk`|6Z23SPhyK z>g=BX1iYevQT1&t2myOVM`4!XJnqaDL>;6Gw8pj|mj8?}^z${e0&4GhyzsNbmDca* z;JND9QZejlft4fQohb5{h^*>eOa19~{e9x-Z{J_(rJ#-v5`1{(HCX;1fg}6gMM+&< z5OG>#7rRt8X&!f{vi;9h^cZe}GoWscmD@dP%FMglGvPl18`FeS{>WvqDg09w!Y!mT z7S%4faB#~|?SMf8txsZsU#u`+)0B;|&4hku86wsnGU>v1otj~IaJ(eaAE+8qF1 zN{1}{7WsnNN(Ddxf!6y);U>za00J6CAeOr!feFt&i;gxy(WDPfh z{Yg3|Nt{>!HRzn8eB<}BND{_@pnfwpWdbxDL9z9J*n4I|)ai-hf&%BHb{4DSH&@xR zynhz^Zvq7H30ODaC(0(G{sLDD%+m^go1ydo&KQ?_%DM%efHms@BizMK;Qv&OOQ%Hy+B+Ayil17@cO_1 zC_M^A&jLimR|Oa($32};O(@Lu`^#PC@857GeB5fQy?1c>x#k`jAY8XHx72DVKmYWh zZqt%ZIji^&Kbp`fiqr8w9|+EL>+1oD*B_LmPj^!-hXG<{i-{C*&=4Xp@Gkv?M#FRC*HV@jt9PRR zYy~;0zx?F&)q!V#+AHie#o_v3XH}Z0jT|W9U3hHf` zAU)@;u(DQclVCeuSw+PL?_r1FLrblPR|dI}ccQPev9f}|rpi^>Skq`$sL2$TvesQX z)J{vf$g;Fx1FaygE=-owoF09Rd0b&N$M#;RM=P4-c~T}kyX~iyTXcbjjlSiAM(P^K z-DQPMG^hVroixPe)hfsf?+V-ap={}W(6Piqr^B%LS{)D0%23tQXabO+05|#fMI7N~ zbWR3MS0H$Mc=lYkfqukhBzv{`%@O-l6><62SP}9oE|e;KY)r{~s#G4S6CZSUkpx9N zhDNjhp%*EUR4}t>y{QS9mm}#UZMIrJ)QR2O07qY%h>GoLMLk`qI7l^f3xrJPuEeoy^!pnyNl_tl^ z0OCgK@24i@f&u&2aIvwBOT58d)D^436n?9t|HoSC|D7)T-+b%;G%~1R|NlY1|8Ha> z)JF(9xB^1sYpMn0`-e5QmyYo<`JflTKs@n7va;jWHbBG+x&x8WO*m|iRs-(+2mY{B zK;wZHVesF?vAUZDe$&#Dtw>)ST&{vPE|@SQZ)Bv9ZbQe^c!d&Z{j3}{J(=Ecq2AVi zQOEW3#puK>t*WA5Awj$T{@$)mi;qwWB0?3=Dc)(&9xIX_aABkJ6Z`r8av%mQXTS`y zthHCdZL?Z`X@0uKi$msCsazn}to-kJrhcvZ*Aq)Yf^po<>D47gq4Mj-LF1gq-1fg1 z8c_B#iDbfVB`w>(Z{1s$jg!?jdkORd2zi#--O^0G)S&W*&Sau&C$!D*W}|J2XOB#^ zFyP8hh)jKSAADWN<*xwH_>Z?2Ql6av+kgi2JD9GiK_NWkZPfwJ9RwxHWk zMsZNt5yF8|5-#=dK4~mcgSz`<#2%qD)b>G0?Vgd5(PgEp2%xBL`lfRM4KM?zNBrxz zB;np4)4$QG$t~equBNHv5@4lkU3JRQ`r3MfxX=6H>OZt8Sfl)E^4I5V*2}f(2-z*h z)mrErm9UiP@o8yX0wVNpzm#ZGOk0c;eHz+JO>s5qTrpsl%OG5-E;q^19B{szY@mw*pI^`I0_+pOV|Ars+$ zv2X>8fDRv!DTB5YQh`C3u*EJADLf|QJ_L&ng&Nrr8fEF4t?~MLw^(;=ka=Ab~1l`hM$|4``7Qe0e{+0-;-%_!^)9H&Ul#*U3 z0H~Cfo|8+!zgg(P|1?FPb8~d2%E~TQLM5gOpjVI%n++&8c;kyZ|0b9pC}mk6vZcn2 z;{P`;7pR+k#Wrt_O0K8I{xKkt9A?^I-eV373OtDicR^cJFw%?#2Z-HeL0;;*ns0+m zl~#8W=We(zmA%zh*h~MW>3wDodNB}`R8p9i#e}1yzC`x~Mdbos|C-WKj%o=gtzJoekjmg*oE#-vl5Z#(Y*kP%wsHj^xmm%%-9&@a`8>aB$|Gs+FF z(^@+_AY@|)}MOZ+CKbEAo&;d|sjyW`YZ z^^i?}mQkj-O*h~6Be}2qiTiVz?tXzw7`D@E@$l-B+L$tOHnTI(igTQx!g6LR{rE}6 znRIQele67YTyG)u;18OC<@f$^J1_GCvi)}33FU4&3rk5$Z)qO`7-d?9^-P%)=phNZ zaXs_B*IDJ2f9V&DdJiPR)VObb$a#?r#-lNJLi@NH9tJCxR z`0)y^v7lH|K+~!y7 z_~E{HG6LGSPHAgTCvu~@u_w*e4E~sfWzwO8tf5ephIS|8MTwDE_)lzY3nk|9HMR^{ zacSv^vSlyFcF-`f=8#mQ&M~z*NVDg1WgoQ2>N;56_FyqxI4Ou`v>7Ujw%|`pxb)Kn zy{p;iXZs^TH%%93PMhAe&4>zP&={8St;HwOjV5_CPb$p@XAGyb_=EJxWPfyKcW znyr+rluBK{s@{-4lkHtEc66{gW$+^d;WI|`VorAj6M6XP&epW&>`;WP8+}>{=&|3yRS}j5gq-8Z^-AM&-4pS;uH_;*h;{G1OfvQmD&TpY5wZk z{Ut+-b%`>*$%ZTa?+uzx0XjPG%RYCmRMU&&%N)`P2QopQVCS7#c?AVqEv@&< z!B+Ij#o)CuP_llzWSXgSSnj**$UwCL$PNzw@X6)`N@B0)^jF-DQZr5JVRrIMQSbFpG-vDPW(1LXNXmIW6BJL7VEM z)ggpY3DxF!sXgdt{ZA>lHO78@?6ftSZ`a}o{lqxoBn$LJ**e$Z>KG#vQ@vUEtMO(O zNwZ3>&8?|-w)V4GkVH1c0qciQrScN1m7zELzC**qkucnb%d3$ToBH!bKO3G>D^lV| z$8C>vk5{-hb`rWchlhKVaUNxir;+`X8Zv zg(O1q&|F65Ud+!T+{T~dAE`8oa<{^{^fY5GqO&qK3P09XQ^-Gw7S)Sy)nz546B{r4dY+Bv* zF8DsynDxB|PyUOmLkkaTq()te^`=J5s^C?zYpYj%UA5D54vy@#jnadTp2h^2%~a8{ z%lcfSQY^{@Q1dD8A}4D0D`WZvz2-BmUz$}I($W%225>^4hFuTRpFV@o(smdUc8o^1 zTDRh0r`}I)t7j{xFeEmx+NiOA#<+HMRmtYGHJPue&%l;=ynfZkyFTn30xXwY^0(72!%HEiH|Z@lQOtj-QoFA8#OXkiV>V?X|1c3R{xPUU~C81jK5#@ZDb-`Sb~P5r<;_^T7107 zP1o^gz2BEe% zAs9HJX#mfO@F-J5GC+=kF(4| zy{e`xj-7w4f02INsEn9E1wLy&_)BVhJezw@!T z!Djp8$3{Z=JJ4sZBAIO_5q5shpZyrqr73$JdLJGAY#?>XPc*QfG1I`ppzhsNdj)@5 z<{6(>^#Ikf-urUW1jXm|a^`pSQQn{zlFIi_AmkbkZtY5? zFmtJR0!BB!6Aqs*-_`Xa`lB;#yzCV{Jg$H#a=U#m-`I#=la?9T6?;okKc;+5 zEj8a;EM%HmCc#ycbQ&MODNkdd+TfVTfi!Q8Yi(_c+U%UkM?yn`w>^`byE98TL>>^-eE)I()XZ9P4Sd5>kI$LevyP_^8UDxpM1{2$n;k1@{ z@VQw)C*k80e0==77=(^%2VJN;wl#9WW<*B&dB9w z4vAO0#+@&0R{28iSuf^sz6wlGdnMUXVB+M1z;fK=|5E87=G9>1 zq2qBC%k>iPE*a0>c+kNuy9}@4~in3?2*n!YaZzP^QeezOB=4@z<>LZW?NQ(!a(3<+*cd756v5b)PH zad5Ks4>L_nrtg+ZugARL_2{IF9Ozr!oHqjN$KbeL9}id&>&4?clT#X>L?!B#Y(Nj` z<%AnW&p*eJ5Sg2qsgEG)Xk2~!-|MS@%D0`q(XTJbNGBcaOR=%Bub-Vr!&zKc1BS$c zDd{P?yFE6^AKR|kk7raWcLyJRd{Sz?s8IwxJ%Ndy8UmHkgKu)`>W204jiKRaw)gol zy~@C%5NbJk^)!Ons{UWaU1wC2&AP=83!;FEf`Bv;6cAAn5HKLp1Pq8slaBOWq=lw5 z6$J$Zqy_2H5=!Vz;G;&0v`_+3dO|Qr52W1ToO{>0cbz}?*E#=^ti11wo7}&ODG|ylc$9kk2(4wfmAZeNg;&n729J%>PLe1c@t85xQw8AQ z*0@z_F^xbpqgVdudwY9pSu{U*y8u!884XoZn36@^L|VA^P4Lf~o2T{QaO_9h!0;t% zT?jL-wFu3>D~_%ueJjCuTiZ4i=_)J?vvC|D8NEMK`poGPOJ z4mn}j)RQ;=>g77SakYq!%!--3KxikS+Jl*SR%%*vB-b^=4A%4AFJJJVW{ z{Dk!0OLBLo2Q&BW_T1)!`h^IkZltCrM9P`+DS^^x(&~lo)J_S3y$Bg(Hlu9IYMYQo32ANYa>J3j1U%*Y1f73JJ%Ha-P{PGrI^v7ucnRTsi;$x(jK$tw^lCft^Y{O zk}ahc8$!u{w^+ba=Snk!cC9R@FxKd{eAMsJ6ihMO`qTGNLQW_ej0DKm8N0nwc$clTV%|IzFL3(L?I z48bM{+#^}3GBK)xeugkpcnf^?b)VVg`0p8w<@yIQDk{f801GuJSt2@tLQpylAGX`AH|Un$Op#MLEV9!lu`+U90onhDnH{iw(rZ^Gc~n1}fA0m3p~6vR8{rN_JNW zg4$MAUy#VcAZiy>o{x!1=@w&1a~mTg!1);SM)>%*6dsmjm^cH&6C1B&I2=Jdl_0=0 zS6a%JM4k!STt#a|j3wzzn$JtF@Nqr)l_PV_Re{8ge<8mxnk10QvPbJKE>xwM#@D0nLlaGL%>Z$GMZ@~7t8XYO!vu?+p*$y2WUM>7v0>}$ zZh`I-lq!?5<6!QJH{dfj^gI`gy*WMT0F#|3?}kAHw)SIRkjoC^`Ds?93(3ZyHC{LC z)v~176Laiduhe0cuz|h=k#d(gk*u-lDeLJ02!PSgm*0G?Ct<*63 z8EqF0OpUn?Twa2y9kkCBwd!oq-J*dEW#dy6)2t_@CB-Zy*l+LkjvTfg<;N}MMTna@ z5HmFv55AP8$T;nXC#z{hWT`ybyC;;3`;whJ9_I(e(l`%t+gX!vr;39D0F7X7>==<1 z5V-cJ<})fJC3x_<{AUwiD4!w>D{l2s4RQkX9&Ln(56g|oF}S44ux|GQgF*MfL;;)W zDQ`%{-*0Hp`tPMsv| zPS>+biNv~07bI!s$9VC`^jf0tdEcAFVcsz=ggm^G@F&sXINM5kaNK{| z)%8Q_DjGK#^f|OB3^yxG@xRQ{JG;zStapc8K60cv;L0jLmtcvEdyLDtG%RALW5tFh=OH6a4Mo}6ZEvz8c!}Jq zl24k@g@WhCXnoFIpS)UWe%o_6k|-FAllt~*{lG7o47X5(JWWAd=FhhuFgI>A>@Q;P zep+nY*50(gp$5Rzhf~s0bVlB(D?>~~c=;>(2LE3l=7;R}U1y!L~pbB;Kx z(Kk1r3*}B(nn3xK2$W7s1ET}5SRl;}Q_|%i1rh%yw9Yu=0(mWOQvd4HB3B!03&^$d zK@dLkBc8;#(GwgDTzw!RJ6Lp8RaIgHAm%793d|3eQ3u@#n=52VuSO@Y%~p>rV_#{9 zu9Ea;t_1HB13w`626R~WVX*9oeIr?^q^xLDYss&(cEzA`!#!>8wyIx&0S|Cqb6#B( zOOmJ&nVo5rSD4~iStTF9wzmi}4zz^bEEi^m7J+_qZl%GVI`V-8-sZ6tKcE^R`==-B zghhqs)pT?Q0N(OJU|Dy`-9LF#LIM@jn=I!v@K}{$`{)?m^T}Q1ON&GS%j}9a@`U5 zqE=OKMA=%wj}%0ezbd6q>1fyk^ZO{<%5;v&NDh8x<{99lP3A#6fcYnUI2qCmL0UAP zt+lR7ao_|FFgUjN1r+nVLBrI!)o_wnHs z&{G}`nTIvAmC=C}4i51T7l!NNw>&_3EEN?-Qyv#UWtF_kOZAM614uMRH*{t3bnAi% z9y6{#V-_y(wWvt+p7&4EYPBam#!X>&XJrZevkuTP)8)8^#Xhg{CN^-(DGHLFSzUU4 z@x*cuDayt_l^K-&el$e}xdLWB$|BvJovdj((oQp#^{y3AaZKGF)c^OacrS5*VVyMe zt*c?j$Pn-?RyLLdxQvmc<;7ZSAL7(@ZZ|ZZ3HVr2yWwOONpjQpeXs zBoul$^5p=R>Mt&c5@3psKAs*HlX=t5UIjTZ%YD79|?AgN+?>4vfV z-Fr`96#+D2(Z)!mm?NgZ;CsT#m!V(QLZh7hnSv~IGjndEvL|F(#OqMC+0Urw{?p?z zx)=UB8p)qTBnoUU;HxOI9?4y$&$z3!LYs-i%^B2J2~+>;aD|TEA3C~5yT5X~K?GgU zkOXsJwu){wuO3%i=yXGdKmT^831ER2`gqXDQsZpOqPS7MDenCFYnf+P&EEyZi-wvi zGzG+`rlc5U2M3b3nGMAHdWP=PijC!XGXV=kOEQNG48V$V{)zVIj<$WfZ>g=V9a`d4 zT`Z9<8#`TIq#^sELqiwaNdJZ-&Bt<5nf%waj8)&wb1GS7gG3q|XItFsy*I2sQ!K9e zrHIBd(pONDw>LL7`smc=#n|^+7VIGq^U0)R%4hO2DZe^2JWeqQlc+w%4C`FXEX)~= zo=%RQVfd5!SI%w!+5A1fTz048;gz(T_J^jZt&6Ra-ZeXekvZ#qVY2kv%*<3HC%tsL zPOqaomfIqxjALo+g4RDE|JQ8wDxz6{%y%7P~<-5BuKL*{Y z&U2Io7Fx*i6b|0-)W}H5C%83%^i1-0?b-zUZijcnr5Kx@mBKt^q4>GCEnG6H)-Mhs z(!+w4fZC8M|6sXlhyd{B?Ch+@?>3AXi8XB|+Vr_wZI6zPWkKlE3q877 zkVFg9+e4U&xC(#j9#R#JL1oNj(Z?Hst2VBTOT58Lp97nUggu3hallkuj zyAFN)rA2B_y2+nER^?(W(VR|>Lh~9| zICj@+Z0$=yfjw?$=L;4|#Zst-z}Vr>rfSEh$1K5kz!HXv8JJZDFM{}k^HV-g#~%3# zOEHiw@gvb9h(yeA=ay-$2UTt+1LeRc?`^g-3yN{iK58}C7#2SFmNIC(rWEiuc*{a+ z#JQ8BY)jy;my(~-(x%{fERVh#ek##U48jb4;Nqewd-Ev40>01Hqr`xQ(7=W81a2;o z1kdIo%v)0v6McZU1)?|(nl3>hmye{?v2cb_CG;~cH0~5?c+{)?Bs#W+_Q!N+=p2m_ zdwWtFuB#W{@!G7#+pYbRi7s5R5zMvqq4+dzZk6N5v}$!ONo};{YA9!S0JazgTsaW4 zgw%Df*VH{TOqP?A14!R9$WfNV+wztPDiO4sZvLm`uf(^w@i>g!WK3P z=V04Z7b-+;bkZY@N}+jpJg^iWErJQ4DrS@dmh^KPk*|BAKNsJ4hF1n=%wy(zDtm7a zjRXZJl~IcDOz=K9&eiuFRorO$?&pys!(~;4&`SzlFt%0(Y2R&kKHh>y7n|Ld`SQtk z~-c`^h7W8H+E&yu8<>k^_U`^VptUM}tOBhCi-rfvkx zbd9L{I5ujSKg+YciuVAjTG3hU@4vY^3Gws{03=Pzjvb#z7Mp6z*UVjfxAZ;6qRDZx zuc6vS4Vr$}&Ty;SiSD1@P(NFcqPrXTF2xr=YGFyrrQLa4l!ZOGqI;aCAN$2S#h}P_ zgW4`PVy}jzI^~)(rmv(Xs>fws2|{Wos-rp7`{A{=Z3!1w8>$_2)9?DtjaJ&Bw2V;= z4O5>^K95$5(^mBD>KU?fT_{H`if|a2qlP1Z74qf$D9wfa!Xx_PAEo#;V~o}dkxv5^ z?4l6Z_H>J!&i42FSxGj}+a)h^Q_3r=ET^S?SEr{nU?!^7z=1ed~Rfb1X4cm6tDfQ}?IGGBPqsFz%76s!NND+l0vo3l;{z zj!ttOQ{?iX(|WJ!^Gu9eo{?UASJHHcpkHUOm(_6VE3yzXIw^YD8w++D?nvL^CA!LX zmI3=hy&HIrmS>{hhh7AC0CwYgr*+S@L*Mbt|FRgt&UEDAZ!g0N>Dav6haVY4|KVXV zF#JDV%Ky0q!(WHT9sXUH{llgLC$t3q#WnrUwCBHj|IL!d7I{fz Date: Fri, 14 Aug 2026 06:55:27 +0000 Subject: [PATCH 09/15] feat: add native FP8 Sol-Attn for Wan --- docs/en/attention.md | 11 +- docs/zh/attention.md | 6 +- examples/wan_video/README.md | 21 +- telefuser/kernel/sol_attn/common/runtime.py | 5 +- telefuser/kernel/sol_attn/interface.py | 161 +++- telefuser/kernel/sol_attn/sm90/atoms.py | 17 +- telefuser/kernel/sol_attn/sm90/fwd.py | 14 +- telefuser/kernel/sol_attn/sm90/kernel.py | 15 +- telefuser/kernel/sol_attn/sm90/mainloop.py | 771 ++++++++++-------- .../kernel/sol_attn/triton_ref/preprocess.py | 205 ++++- telefuser/models/wan_video_dit.py | 16 +- telefuser/ops/attention/attention_impl.py | 34 +- telefuser/ops/fp8_attention.py | 178 +++- .../models/test_wan_video_sol_attention.py | 133 ++- 14 files changed, 1108 insertions(+), 479 deletions(-) diff --git a/docs/en/attention.md b/docs/en/attention.md index 4e58877..20d1ab4 100644 --- a/docs/en/attention.md +++ b/docs/en/attention.md @@ -71,6 +71,7 @@ class SparseAttentionConfig: sol_tau: float = 1.0 # Sol-Attn routing threshold sol_threshold_type: str = "diag" # "diag" or "exact" sol_kv_splits: int | str = "auto" # "auto", 1, 2, or 4 + sol_fp8: bool = False # Native FP8 Q/K/V Sol-Attn on SM90 ``` ## Calling Flow @@ -197,10 +198,12 @@ config = AttentionConfig.sol_attention() pipe_config.dit_config.attention_config = config ``` -Sol-Attn is used only for contiguous, noncausal BF16 self-attention with equal Q/K/V -shapes and head dimension 128. Unsupported calls, dense warmup layers or timesteps, -and kernel runtime failures fall back to the existing dense attention path. Ring/USP -also remains dense because its online merge requires log-sum-exp output. +Sol-Attn is used for contiguous, noncausal self-attention with equal Q/K/V shapes +and head dimension 128. BF16 is supported by the architecture-specific kernels; +SM90 additionally supports E4M3 Q/K/V with FP32 accumulation. Unsupported calls, +dense warmup layers or timesteps, and kernel runtime failures fall back to the +existing dense attention path. Ring/USP remains dense because its online merge +requires log-sum-exp output. ### QwenImagePipeline / ZImagePipeline diff --git a/docs/zh/attention.md b/docs/zh/attention.md index 475d648..bc8cdec 100644 --- a/docs/zh/attention.md +++ b/docs/zh/attention.md @@ -71,6 +71,7 @@ class SparseAttentionConfig: sol_tau: float = 1.0 # Sol-Attn 路由阈值 sol_threshold_type: str = "diag" # "diag" 或 "exact" sol_kv_splits: int | str = "auto" # "auto"、1、2 或 4 + sol_fp8: bool = False # SM90 原生 FP8 Q/K/V Sol-Attn ``` ## 调用流程 @@ -197,8 +198,9 @@ config = AttentionConfig.sol_attention() pipe_config.dit_config.attention_config = config ``` -Sol-Attn 仅用于连续、非因果、BF16、Q/K/V 形状相同且 head dimension 为 128 的 -self-attention。其他调用、dense 预热层/时间步以及内核运行失败都会回退到现有密集路径。 +Sol-Attn 用于连续、非因果、Q/K/V 形状相同且 head dimension 为 128 的 +self-attention。各架构内核支持 BF16，SM90 还支持使用 FP32 累加的 E4M3 Q/K/V。 +其他调用、dense 预热层/时间步以及内核运行失败都会回退到现有密集路径。 Ring/USP 需要 LSE 做在线合并，因此仍使用支持 LSE 的密集后端。 ### QwenImagePipeline / ZImagePipeline diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index a7d6daf..360a751 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -182,7 +182,7 @@ Attention choices: - `dense`: PyTorch SDPA - `sol`: Sol-Attn with dense warm-up and fallback calls -- `sol-fp8`: FP8 Sol-Attn mode; on H100 it keeps QKV in BF16 for the optimized CuTe kernel +- `sol-fp8`: native E4M3 Q/K/V Sol-Attn on H100 Quantization choices: @@ -221,11 +221,12 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ ``` `tf-kernel-fp8` is the FP8-GEMM implementation benchmarked below. On H100, -`sol-fp8` keeps QKV in BF16 so it can use the optimized CuTe SM90 Sol-Attn -mainloop; this avoids a slower quantize/dequantize detour because a native FP8 -CuTe Sol-Attn mainloop is not available yet. On architectures with a native FP8 -Sol-Attn backend, it quantizes post-RoPE Q/K/V per 64-token block and runs exact -FP8 QK/PV GEMMs with FP32 accumulation. The SOL tuning options apply to both modes. +`sol-fp8` quantizes post-RoPE Q/K/V to E4M3 and runs QK and PV through the CuTe +SM90 WGMMA mainloop. Q/K use one scale per 64-token block, V uses per-channel +scales and a K-major layout, and FP32 accumulators are used throughout. `auto` +selects four KV splits for long FP8 sequences to bound Hopper FP8 accumulation +error. Partial tiles are physically padded while the original sequence length +remains masked in the kernel. The Sol tuning options apply to both modes. The final log reports generation time, frames per second, and peak allocated and reserved CUDA memory. @@ -240,10 +241,10 @@ same generation interval. | Quantization | Attention | Throughput (frames/s) | Peak allocated (GiB) | | --- | --- | ---: | ---: | -| BF16 | Dense | 0.854 | 16.147 | -| BF16 | Sol-Attn | 1.105 | 17.023 | -| tf-kernel FP8 | Dense | 0.879 | 14.855 | -| tf-kernel FP8 | Sol-FP8 (H100 CuTe BF16 QKV) | 1.146 | 15.730 | +| BF16 | Dense | 0.8491 | 16.147 | +| BF16 | Sol-Attn | 1.1090 | 17.023 | +| tf-kernel FP8 | Dense | 0.8771 | 14.855 | +| tf-kernel FP8 | Sol-FP8 (native SM90 QKV) | 1.1879 | 15.730 | The benchmark prompt is: diff --git a/telefuser/kernel/sol_attn/common/runtime.py b/telefuser/kernel/sol_attn/common/runtime.py index 5182d0c..5c93cdd 100644 --- a/telefuser/kernel/sol_attn/common/runtime.py +++ b/telefuser/kernel/sol_attn/common/runtime.py @@ -4,11 +4,14 @@ def to_cute_tensor(tensor): + leading_dim = tensor.ndim - 1 + if tensor.stride(leading_dim) != 1: + leading_dim = next(i for i, stride in enumerate(tensor.stride()) if stride == 1) return from_dlpack( tensor, assumed_align=16, enable_tvm_ffi=True, - ).mark_layout_dynamic(leading_dim=tensor.ndim - 1) + ).mark_layout_dynamic(leading_dim=leading_dim) __all__ = ["to_cute_tensor"] diff --git a/telefuser/kernel/sol_attn/interface.py b/telefuser/kernel/sol_attn/interface.py index 6fc48a2..0800ca4 100644 --- a/telefuser/kernel/sol_attn/interface.py +++ b/telefuser/kernel/sol_attn/interface.py @@ -15,6 +15,22 @@ _compiled = {} +def _is_token_contiguous_bthd(x: torch.Tensor) -> bool: + batch, tokens, heads, head_dim = x.shape + return x.stride() == ( + heads * head_dim * tokens, + 1, + head_dim * tokens, + tokens, + ) + + +def _to_token_contiguous_bthd(x: torch.Tensor) -> torch.Tensor: + if _is_token_contiguous_bthd(x): + return x + return x.permute(0, 2, 3, 1).contiguous().permute(0, 3, 1, 2) + + def _validate_inputs( q, k, @@ -31,8 +47,9 @@ def _validate_inputs( raise TypeError("q, k, and v must use torch.bfloat16 or torch.float8_e4m3fn") if q.device.type != "cuda" or k.device != q.device or v.device != q.device: raise ValueError("q, k, and v must be on the same CUDA device") - if not (q.is_contiguous() and k.is_contiguous() and v.is_contiguous()): - raise ValueError("q, k, and v must be contiguous BTHD tensors") + v_layout_valid = v.is_contiguous() or (v.dtype == torch.float8_e4m3fn and _is_token_contiguous_bthd(v)) + if not (q.is_contiguous() and k.is_contiguous() and v_layout_valid): + raise ValueError("q and k must be contiguous BTHD; FP8 v may also be token-contiguous BTHD") if thresh_type not in ("diag", "exact"): raise ValueError("thresh_type must be 'diag' or 'exact'") if not isinstance(sink_tokens, int): @@ -70,17 +87,10 @@ def _backend_for_arch( """Select CuTe when specialized and available, otherwise Triton.""" if arch[0] < 8: - raise RuntimeError( - "Sol-Attn requires an NVIDIA GPU with compute capability >= 8.0; " - f"got SM{arch[0]}{arch[1]}" - ) + raise RuntimeError(f"Sol-Attn requires an NVIDIA GPU with compute capability >= 8.0; got SM{arch[0]}{arch[1]}") cute_backend = _CUTE_BACKENDS.get(arch) if cute_backend is not None: - available = ( - _cute_runtime_available() - if cute_available is None - else cute_available - ) + available = _cute_runtime_available() if cute_available is None else cute_available if available: return cute_backend return "triton" @@ -125,18 +135,20 @@ def _compile_sm90( kv_splits, sink_range, stream, + fp8_inputs, ): import cutlass.cute as cute from .sm90 import make_kernel - operator = make_kernel(tokens, kv_splits) + operator = make_kernel(tokens, kv_splits, fp8_inputs=fp8_inputs) args = _to_cute_tensors(tensors) compiled = cute.compile( operator, *args, scale, sink_range, + tokens, stream=stream, options="--enable-tvm-ffi", ) @@ -209,28 +221,68 @@ def _sol_attn_cute( kv_splits, sink_tokens, sink_start, + q_scale=None, + k_scale=None, + v_scale=None, ): - from .preprocess import prepare - batch, tokens, heads, _ = q.shape + fp8_inputs = q.dtype == torch.float8_e4m3fn with torch.cuda.device(q.device): - kc, vc, threshold = prepare( - q, - k, - v, - scale=scale, - tau=tau, - thresh_type=thresh_type, - ) - output = torch.empty_like(v) + if fp8_inputs: + from .triton_ref.preprocess import prepare_sm90_fp8 + + kc, vc, threshold, kc_scale = prepare_sm90_fp8( + q, + k, + v, + scale=scale, + tau=tau, + thresh_type=thresh_type, + tokens=tokens, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, + ) + else: + from .preprocess import prepare + + kc, vc, threshold = prepare( + q, + k, + v, + scale=scale, + tau=tau, + thresh_type=thresh_type, + ) + dummy_scale = torch.ones((1,), device=q.device, dtype=torch.float32) + q_scale = k_scale = v_scale = kc_scale = dummy_scale + if fp8_inputs and tokens % BLOCK_SIZE: + padded_tokens = ((tokens + BLOCK_SIZE - 1) // BLOCK_SIZE) * BLOCK_SIZE + q_padded = torch.zeros( + (batch, padded_tokens, heads, q.shape[-1]), + device=q.device, + dtype=q.dtype, + ) + k_padded = torch.zeros_like(q_padded) + q_padded[:, :tokens].copy_(q) + k_padded[:, :tokens].copy_(k) + v_storage = torch.zeros( + (batch, heads, v.shape[-1], padded_tokens), + device=v.device, + dtype=v.dtype, + ) + v_storage[..., :tokens].copy_(v.permute(0, 2, 3, 1)) + q, k = q_padded, k_padded + v = v_storage.permute(0, 3, 1, 2) + output = torch.empty(v.shape, device=v.device, dtype=torch.bfloat16) lse = torch.empty( - (batch, tokens, heads), + (batch, q.shape[1], heads), device=q.device, dtype=torch.float32, ) stream = _stream(q.device) - key = (q.device.index, arch, batch, tokens, heads, kv_splits) + key = (q.device.index, arch, batch, tokens, heads, kv_splits, q.dtype) if arch == (9, 0): if sink_tokens: @@ -242,17 +294,17 @@ def _sol_attn_cute( sink_range = sink_start_block | (sink_end_block << 16) else: sink_range = 0 - tensors = [q, k, v, output, kc, vc, threshold, lse] + tensors = [q, k, v, output, kc, vc, threshold, lse, q_scale, k_scale, v_scale, kc_scale] if kv_splits > 1: tensors.extend( [ torch.empty( - (batch, tokens, kv_splits * heads, 128), + (batch, q.shape[1], kv_splits * heads, 128), device=q.device, dtype=torch.bfloat16, ), torch.empty( - (batch, tokens, kv_splits * heads), + (batch, q.shape[1], kv_splits * heads), device=q.device, dtype=torch.float32, ), @@ -268,6 +320,7 @@ def _sol_attn_cute( kv_splits, sink_range, stream, + fp8_inputs, ) else: args = _to_cute_tensors(tensors) @@ -275,6 +328,7 @@ def _sol_attn_cute( *args, scale, sink_range, + tokens, stream=stream, ) elif arch == (10, 0): @@ -329,7 +383,7 @@ def _sol_attn_cute( sink_end_block, stream=stream, ) - return output + return output[:, :tokens] def sol_attn( @@ -356,37 +410,53 @@ def sol_attn( fp8_inputs = any(x.dtype == torch.float8_e4m3fn for x in (q, k, v)) if fp8_inputs: + if kv_splits not in (1, 2, 4): + raise ValueError("kv_splits must be 1, 2, or 4") if not all(x.dtype == torch.float8_e4m3fn for x in (q, k, v)): raise TypeError("q, k, and v must all use the same dtype") if any(scale is None for scale in (q_scale, k_scale, v_scale)): raise ValueError("FP8 Sol-Attn requires q_scale, k_scale, and v_scale") _validate_inputs(q, k, v, thresh_type, sink_tokens, sink_start) - if kv_splits != 1: - raise ValueError("FP8 Sol-Attn currently supports kv_splits=1") + arch = tuple(torch.cuda.get_device_capability(q.device)) + native_sm90_fp8 = arch == (9, 0) and _cute_runtime_available() blocks = (q.shape[1] + BLOCK_SIZE - 1) // BLOCK_SIZE - expected_scale_shape = (q.shape[0], blocks, q.shape[2]) - for name, tensor in (("q_scale", q_scale), ("k_scale", k_scale), ("v_scale", v_scale)): + expected_scale_shape = ( + (q.shape[0], blocks * BLOCK_SIZE, q.shape[2]) + if native_sm90_fp8 + else (q.shape[0], blocks, q.shape[2]) + ) + for name, tensor in (("q_scale", q_scale), ("k_scale", k_scale)): if tensor.shape != expected_scale_shape or tensor.device != q.device: raise ValueError(f"{name} must have shape {expected_scale_shape} on the Q/K/V device") if not tensor.is_contiguous(): raise ValueError(f"{name} must be contiguous") - # Use the production CuTe path on SM90 until its native FP8 mainloop - # is available; the pointer Triton fallback is substantially slower. - arch = tuple(torch.cuda.get_device_capability(q.device)) - if arch == (9, 0) and _cute_runtime_available(): - from telefuser.ops.fp8_attention import dequantize_fp8_per_block - - return sol_attn( - dequantize_fp8_per_block(q, q_scale, torch.bfloat16), - dequantize_fp8_per_block(k, k_scale, torch.bfloat16), - dequantize_fp8_per_block(v, v_scale, torch.bfloat16), + if native_sm90_fp8: + expected_v_scale_shape = (q.shape[0], q.shape[2], q.shape[3]) + if v_scale.shape != expected_v_scale_shape or v_scale.device != q.device: + raise ValueError("SM90 FP8 Sol-Attn requires v_scale with shape [B, H, D]") + if not v_scale.is_contiguous(): + raise ValueError("v_scale must be contiguous") + v = _to_token_contiguous_bthd(v) + scale = q.shape[-1] ** -0.5 if scale is None else float(scale) + return _sol_attn_cute( + q, + k, + v, + arch=arch, scale=scale, - tau=tau, + tau=float(tau), thresh_type=thresh_type, kv_splits=kv_splits, sink_tokens=sink_tokens, sink_start=sink_start, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, ) + if v_scale.shape != expected_scale_shape or v_scale.device != q.device: + raise ValueError(f"v_scale must have shape {expected_scale_shape} on the Q/K/V device") + if not v_scale.is_contiguous(): + raise ValueError("v_scale must be contiguous") from .triton_ref import sol_attn as triton_sol_attn return triton_sol_attn( @@ -445,6 +515,9 @@ def sol_attn( kv_splits=kv_splits, sink_tokens=sink_tokens, sink_start=sink_start, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, ) diff --git a/telefuser/kernel/sol_attn/sm90/atoms.py b/telefuser/kernel/sol_attn/sm90/atoms.py index e40d2a5..70add7b 100644 --- a/telefuser/kernel/sol_attn/sm90/atoms.py +++ b/telefuser/kernel/sol_attn/sm90/atoms.py @@ -7,15 +7,22 @@ from ._compat import sm90_utils -def make_pv_mma(tile_m: int = 64, tile_v: int = 128) -> cute.TiledMma: +def make_pv_mma( + tile_m: int = 64, + tile_v: int = 128, + a_dtype=cutlass.BFloat16, + b_dtype=cutlass.BFloat16, + source: str = "RS", +) -> cute.TiledMma: + b_major = "K" if b_dtype is cutlass.Float8E4M3FN else "MN" return sm90_utils.make_tiled_mma( - cutlass.BFloat16, + a_dtype, "K", - "MN", + b_major, tile_v, - source="RS", + source=source, atom_layout_mnk=(tile_m // 64, 1, 1), - b_dtype=cutlass.BFloat16, + b_dtype=b_dtype, acc_dtype=Float32, ) diff --git a/telefuser/kernel/sol_attn/sm90/fwd.py b/telefuser/kernel/sol_attn/sm90/fwd.py index 453ab94..b6fb585 100644 --- a/telefuser/kernel/sol_attn/sm90/fwd.py +++ b/telefuser/kernel/sol_attn/sm90/fwd.py @@ -1,5 +1,7 @@ """Hopper forward operators.""" +import math + import cuda.bindings.driver as cuda import cutlass import cutlass.cute as cute @@ -18,7 +20,7 @@ def __init__(self, *args, **kwargs): head_dim=self.tile_hdimv, tile_m=16, k_block_size=64, - log_max_splits=1 if self.sol_attn_num_splits == 2 else 2, + log_max_splits=int(math.log2(self.sol_attn_num_splits)), num_threads=128, stages=4, partial_dtype=cutlass.BFloat16, @@ -35,10 +37,15 @@ def __call__( vc: cute.Tensor, threshold: cute.Tensor, lse: cute.Tensor, + q_scale: cute.Tensor, + k_scale: cute.Tensor, + v_scale: cute.Tensor, + kc_scale: cute.Tensor, o_partial: cute.Tensor, lse_partial: cute.Tensor, softmax_scale: cutlass.Float32, sink_range: cutlass.Int32, + logical_tokens: cutlass.Int32, stream: cuda.CUstream = None, ): SolAttnMainloopSm90.__call__( @@ -51,8 +58,13 @@ def __call__( vc, threshold, lse_partial, + q_scale, + k_scale, + v_scale, + kc_scale, softmax_scale, sink_range, + logical_tokens, stream=stream, ) diff --git a/telefuser/kernel/sol_attn/sm90/kernel.py b/telefuser/kernel/sol_attn/sm90/kernel.py index c0b3fb7..007a41d 100644 --- a/telefuser/kernel/sol_attn/sm90/kernel.py +++ b/telefuser/kernel/sol_attn/sm90/kernel.py @@ -6,7 +6,7 @@ from .mainloop import SolAttnMainloopSm90 -def make_kernel(tokens: int, kv_splits: int): +def make_kernel(tokens: int, kv_splits: int, fp8_inputs: bool = False): blocks = (tokens + 63) // 64 full_groups, tail = divmod(blocks, 64) has_full_groups = tail == 0 @@ -25,22 +25,17 @@ def make_kernel(tokens: int, kv_splits: int): tile_n=64, num_stages=1, num_threads=128, - sol_attn_assume_lane_group_route_reduce=( - has_full_blocks and has_full_groups - ), + sol_attn_assume_lane_group_route_reduce=(has_full_blocks and has_full_groups), sol_attn_assume_full_k_exact_blocks=has_full_blocks, sol_attn_tail_exact_words1=0 < tail <= 8, sol_attn_assume_full_route_groups=has_full_groups, - sol_attn_static_num_full_route_groups=( - -1 if has_full_groups else full_groups - ), + sol_attn_static_num_full_route_groups=(-1 if has_full_groups else full_groups), sol_attn_static_tail_valid_count=(-1 if has_full_groups else tail), sol_attn_tail_physical_tile16=0 < tail <= 16, - sol_attn_exact_mask_seqlen_last_only=( - not has_full_blocks - ), + sol_attn_exact_mask_seqlen_last_only=(not has_full_blocks), sol_attn_tail16_lane_group_route_reduce=tail == 16, sol_attn_num_splits=kv_splits, + fp8_inputs=fp8_inputs, ) diff --git a/telefuser/kernel/sol_attn/sm90/mainloop.py b/telefuser/kernel/sol_attn/sm90/mainloop.py index 47ba69f..a76f47e 100644 --- a/telefuser/kernel/sol_attn/sm90/mainloop.py +++ b/telefuser/kernel/sol_attn/sm90/mainloop.py @@ -1,48 +1,47 @@ # Copyright (c) 2025, Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao. # SM90 (Hopper) forward pass for flash attention, extracted from flash_fwd.py. +from functools import partial from types import SimpleNamespace from typing import Callable, Optional -from functools import partial import cuda.bindings.driver as cuda - import cutlass import cutlass.cute as cute -from cutlass import Float32, Int32, const_expr -from cutlass.cute.nvgpu import cpasync, warpgroup -from cutlass.utils import LayoutEnum import cutlass.utils.hopper_helpers as sm90_utils_basic -from cutlass import pipeline -from cutlass.pipeline import pipeline_init_arrive, pipeline_init_wait +from cutlass import Float32, Int32, const_expr, pipeline from cutlass.base_dsl.arch import Arch +from cutlass.cute.nvgpu import cpasync, warpgroup +from cutlass.pipeline import pipeline_init_arrive, pipeline_init_wait +from cutlass.utils import LayoutEnum -from ._compat import copy_utils -from ._compat import layout_utils -from ._compat import sm90_utils - -from telefuser.kernel.sol_attn._vendor.flash_attn.cute.cute_dsl_utils import assume_tensor_aligned +from telefuser.kernel.sol_attn._vendor.flash_attn.cute import pipeline as pipeline_custom from telefuser.kernel.sol_attn._vendor.flash_attn.cute import utils -from telefuser.kernel.sol_attn._vendor.flash_attn.cute.mask import AttentionMask -from telefuser.kernel.sol_attn._vendor.flash_attn.cute.softmax import Softmax, apply_score_mod_inner -from telefuser.kernel.sol_attn._vendor.flash_attn.cute.seqlen_info import SeqlenInfoQK from telefuser.kernel.sol_attn._vendor.flash_attn.cute.block_info import BlockInfo from telefuser.kernel.sol_attn._vendor.flash_attn.cute.block_sparsity import BlockSparseTensors -from telefuser.kernel.sol_attn._vendor.flash_attn.cute import pipeline as pipeline_custom -from telefuser.kernel.sol_attn._vendor.flash_attn.cute.pack_gqa import PackGQA, pack_gqa_layout, make_packgqa_tiled_tma_atom +from telefuser.kernel.sol_attn._vendor.flash_attn.cute.cute_dsl_utils import assume_tensor_aligned +from telefuser.kernel.sol_attn._vendor.flash_attn.cute.flash_fwd import FlashAttentionForwardBase +from telefuser.kernel.sol_attn._vendor.flash_attn.cute.mask import AttentionMask from telefuser.kernel.sol_attn._vendor.flash_attn.cute.named_barrier import NamedBarrierFwd -from ._compat.cute_dsl_utils import ParamsBase +from telefuser.kernel.sol_attn._vendor.flash_attn.cute.pack_gqa import ( + PackGQA, + make_packgqa_tiled_tma_atom, + pack_gqa_layout, +) +from telefuser.kernel.sol_attn._vendor.flash_attn.cute.seqlen_info import SeqlenInfoQK +from telefuser.kernel.sol_attn._vendor.flash_attn.cute.softmax import Softmax, apply_score_mod_inner from telefuser.kernel.sol_attn._vendor.flash_attn.cute.tile_scheduler import ( - TileSchedulerArguments, - SingleTileScheduler, SingleTileLPTScheduler, + SingleTileScheduler, SingleTileVarlenScheduler, + TileSchedulerArguments, ) -from telefuser.kernel.sol_attn._vendor.flash_attn.cute.flash_fwd import FlashAttentionForwardBase -from . import atoms as sol_attn_atoms -from . import exact as exact_stream from telefuser.kernel.sol_attn.common import selector as sol_attn_selector +from . import atoms as sol_attn_atoms +from . import exact as exact_stream +from ._compat import copy_utils, layout_utils, sm90_utils +from ._compat.cute_dsl_utils import ParamsBase SOL_ATTN_ROUTE_MASK_BARRIER_ID = 7 SOL_ATTN_ROUTE_SUM_BARRIER_ID = 8 @@ -62,11 +61,15 @@ def __init__( sol_attn_exact_mask_seqlen_last_only: bool = False, sol_attn_tail16_lane_group_route_reduce: bool = False, sol_attn_num_splits: int = 1, + fp8_inputs: bool = False, **kwargs, ): super().__init__(*args, **kwargs) - self.qk_dtype = cutlass.BFloat16 - self.pv_dtype = self.dtype + self.fp8_inputs = fp8_inputs + self.qk_dtype = cutlass.Float8E4M3FN if fp8_inputs else cutlass.BFloat16 + self.p_dtype = cutlass.Float8E4M3FN if fp8_inputs else self.dtype + self.v_dtype = cutlass.Float8E4M3FN if fp8_inputs else self.dtype + self.fp8_probability_scale = 1.0 self.sol_attn_group_size = 64 self.sol_attn_group_words = 2 self.mma_pv_is_rs = True @@ -82,9 +85,7 @@ def __init__( self.sol_attn_assume_full_route_groups = sol_attn_assume_full_route_groups self.sol_attn_static_num_full_route_groups = sol_attn_static_num_full_route_groups self.sol_attn_static_tail_valid_count = sol_attn_static_tail_valid_count - self.sol_attn_tail_exact_words1 = ( - sol_attn_tail_exact_words1 and 0 < self.sol_attn_static_tail_valid_count <= 32 - ) + self.sol_attn_tail_exact_words1 = sol_attn_tail_exact_words1 and 0 < self.sol_attn_static_tail_valid_count <= 32 self.sol_attn_tail_route_mask_words1 = False self.sol_attn_tail_physical_tile16 = ( sol_attn_tail_physical_tile16 and 0 < self.sol_attn_static_tail_valid_count <= 16 @@ -109,25 +110,22 @@ def __init__( def _get_smem_layout_atom(self): sQ_layout_atom = warpgroup.make_smem_layout_atom( - sm90_utils_basic.get_smem_layout_atom( - LayoutEnum.ROW_MAJOR, self.qk_dtype, self.tile_hdim - ), + sm90_utils_basic.get_smem_layout_atom(LayoutEnum.ROW_MAJOR, self.qk_dtype, self.tile_hdim), self.qk_dtype, ) sK_layout_atom = sQ_layout_atom sV_layout_atom = warpgroup.make_smem_layout_atom( - sm90_utils_basic.get_smem_layout_atom( - LayoutEnum.ROW_MAJOR, self.pv_dtype, self.tile_hdimv - ), - self.pv_dtype, + sm90_utils_basic.get_smem_layout_atom(LayoutEnum.ROW_MAJOR, self.v_dtype, self.tile_hdimv), + self.v_dtype, + ) + sO_layout_atom = warpgroup.make_smem_layout_atom( + sm90_utils_basic.get_smem_layout_atom(LayoutEnum.ROW_MAJOR, self.dtype, self.tile_hdimv), + self.dtype, ) - sO_layout_atom = sV_layout_atom if not self.mma_pv_is_rs: sP_layout_atom = warpgroup.make_smem_layout_atom( - sm90_utils_basic.get_smem_layout_atom( - LayoutEnum.ROW_MAJOR, self.pv_dtype, self.tile_n - ), - self.pv_dtype, + sm90_utils_basic.get_smem_layout_atom(LayoutEnum.ROW_MAJOR, self.p_dtype, self.tile_n), + self.p_dtype, ) else: sP_layout_atom = None @@ -135,18 +133,21 @@ def _get_smem_layout_atom(self): def _get_tiled_mma(self): tiled_mma_qk = sm90_utils.make_tiled_mma( - cutlass.BFloat16, + self.qk_dtype, "K", "K", self.tile_n, source="SS", atom_layout_mnk=(self.tile_m // 64, 1, 1), - b_dtype=cutlass.BFloat16, + b_dtype=self.qk_dtype, acc_dtype=Float32, ) tiled_mma_pv = sol_attn_atoms.make_pv_mma( tile_m=self.tile_m, tile_v=self.tile_hdimv, + a_dtype=self.p_dtype, + b_dtype=self.v_dtype, + source="RS" if self.mma_pv_is_rs else "SS", ) return tiled_mma_qk, tiled_mma_pv @@ -175,27 +176,193 @@ def sol_attn_qk_gemm_zero_init( swap_AB, ) + @cute.jit + def sol_attn_pv_gemm( + self, + tiled_mma: cute.TiledMma, + acc: cute.Tensor, + tile_acc: Optional[cute.Tensor], + tCrA: cute.Tensor, + tCrB: cute.Tensor, + zero_init, + B_idx: Int32, + wg_wait: cutlass.Constexpr[int], + ): + if const_expr(self.fp8_inputs and self.sol_attn_num_splits == 1): + sm90_utils.gemm_w_idx( + tiled_mma, + tile_acc, + tCrA, + tCrB, + zero_init=True, + B_idx=B_idx, + wg_wait=0, + ) + acc.store(acc.load() + tile_acc.load()) + elif const_expr(self.fp8_inputs): + sm90_utils.gemm_w_idx( + tiled_mma, + acc, + tCrA, + tCrB, + zero_init=False, + B_idx=B_idx, + wg_wait=wg_wait, + ) + else: + sm90_utils.gemm_w_idx( + tiled_mma, + acc, + tCrA, + tCrB, + zero_init=zero_init, + B_idx=B_idx, + wg_wait=wg_wait, + ) + + @cute.jit + def sol_attn_scale_exact_scores( + self, + acc_S: cute.Tensor, + q_block: Int32, + n_block: Int32, + q_scale: cute.Tensor, + k_scale: cute.Tensor, + ): + """Apply one Q/K scale per N64 tile to an FP8 QK accumulator.""" + + acc_S_mn = layout_utils.reshape_acc_to_mn(acc_S) + factor = Float32(q_scale[q_block * Int32(self.tile_m)]) * Float32( + k_scale[n_block * Int32(self.tile_n)] + ) + for i in cutlass.range_constexpr(cute.size(acc_S_mn)): + acc_S_mn[i] = Float32(acc_S_mn[i]) * factor + + @cute.jit + def sol_attn_scale_route_scores( + self, + acc_S: cute.Tensor, + tScS_mn: cute.Tensor, + q_block: Int32, + route_n_block: Int32, + q_scale: cute.Tensor, + kc_scale: cute.Tensor, + ): + """Apply block-scaled Q and per-centroid K dequantization.""" + + acc_S_mn = layout_utils.reshape_acc_to_mn(acc_S) + q_factor = Float32(q_scale[q_block * Int32(self.tile_m)]) + for i in cutlass.range_constexpr(cute.size(acc_S_mn)): + col = tScS_mn[i][1] + factor = q_factor * Float32(kc_scale[route_n_block + col]) + acc_S_mn[i] = Float32(acc_S_mn[i]) * factor + + @cute.jit + def sol_attn_scale_route_probabilities( + self, + acc_S: cute.Tensor, + tScS_mn: cute.Tensor, + route_n_block: Int32, + seqlen: SeqlenInfoQK, + ): + """Convert centroid probabilities to equivalent block-sum weights.""" + + acc_S_mn = layout_utils.reshape_acc_to_mn(acc_S) + for i in cutlass.range_constexpr(cute.size(acc_S_mn)): + col = tScS_mn[i][1] + n_block = route_n_block + col + current_len = seqlen.seqlen_k - n_block * Int32(self.tile_n) + if current_len > Int32(self.tile_n): + current_len = Int32(self.tile_n) + if current_len < Int32(0): + current_len = Int32(0) + acc_S_mn[i] = Float32(acc_S_mn[i]) * Float32(current_len) + + @cute.jit + def sol_attn_convert_probability( + self, + src: cute.Tensor, + dst: cute.Tensor, + ): + """Convert post-softmax probabilities to the PV MMA operand dtype.""" + + if const_expr(self.fp8_inputs and self.mma_pv_is_rs): + for i in cutlass.range_constexpr(cute.size(src)): + dst[i] = cutlass.Float8E4M3FN(Float32(src[i]) * Float32(self.fp8_probability_scale)) + + tid = cute.arch.thread_idx()[0] % Int32(4) + values_u32 = cute.recast_tensor(dst, cutlass.Uint32) + for n in cutlass.range_constexpr(cute.size(values_u32, mode=[1])): + for k in cutlass.range_constexpr(cute.size(values_u32, mode=[2])): + for ii in cutlass.range_constexpr(0, 8, 4): + value0 = values_u32[ii // 2, n, k] + value1 = values_u32[ii // 2 + 1, n, k] + + send_high = 1 + if tid == Int32(1) or tid == Int32(2): + send_high = 0 + recv_lane = (Int32(0x3021) >> (tid * Int32(4))) & Int32(0xF) + value_a = value1 + if send_high == 0: + value_a = value0 + value_a = cute.arch.shuffle_sync_op(value_a, recv_lane, 0xFFFFFFFF, 7199) + + send_high = 1 - send_high + recv_lane = (Int32(0x2130) >> (tid * Int32(4))) & Int32(0xF) + value_b = value1 + if send_high == 0: + value_b = value0 + value_b = cute.arch.shuffle_sync_op(value_b, recv_lane, 0xFFFFFFFF, 7199) + + order0 = 0x5410 + order1 = 0x7632 + if send_high == 0: + order0 = 0x1054 + order1 = 0x3276 + values_u32[ii // 2, n, k] = cute.arch.prmt(value_a, value_b, order0) + values_u32[ii // 2 + 1, n, k] = cute.arch.prmt(value_a, value_b, order1) + elif const_expr(self.fp8_inputs): + for i in cutlass.range_constexpr(cute.size(src)): + dst[i] = cutlass.Float8E4M3FN(Float32(src[i]) * Float32(self.fp8_probability_scale)) + else: + utils.cvt_f16(src, dst) + + @cute.jit + def sol_attn_apply_v_scale( + self, + acc_O: cute.Tensor, + tiled_mma_pv: cute.TiledMma, + tidx: Int32, + v_scale: cute.Tensor, + ): + """Apply the per-channel V scale after all FP8 PV accumulations.""" + + thr_mma = tiled_mma_pv.get_slice(tidx) + cO = cute.make_identity_tensor((self.tile_m, self.tile_hdimv)) + taccOcO = layout_utils.reshape_acc_to_mn(thr_mma.partition_C(cO)) + acc_O_mn = layout_utils.reshape_acc_to_mn(acc_O) + for i in cutlass.range(cute.size(acc_O_mn), unroll_full=True): + col = taccOcO[i][1] + acc_O_mn[i] = Float32(acc_O_mn[i]) * Float32(v_scale[col]) / Float32(self.fp8_probability_scale) + def _get_shared_storage_cls(self): - sQ_struct, sK_struct = [ - cute.struct.Align[ - cute.struct.MemRange[self.qk_dtype, cute.cosize(layout)], self.buffer_align_bytes - ] - for layout in (self.sQ_layout, self.sK_layout) + sQ_elements = cute.cosize(self.sQ_layout) + if const_expr(self.fp8_inputs): + sQ_elements = max(sQ_elements, cute.cosize(self.sO_layout) * 2) + sQ_struct = cute.struct.Align[cute.struct.MemRange[self.qk_dtype, sQ_elements], self.buffer_align_bytes] + sK_struct = cute.struct.Align[ + cute.struct.MemRange[self.qk_dtype, cute.cosize(self.sK_layout)], self.buffer_align_bytes ] sV_struct = cute.struct.Align[ - cute.struct.MemRange[self.pv_dtype, cute.cosize(self.sV_layout)], + cute.struct.MemRange[self.v_dtype, cute.cosize(self.sV_layout)], self.buffer_align_bytes, ] cosize_sQV = max(cute.cosize(self.sQ_layout), cute.cosize(self.sV_layout)) - sQV_struct = cute.struct.Align[cute.struct.MemRange[self.pv_dtype, cosize_sQV], 1024] + sQV_struct = cute.struct.Align[cute.struct.MemRange[self.v_dtype, cosize_sQV], 1024] cosize_sP = cute.cosize(self.sP_layout) if const_expr(self.sP_layout is not None) else 0 - sP_struct = cute.struct.Align[cute.struct.MemRange[self.pv_dtype, cosize_sP], 1024] - route_mask_struct = cute.struct.Align[ - cute.struct.MemRange[Int32, 4], 16 - ] - route_sums_struct = cute.struct.Align[ - cute.struct.MemRange[Float32, 4 * self.tile_n], 16 - ] + sP_struct = cute.struct.Align[cute.struct.MemRange[self.p_dtype, cosize_sP], 1024] + route_mask_struct = cute.struct.Align[cute.struct.MemRange[Int32, 4], 16] + route_sums_struct = cute.struct.Align[cute.struct.MemRange[Float32, 4 * self.tile_n], 16] # 1 stage * 2 for Q pipeline (full + empty), self.num_stages*2 for K, self.num_stages*2 for V, mbar_ptr_Q_struct = cute.struct.MemRange[cutlass.Int64, 1 * 2] mbar_ptr_K_struct = cute.struct.MemRange[cutlass.Int64, self.num_stages * 2] @@ -264,7 +431,7 @@ def sol_attn_reduce_route_sums_guarded( ): """Fallback route-column reduction that ignores invalid q rows.""" - for off in cutlass.range_constexpr(self.tile_n): + for off in cutlass.range(self.tile_n, unroll_full=True): partial = Float32(0.0) for i in cutlass.range(cute.size(acc_S_mn), unroll_full=True): row = tScS_mn[i][0] @@ -502,12 +669,8 @@ def sol_attn_build_route_mask_from_acc( off1 = Int32(32) + lane route_col0 = route_col_offset + off0 route_col1 = route_col_offset + off1 - col_sum0 = Float32(route_sums[0, route_col0]) + Float32( - route_sums[1, route_col0] - ) - col_sum1 = Float32(route_sums[0, route_col1]) + Float32( - route_sums[1, route_col1] - ) + col_sum0 = Float32(route_sums[0, route_col0]) + Float32(route_sums[1, route_col0]) + col_sum1 = Float32(route_sums[0, route_col1]) + Float32(route_sums[1, route_col1]) col_sum0 += Float32(route_sums[2, route_col0]) col_sum1 += Float32(route_sums[2, route_col1]) col_sum0 += Float32(route_sums[3, route_col0]) @@ -547,12 +710,7 @@ def sol_attn_build_route_mask_from_acc( off = Int32(word * 32) + lane route_col = route_col_offset + off valid = True - if const_expr( - not ( - self.sol_attn_assume_full_route_groups - or assume_full_route_group - ) - ): + if const_expr(not (self.sol_attn_assume_full_route_groups or assume_full_route_group)): valid = off < valid_count exact = False if valid: @@ -572,10 +730,8 @@ def sol_attn_build_route_mask_from_acc( ) if sink_enabled: exact = exact or ( - group_start_n_block + off - >= sink_start_block - and group_start_n_block + off - < sink_end_block + group_start_n_block + off >= sink_start_block + and group_start_n_block + off < sink_end_block ) if const_expr(self.sol_attn_approx_colmask): column_mask = -Float32.inf @@ -588,21 +744,11 @@ def sol_attn_build_route_mask_from_acc( word_bits = Int32(0) if exact: word_bits = Int32(1) << lane - word_bits = word_bits | cute.arch.shuffle_sync_down( - word_bits, 16 - ) - word_bits = word_bits | cute.arch.shuffle_sync_down( - word_bits, 8 - ) - word_bits = word_bits | cute.arch.shuffle_sync_down( - word_bits, 4 - ) - word_bits = word_bits | cute.arch.shuffle_sync_down( - word_bits, 2 - ) - word_bits = word_bits | cute.arch.shuffle_sync_down( - word_bits, 1 - ) + word_bits = word_bits | cute.arch.shuffle_sync_down(word_bits, 16) + word_bits = word_bits | cute.arch.shuffle_sync_down(word_bits, 8) + word_bits = word_bits | cute.arch.shuffle_sync_down(word_bits, 4) + word_bits = word_bits | cute.arch.shuffle_sync_down(word_bits, 2) + word_bits = word_bits | cute.arch.shuffle_sync_down(word_bits, 1) if lane == Int32(0): if const_expr(word == 0): mask0 = word_bits @@ -619,9 +765,7 @@ def sol_attn_build_route_mask_from_acc( for off in cutlass.range_constexpr(self.sol_attn_group_size): route_col = route_col_offset + Int32(off) valid = True - if const_expr( - not (self.sol_attn_assume_full_route_groups or assume_full_route_group) - ): + if const_expr(not (self.sol_attn_assume_full_route_groups or assume_full_route_group)): valid = Int32(off) < valid_count col_sum = ( Float32(route_sums[0, route_col]) @@ -640,16 +784,12 @@ def sol_attn_build_route_mask_from_acc( ) if sink_enabled: exact = exact or ( - group_start_n_block + Int32(off) - >= sink_start_block - and group_start_n_block + Int32(off) - < sink_end_block + group_start_n_block + Int32(off) >= sink_start_block + and group_start_n_block + Int32(off) < sink_end_block ) if exact: - mask0, mask1, mask2, mask3 = ( - sol_attn_selector.sol_attn_set_exact_bit( - mask0, mask1, mask2, mask3, Int32(off) - ) + mask0, mask1, mask2, mask3 = sol_attn_selector.sol_attn_set_exact_bit( + mask0, mask1, mask2, mask3, Int32(off) ) return mask0, mask1, mask2, mask3 @@ -685,19 +825,14 @@ def sol_attn_mask_route_approx_columns( valid = group_col >= Int32(0) if valid: valid = group_col < valid_count - elif const_expr( - not (self.sol_attn_assume_full_route_groups or assume_full_route_group) - ): + elif const_expr(not (self.sol_attn_assume_full_route_groups or assume_full_route_group)): valid = col < valid_count exact = False if valid: route_mask_words = self.sol_attn_group_words if const_expr(route_mask_words_override != 0): route_mask_words = route_mask_words_override - if const_expr( - self.sol_attn_tail_route_mask_words1 - and not assume_full_route_group - ): + if const_expr(self.sol_attn_tail_route_mask_words1 and not assume_full_route_group): route_mask_words = 1 exact = sol_attn_selector.sol_attn_test_exact_bit_limited_words( mask0, mask1, mask2, mask3, group_col, route_mask_words @@ -771,9 +906,7 @@ def sol_attn_apply_route_current_lens_to_row_sum( """Correct route approx denominator for VC tiles that are block sums.""" acc_S_mn = layout_utils.reshape_acc_to_mn(acc_S) - last_n_block = ( - (seqlen.seqlen_k + Int32(self.tile_n - 1)) // Int32(self.tile_n) - ) - Int32(1) + last_n_block = ((seqlen.seqlen_k + Int32(self.tile_n - 1)) // Int32(self.tile_n)) - Int32(1) tail_len = seqlen.seqlen_k - last_n_block * Int32(self.tile_n) for r in cutlass.range(cute.size(softmax.row_sum), unroll_full=True): extra = Float32(0.0) @@ -807,9 +940,7 @@ def sol_attn_apply_route_current_lens_to_row_sum_fast( ): """Fast denominator correction for full-length route groups.""" - last_n_block = ( - (seqlen.seqlen_k + Int32(self.tile_n - 1)) // Int32(self.tile_n) - ) - Int32(1) + last_n_block = ((seqlen.seqlen_k + Int32(self.tile_n - 1)) // Int32(self.tile_n)) - Int32(1) tail_len = seqlen.seqlen_k - last_n_block * Int32(self.tile_n) group_end = group_start_n_block + valid_count full_len_group = (tail_len == Int32(self.tile_n)) or (group_end <= last_n_block) @@ -846,8 +977,13 @@ def __call__( mVC: cute.Tensor, mGlobalThresh: cute.Tensor, mLSE: Optional[cute.Tensor], + mQScale: cute.Tensor, + mKScale: cute.Tensor, + mVScale: cute.Tensor, + mKCScale: cute.Tensor, softmax_scale: Float32, sink_range: Int32, + logical_tokens: Int32, stream: cuda.CUstream = None, ): """Configure and launch the Hopper Sol-Attn kernel.""" @@ -866,34 +1002,27 @@ def __call__( aux_tensors = None self.varlen_q = mCuSeqlensQ is not None or mSeqUsedQ is not None - mQ, mK, mV, mO, mKC, mVC, mGlobalThresh = [ + mQ, mK, mV, mO, mKC, mVC, mGlobalThresh, mQScale, mKScale, mVScale, mKCScale = [ assume_tensor_aligned(t) - for t in (mQ, mK, mV, mO, mKC, mVC, mGlobalThresh) + for t in (mQ, mK, mV, mO, mKC, mVC, mGlobalThresh, mQScale, mKScale, mVScale, mKCScale) ] if const_expr(piecewise_k is not None): - piecewise_k, piecewise_v = [ - assume_tensor_aligned(t) for t in (piecewise_k, piecewise_v) - ] + piecewise_k, piecewise_v = [assume_tensor_aligned(t) for t in (piecewise_k, piecewise_v)] SOL_ATTN_BTHD_TRANSPOSE = [1, 3, 2, 0] SOL_ATTN_BNH_TRANSPOSE = [1, 2, 0] - mQ, mK, mV, mO, mKC, mVC = [ - layout_utils.select(t, SOL_ATTN_BTHD_TRANSPOSE) - for t in (mQ, mK, mV, mO, mKC, mVC) - ] - mGlobalThresh = layout_utils.select( - mGlobalThresh, SOL_ATTN_BNH_TRANSPOSE - ) + mQ, mK, mV, mO, mKC, mVC = [layout_utils.select(t, SOL_ATTN_BTHD_TRANSPOSE) for t in (mQ, mK, mV, mO, mKC, mVC)] + mGlobalThresh = layout_utils.select(mGlobalThresh, SOL_ATTN_BNH_TRANSPOSE) + if const_expr(self.fp8_inputs): + mQScale, mKScale, mKCScale = [ + layout_utils.select(t, SOL_ATTN_BNH_TRANSPOSE) for t in (mQScale, mKScale, mKCScale) + ] + mVScale = layout_utils.select(mVScale, [2, 1, 0]) if const_expr(piecewise_k is not None): piecewise_k, piecewise_v = [ - layout_utils.select(t, SOL_ATTN_BTHD_TRANSPOSE) - for t in (piecewise_k, piecewise_v) + layout_utils.select(t, SOL_ATTN_BTHD_TRANSPOSE) for t in (piecewise_k, piecewise_v) ] LSE_layout_transpose = [1, 2, 0] - mLSE = ( - layout_utils.select(mLSE, LSE_layout_transpose) - if const_expr(mLSE is not None) - else None - ) + mLSE = layout_utils.select(mLSE, LSE_layout_transpose) if const_expr(mLSE is not None) else None tiled_mma_qk, tiled_mma_pv = self._get_tiled_mma() self.num_mma_threads = tiled_mma_qk.size @@ -906,9 +1035,7 @@ def __call__( self.num_producer_threads = 32 self.num_Q_load_threads = self.num_threads_per_warp_group # If not TMA_Q self.num_epilogue_threads = self.num_mma_threads - self.num_mma_regs, self.num_producer_regs = {1: (256, 56), 2: (240, 24), 3: (160, 32)}[ - self.num_wg_mma - ] + self.num_mma_regs, self.num_producer_regs = {1: (256, 56), 2: (240, 24), 3: (160, 32)}[self.num_wg_mma] self.use_block_sparsity = cutlass.const_expr(blocksparse_tensors is not None) self.has_piecewise_kv = cutlass.const_expr(piecewise_k is not None) if const_expr(self.use_block_sparsity): @@ -917,17 +1044,13 @@ def __call__( raise NotImplementedError("one-warpgroup SOL_ATTN path does not support piecewise KV") self.use_scheduler_barrier = self.num_wg_mma == 2 - self.use_tma_Q = self.arch >= Arch.sm_90 and not ( - self.pack_gqa and self.tile_m % self.qhead_per_kvhead != 0 - ) + self.use_tma_Q = self.arch >= Arch.sm_90 and not (self.pack_gqa and self.tile_m % self.qhead_per_kvhead != 0) if const_expr(not self.use_tma_Q): raise NotImplementedError("one-warpgroup SOL_ATTN path requires TMA Q/O") # FP32 split partials require a direct register-to-global epilogue. # A BF16 split partial matches V/O dtype and can reuse the shared-memory # plus TMA-O epilogue. - self.use_tma_O = ( - self.sol_attn_num_splits == 1 or mO.element_type == self.dtype - ) + self.use_tma_O = self.sol_attn_num_splits == 1 or mO.element_type == self.dtype # Producer needs more registers when doing cp.async Q or KV loads if const_expr(self.num_wg_mma == 2 and (not self.use_tma_Q or not self.use_tma_KV)): self.num_mma_regs, self.num_producer_regs = 224, 40 @@ -936,18 +1059,26 @@ def __call__( self.rescale_O_before_gemm = False self._setup_attributes() # TODO: we prob don't need most of what's in _setup_attributes - self.sQ_layout, self.sK_layout, self.sV_layout, self.sO_layout = [ + self.sQ_layout, self.sK_layout = [ sm90_utils.make_smem_layout(mX.element_type, LayoutEnum.ROW_MAJOR, shape, stage) for mX, shape, stage in [ (mQ, (self.tile_m, self.tile_hdim), None), (mK, (self.tile_n, self.tile_hdim), self.num_stages), - (mV, (self.tile_n, self.tile_hdimv), self.num_stages), - # sO always holds the BF16 PV epilogue tile. Split-KV's - # global mO is an FP32 partial workspace, so derive this - # shared-memory layout from V instead of global O. - (mV, (self.tile_m, self.tile_hdimv), None), ] ] + v_layout = LayoutEnum.COL_MAJOR if const_expr(self.fp8_inputs) else LayoutEnum.ROW_MAJOR + self.sV_layout = sm90_utils.make_smem_layout( + mV.element_type, + v_layout, + (self.tile_n, self.tile_hdimv), + self.num_stages, + ) + self.sO_layout = sm90_utils.make_smem_layout( + self.dtype, + LayoutEnum.ROW_MAJOR, + (self.tile_m, self.tile_hdimv), + None, + ) self.sP_layout = None if const_expr(not self.mma_pv_is_rs): self.sP_layout = sm90_utils.make_smem_layout( @@ -1042,9 +1173,7 @@ def __call__( if const_expr(self.use_tma_O): mO_tma = mO_og if const_expr(self.pack_gqa) else mO if const_expr(self.varlen_q): - mO_tma = copy_utils.create_ragged_tensor_for_tma( - mO_tma, ragged_dim=0, ptr_shift=True - ) + mO_tma = copy_utils.create_ragged_tensor_for_tma(mO_tma, ragged_dim=0, ptr_shift=True) tma_atom_O, tma_tensor_O = make_tiled_tma_atom_fn( gmem_tiled_copy_O, mO_tma, @@ -1055,20 +1184,14 @@ def __call__( TileScheduler = SingleTileVarlenScheduler else: TileScheduler = ( - SingleTileScheduler - if const_expr(not self.is_causal or self.is_local) - else SingleTileLPTScheduler + SingleTileScheduler if const_expr(not self.is_causal or self.is_local) else SingleTileLPTScheduler ) tile_sched_args = TileSchedulerArguments( cute.ceil_div(cute.size(mQ.shape[0]), self.tile_m), cute.size(mQ.shape[2]), - cute.size(mQ.shape[3]) - if const_expr(mCuSeqlensQ is None) - else cute.size(mCuSeqlensQ.shape[0] - 1), + cute.size(mQ.shape[3]) if const_expr(mCuSeqlensQ is None) else cute.size(mCuSeqlensQ.shape[0] - 1), self.sol_attn_num_splits, - cute.size(mK.shape[0]) - if const_expr(mPageTable is None) - else mK.shape[0] * mPageTable.shape[1], + cute.size(mK.shape[0]) if const_expr(mPageTable is None) else mK.shape[0] * mPageTable.shape[1], mQ.shape[1], mV.shape[1], total_q=cute.size(mQ.shape[0]) @@ -1085,14 +1208,10 @@ def __call__( ) tile_sched_params = TileScheduler.to_underlying_arguments(tile_sched_args) grid_dim = TileScheduler.get_grid_shape(tile_sched_params) - softmax_scale_log2, softmax_scale = utils.compute_softmax_scale_log2( - softmax_scale, self.score_mod - ) + softmax_scale_log2, softmax_scale = utils.compute_softmax_scale_log2(softmax_scale, self.score_mod) window_size_left = Int32(window_size_left) if window_size_left is not None else None window_size_right = Int32(window_size_right) if window_size_right is not None else None - fastdiv_mods = utils.compute_fastdiv_mods( - mQ, mK, self.qhead_per_kvhead, self.pack_gqa, aux_tensors, mPageTable - ) + fastdiv_mods = utils.compute_fastdiv_mods(mQ, mK, self.qhead_per_kvhead, self.pack_gqa, aux_tensors, mPageTable) self.kernel( tma_tensor_Q if const_expr(self.use_tma_Q) else mQ, @@ -1105,6 +1224,10 @@ def __call__( tma_tensor_O if const_expr(self.use_tma_O) else mO, mGlobalThresh, mLSE, + mQScale, + mKScale, + mVScale, + mKCScale, mCuSeqlensQ, mCuSeqlensK, mSeqUsedQ, @@ -1121,6 +1244,7 @@ def __call__( softmax_scale_log2, softmax_scale, sink_range, + logical_tokens, window_size_left, window_size_right, learnable_sink, @@ -1160,6 +1284,10 @@ def kernel( mO: cute.Tensor, mGlobalThresh: cute.Tensor, mLSE: Optional[cute.Tensor], + mQScale: cute.Tensor, + mKScale: cute.Tensor, + mVScale: cute.Tensor, + mKCScale: cute.Tensor, mCuSeqlensQ: Optional[cute.Tensor], mCuSeqlensK: Optional[cute.Tensor], mSeqUsedQ: Optional[cute.Tensor], @@ -1176,6 +1304,7 @@ def kernel( softmax_scale_log2: Float32, softmax_scale: Optional[Float32], sink_range: Int32, + logical_tokens: Int32, window_size_left: Optional[Int32], window_size_right: Optional[Int32], learnable_sink: Optional[cute.Tensor], @@ -1290,9 +1419,7 @@ def kernel( if const_expr(not self.Q_in_regs): sV = storage.sV.get_tensor(sV_layout.outer, swizzle=sV_layout.inner) else: - sV = storage.sQ.get_tensor( - sV_layout.outer, swizzle=sV_layout.inner, dtype=mV.element_type - ) + sV = storage.sQ.get_tensor(sV_layout.outer, swizzle=sV_layout.inner, dtype=mV.element_type) # Transpose view of V to tensor with layout (head_dim_v, tile_n) for tiled mma sVt = layout_utils.transpose_view(sV) sP = None @@ -1300,9 +1427,7 @@ def kernel( sP = storage.sP.get_tensor(sP_layout.outer, swizzle=sP_layout.inner) # reuse sQ's data iterator sO = storage.sQ.get_tensor(sO_layout.outer, swizzle=sO_layout.inner, dtype=self.dtype) - route_mask = storage.route_mask.get_tensor( - cute.make_layout((4,)) - ) + route_mask = storage.route_mask.get_tensor(cute.make_layout((4,))) route_sums = storage.route_sums.get_tensor(cute.make_layout((4, self.tile_n))) block_info = BlockInfo( @@ -1317,10 +1442,8 @@ def kernel( ) SeqlenInfoCls = partial( SeqlenInfoQK.create, - seqlen_q_static=mQ.shape[0] if const_expr(not self.pack_gqa) else mQ.shape[0][1], - seqlen_k_static=mK.shape[0] - if const_expr(mPageTable is None) - else mK.shape[0] * mPageTable.shape[1], + seqlen_q_static=(logical_tokens if const_expr(not self.pack_gqa) else mQ.shape[0][1]), + seqlen_k_static=(logical_tokens if const_expr(mPageTable is None) else mK.shape[0] * mPageTable.shape[1]), mCuSeqlensQ=mCuSeqlensQ, mCuSeqlensK=mCuSeqlensK, mSeqUsedQ=mSeqUsedQ, @@ -1371,6 +1494,10 @@ def kernel( AttentionMaskCls, TileSchedulerCls, mGlobalThresh, + mQScale, + mKScale, + mVScale, + mKCScale, route_mask, route_sums, softmax_scale_log2, @@ -1403,9 +1530,7 @@ def epilogue_one_warpgroup_tma_o( barrier_id=int(NamedBarrierFwd.Epilogue), number_of_threads=self.num_epilogue_threads, ) - smem_copy_atom_O = utils.get_smem_store_atom( - self.arch.major * 10 + self.arch.minor, self.dtype - ) + smem_copy_atom_O = utils.get_smem_store_atom(self.arch.major * 10 + self.arch.minor, self.dtype) smem_thr_copy_O = cute.make_tiled_copy_C(smem_copy_atom_O, tiled_mma).get_slice(tidx) taccOrO = smem_thr_copy_O.retile(rO) taccOsO = smem_thr_copy_O.partition_D(sO) @@ -1415,9 +1540,7 @@ def epilogue_one_warpgroup_tma_o( if const_expr(mLSE is not None): mLSE_cur = mLSE[None, head_idx, batch_idx] gLSE = cute.local_tile(mLSE_cur, (self.tile_m,), (m_block,)) - gLSE_expanded_layout = cute.append( - gLSE.layout, cute.make_layout((self.tile_hdimv,), stride=(0,)) - ) + gLSE_expanded_layout = cute.append(gLSE.layout, cute.make_layout((self.tile_hdimv,), stride=(0,))) gLSE_expanded = cute.make_tensor(gLSE.iterator, gLSE_expanded_layout) thr_mma = tiled_mma.get_slice(tidx) taccOgLSE = layout_utils.reshape_acc_to_mn(thr_mma.partition_C(gLSE_expanded)) @@ -1425,10 +1548,7 @@ def epilogue_one_warpgroup_tma_o( t0accOcO = layout_utils.reshape_acc_to_mn(thr_mma.get_slice(0).partition_C(cO)) if taccOcO[0][1] == 0: for m in cutlass.range_constexpr(cute.size(taccOgLSE.shape[1])): - if ( - t0accOcO[m, 0][0] - < seqlen.seqlen_q - m_block * self.tile_m - taccOcO[0][0] - ): + if t0accOcO[m, 0][0] < seqlen.seqlen_q - m_block * self.tile_m - taccOcO[0][0]: taccOgLSE[m, 0] = lse[m] mO_cur = mO[None, None, head_idx, batch_idx] @@ -1438,9 +1558,7 @@ def epilogue_one_warpgroup_tma_o( number_of_threads=self.num_epilogue_threads, ) gO = cute.local_tile(mO_cur, (self.tile_m, self.tile_hdimv), (m_block, 0)) - store_O, _, _ = copy_utils.tma_get_copy_fn( - tma_atom_O, 0, cute.make_layout(1), sO, gO, single_stage=True - ) + store_O, _, _ = copy_utils.tma_get_copy_fn(tma_atom_O, 0, cute.make_layout(1), sO, gO, single_stage=True) warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx()) if warp_idx == Int32(0): store_O() @@ -1469,9 +1587,7 @@ def epilogue_one_warpgroup_split_partial( """ mO_cur = mO[None, None, partial_head_idx, batch_idx] - gO = cute.local_tile( - mO_cur, (self.tile_m, self.tile_hdimv), (m_block, 0) - ) + gO = cute.local_tile(mO_cur, (self.tile_m, self.tile_hdimv), (m_block, 0)) copy_atom = cute.make_copy_atom( cute.nvgpu.CopyUniversalOp(), Float32, @@ -1486,29 +1602,16 @@ def epilogue_one_warpgroup_split_partial( mLSE_cur = mLSE[None, partial_head_idx, batch_idx] gLSE = cute.local_tile(mLSE_cur, (self.tile_m,), (m_block,)) - gLSE_expanded_layout = cute.append( - gLSE.layout, cute.make_layout((self.tile_hdimv,), stride=(0,)) - ) - gLSE_expanded = cute.make_tensor( - gLSE.iterator, gLSE_expanded_layout - ) + gLSE_expanded_layout = cute.append(gLSE.layout, cute.make_layout((self.tile_hdimv,), stride=(0,))) + gLSE_expanded = cute.make_tensor(gLSE.iterator, gLSE_expanded_layout) thr_mma = tiled_mma.get_slice(tidx) - taccOgLSE = layout_utils.reshape_acc_to_mn( - thr_mma.partition_C(gLSE_expanded) - ) + taccOgLSE = layout_utils.reshape_acc_to_mn(thr_mma.partition_C(gLSE_expanded)) cO = cute.make_identity_tensor((self.tile_m, self.tile_hdimv)) taccOcO = layout_utils.reshape_acc_to_mn(thr_mma.partition_C(cO)) - t0accOcO = layout_utils.reshape_acc_to_mn( - thr_mma.get_slice(0).partition_C(cO) - ) + t0accOcO = layout_utils.reshape_acc_to_mn(thr_mma.get_slice(0).partition_C(cO)) if taccOcO[0][1] == 0: for m in cutlass.range_constexpr(cute.size(taccOgLSE.shape[1])): - if ( - t0accOcO[m, 0][0] - < seqlen.seqlen_q - - m_block * self.tile_m - - taccOcO[0][0] - ): + if t0accOcO[m, 0][0] < seqlen.seqlen_q - m_block * self.tile_m - taccOcO[0][0]: taccOgLSE[m, 0] = lse[m] @cute.jit @@ -1543,6 +1646,10 @@ def mma_one_warpgroup_sol_attn_route_tma( AttentionMaskCls: Callable, TileSchedulerCls: cutlass.Constexpr[Callable], mGlobalThresh: cute.Tensor, + mQScale: cute.Tensor, + mKScale: cute.Tensor, + mVScale: cute.Tensor, + mKCScale: cute.Tensor, route_mask: cute.Tensor, route_sums: cute.Tensor, softmax_scale_log2: Float32, @@ -1560,38 +1667,32 @@ def mma_one_warpgroup_sol_attn_route_tma( else: q_producer_phase = Int32(1) q_consumer_phase = Int32(0) - kv_producer_state = pipeline.make_pipeline_state( - pipeline.PipelineUserType.Producer, self.num_stages - ) - kv_consumer_state = pipeline.make_pipeline_state( - pipeline.PipelineUserType.Consumer, self.num_stages - ) + kv_producer_state = pipeline.make_pipeline_state(pipeline.PipelineUserType.Producer, self.num_stages) + kv_consumer_state = pipeline.make_pipeline_state(pipeline.PipelineUserType.Consumer, self.num_stages) tile_scheduler = TileSchedulerCls() work_tile = tile_scheduler.initial_work_tile_info() if work_tile.is_valid_tile: m_block, head_idx, batch_idx, split_idx = work_tile.tile_idx partial_head_idx = ( - head_idx - + split_idx * mQ.shape[2] - if const_expr(self.sol_attn_num_splits > 1) - else head_idx + head_idx + split_idx * mQ.shape[2] if const_expr(self.sol_attn_num_splits > 1) else head_idx ) seqlen = SeqlenInfoCls(batch_idx) - head_idx_kv = ( - head_idx // self.qhead_per_kvhead - if const_expr(not self.pack_gqa) - else head_idx - ) + head_idx_kv = head_idx // self.qhead_per_kvhead if const_expr(not self.pack_gqa) else head_idx mQ_cur = seqlen.offset_batch_Q(mQ, batch_idx, dim=3)[None, None, head_idx] - mK_cur = seqlen.offset_batch_K(mK, batch_idx, dim=3)[ - None, None, head_idx_kv - ] - mV_cur = seqlen.offset_batch_K(mV, batch_idx, dim=3)[ - None, None, head_idx_kv - ] + mK_cur = seqlen.offset_batch_K(mK, batch_idx, dim=3)[None, None, head_idx_kv] + mV_cur = seqlen.offset_batch_K(mV, batch_idx, dim=3)[None, None, head_idx_kv] mKC_cur = mKC[None, None, head_idx_kv, batch_idx] mVC_cur = mVC[None, None, head_idx_kv, batch_idx] + mQScale_cur = None + mKScale_cur = None + mVScale_cur = None + mKCScale_cur = None + if const_expr(self.fp8_inputs): + mQScale_cur = mQScale[None, head_idx, batch_idx] + mKScale_cur = mKScale[None, head_idx_kv, batch_idx] + mVScale_cur = mVScale[None, head_idx_kv, batch_idx] + mKCScale_cur = mKCScale[None, head_idx_kv, batch_idx] gQ = cute.local_tile(mQ_cur, (self.tile_m, self.tile_hdim), (m_block, 0)) gK = cute.local_tile(mK_cur, (self.tile_n, self.tile_hdim), (None, 0)) @@ -1599,38 +1700,22 @@ def mma_one_warpgroup_sol_attn_route_tma( gKC = cute.local_tile(mKC_cur, (self.tile_n, self.tile_hdim), (None, 0)) gVC = cute.local_tile(mVC_cur, (self.tile_n, self.tile_hdimv), (None, 0)) - load_Q, _, _ = copy_utils.tma_get_copy_fn( - tma_atom_Q, 0, cute.make_layout(1), gQ, sQ, single_stage=True - ) - tma_load_K_fn, _, _ = copy_utils.tma_get_copy_fn( - tma_atom_K, 0, cute.make_layout(1), gK, sK - ) + load_Q, _, _ = copy_utils.tma_get_copy_fn(tma_atom_Q, 0, cute.make_layout(1), gQ, sQ, single_stage=True) + tma_load_K_fn, _, _ = copy_utils.tma_get_copy_fn(tma_atom_K, 0, cute.make_layout(1), gK, sK) tma_load_K_fn = copy_utils.tma_producer_copy_fn(tma_load_K_fn, pipeline_k) - tma_load_V_fn, _, _ = copy_utils.tma_get_copy_fn( - tma_atom_V, 0, cute.make_layout(1), gV, sV - ) + tma_load_V_fn, _, _ = copy_utils.tma_get_copy_fn(tma_atom_V, 0, cute.make_layout(1), gV, sV) tma_load_V_fn = copy_utils.tma_producer_copy_fn(tma_load_V_fn, pipeline_v) - tma_load_KC_fn, _, _ = copy_utils.tma_get_copy_fn( - tma_atom_KC, 0, cute.make_layout(1), gKC, sK - ) - tma_load_KC_fn = copy_utils.tma_producer_copy_fn( - tma_load_KC_fn, pipeline_k - ) - tma_load_VC_fn, _, _ = copy_utils.tma_get_copy_fn( - tma_atom_VC, 0, cute.make_layout(1), gVC, sV - ) - tma_load_VC_fn = copy_utils.tma_producer_copy_fn( - tma_load_VC_fn, pipeline_v - ) + tma_load_KC_fn, _, _ = copy_utils.tma_get_copy_fn(tma_atom_KC, 0, cute.make_layout(1), gKC, sK) + tma_load_KC_fn = copy_utils.tma_producer_copy_fn(tma_load_KC_fn, pipeline_k) + tma_load_VC_fn, _, _ = copy_utils.tma_get_copy_fn(tma_atom_VC, 0, cute.make_layout(1), gVC, sV) + tma_load_VC_fn = copy_utils.tma_producer_copy_fn(tma_load_VC_fn, pipeline_v) if warp_idx == Int32(0): pipeline_q.producer_acquire_w_index_phase(0, q_producer_phase) load_Q(tma_bar_ptr=pipeline_q.sync_object_full.get_barrier(0)) pipeline_q.consumer_wait_w_index_phase(0, q_consumer_phase) - warp_group_thread_layout = cute.make_layout( - 1, stride=self.num_threads_per_warp_group - ) + warp_group_thread_layout = cute.make_layout(1, stride=self.num_threads_per_warp_group) thr_mma_qk = tiled_mma_qk.get_slice(tidx) wg_mma_qk = tiled_mma_qk.get_slice(warp_group_thread_layout(Int32(0))) wg_mma_pv = tiled_mma_pv.get_slice(warp_group_thread_layout(Int32(0))) @@ -1647,23 +1732,32 @@ def mma_one_warpgroup_sol_attn_route_tma( acc_O, tOrP, tOrVt = sm90_utils.partition_fragment_ABC( wg_mma_pv, (self.tile_m, self.tile_hdimv, self.tile_n), sP, sVt ) - mma_pv_fn = partial(sm90_utils.gemm_w_idx, tiled_mma_pv, acc_O, tOrP, tOrVt) + tile_acc_O = ( + cute.make_rmem_tensor_like(acc_O, Float32) + if const_expr(self.fp8_inputs and self.sol_attn_num_splits == 1) + else None + ) + mma_pv_fn = partial( + self.sol_attn_pv_gemm, + tiled_mma_pv, + acc_O, + tile_acc_O, + tOrP, + tOrVt, + ) smem_copy_atom_P = utils.get_smem_store_atom( - self.arch.major * 10 + self.arch.minor, self.dtype + self.arch.major * 10 + self.arch.minor, + self.p_dtype, ) - smem_thr_copy_P = cute.make_tiled_copy_C( - smem_copy_atom_P, tiled_mma_qk - ).get_slice(tidx) + smem_thr_copy_P = cute.make_tiled_copy_C(smem_copy_atom_P, tiled_mma_qk).get_slice(tidx) tPsP = smem_thr_copy_P.partition_D(sP) if const_expr(sP is not None) else None + cS_route = cute.make_identity_tensor((self.tile_m, self.tile_n)) smem_copy_params = SimpleNamespace( smem_thr_copy_P=smem_thr_copy_P, tPsP=tPsP, ) acc_O.fill(0.0) - cS_route = cute.make_identity_tensor((self.tile_m, self.tile_n)) - tScS_route_mn = layout_utils.reshape_acc_to_mn( - thr_mma_qk.partition_C(cS_route) - ) + tScS_route_mn = layout_utils.reshape_acc_to_mn(thr_mma_qk.partition_C(cS_route)) mask = AttentionMaskCls(seqlen) mask_fn = partial( mask.apply_mask, @@ -1696,6 +1790,14 @@ def mma_one_warpgroup_sol_attn_route_tma( if const_expr(self.sol_attn_neutral_softmax_state): softmax.row_max.fill(-Float32.inf) softmax.row_sum.fill(0.0) + exact_score_scale_fn = None + if const_expr(self.fp8_inputs): + exact_score_scale_fn = partial( + self.sol_attn_scale_exact_scores, + q_block=m_block, + q_scale=mQScale_cur, + k_scale=mKScale_cur, + ) exact_mma_one_n_block = partial( self.mma_one_n_block, mma_qk_fn=mma_qk_fn, @@ -1706,7 +1808,7 @@ def mma_one_warpgroup_sol_attn_route_tma( smem_copy_params=smem_copy_params, softmax=softmax, score_mod_fn=score_mod_fn, - score_scale_fn=None, + score_scale_fn=exact_score_scale_fn, check_inf=not self.sol_attn_assume_nonempty_rows, ) n_block_min, n_block_max = block_info.get_n_block_min_max(seqlen, m_block) @@ -1718,16 +1820,11 @@ def mma_one_warpgroup_sol_attn_route_tma( else: tail_valid_count = Int32(0) elif const_expr(self.sol_attn_assume_full_route_groups): - num_full_route_groups = cute.ceil_div( - route_block_count, self.sol_attn_group_size - ) + num_full_route_groups = cute.ceil_div(route_block_count, self.sol_attn_group_size) tail_valid_count = Int32(0) else: num_full_route_groups = route_block_count // Int32(self.sol_attn_group_size) - tail_valid_count = ( - route_block_count - - num_full_route_groups * Int32(self.sol_attn_group_size) - ) + tail_valid_count = route_block_count - num_full_route_groups * Int32(self.sol_attn_group_size) num_route_groups = num_full_route_groups if tail_valid_count > Int32(0): num_route_groups += Int32(1) @@ -1735,34 +1832,22 @@ def mma_one_warpgroup_sol_attn_route_tma( split_group_begin = Int32(0) split_num_route_groups = num_route_groups else: - groups_per_split = ( - num_route_groups + self.sol_attn_num_splits - 1 - ) // self.sol_attn_num_splits + groups_per_split = (num_route_groups + self.sol_attn_num_splits - 1) // self.sol_attn_num_splits split_group_begin = split_idx * groups_per_split - split_group_end = cutlass.min( - split_group_begin + groups_per_split, num_route_groups - ) - split_num_route_groups = cutlass.max( - split_group_end - split_group_begin, Int32(0) - ) + split_group_end = cutlass.min(split_group_begin + groups_per_split, num_route_groups) + split_num_route_groups = cutlass.max(split_group_end - split_group_begin, Int32(0)) O_should_accumulate = self.sol_attn_neutral_softmax_state - for local_group_iter in cutlass.range( - split_num_route_groups, unroll=1 - ): + for local_group_iter in cutlass.range(split_num_route_groups, unroll=1): group_iter = split_group_begin + local_group_iter group_start = n_block_min + group_iter * Int32(self.sol_attn_group_size) route_valid_count = Int32(self.sol_attn_group_size) if const_expr(not self.sol_attn_assume_full_route_groups): if group_iter == num_full_route_groups and tail_valid_count > Int32(0): route_valid_count = tail_valid_count - route_col_offset = group_start - ( - group_start // Int32(self.tile_n) - ) * Int32(self.tile_n) + route_col_offset = group_start - (group_start // Int32(self.tile_n)) * Int32(self.tile_n) route_n_block = group_start - route_col_offset route_tile = route_n_block // Int32(self.tile_n) - has_next_route_group = ( - local_group_iter + Int32(1) < split_num_route_groups - ) + has_next_route_group = local_group_iter + Int32(1) < split_num_route_groups next_route_tile = Int32(-1) if has_next_route_group: next_group_start = group_start + Int32(self.sol_attn_group_size) @@ -1802,6 +1887,15 @@ def mma_one_warpgroup_sol_attn_route_tma( acc_S = mma_qk_fn(B_idx=kv_consumer_state.index, wg_wait=-1) warpgroup.wait_group(0) pipeline_k.consumer_release(kv_consumer_state) + if const_expr(self.fp8_inputs): + self.sol_attn_scale_route_scores( + acc_S, + tScS_route_mn, + m_block, + route_n_block, + mQScale_cur, + mKCScale_cur, + ) mask0, mask1, mask2, mask3 = self.sol_attn_build_route_mask_from_acc( acc_S, route_sums, @@ -1846,34 +1940,23 @@ def mma_one_warpgroup_sol_attn_route_tma( exact_mask3 = mask3 first_exact_exists = ( - (mask0 != Int32(0)) - or (mask1 != Int32(0)) - or (mask2 != Int32(0)) - or (mask3 != Int32(0)) + (mask0 != Int32(0)) or (mask1 != Int32(0)) or (mask2 != Int32(0)) or (mask3 != Int32(0)) ) if mask0 != Int32(0): first_lowbit = mask0 & (Int32(0) - mask0) - first_exact_n_block += sol_attn_selector.sol_attn_bfind_b32( - first_lowbit - ) + first_exact_n_block += sol_attn_selector.sol_attn_bfind_b32(first_lowbit) exact_mask0 = mask0 & (mask0 - Int32(1)) elif mask1 != Int32(0): first_lowbit = mask1 & (Int32(0) - mask1) - first_exact_n_block += Int32(32) + ( - sol_attn_selector.sol_attn_bfind_b32(first_lowbit) - ) + first_exact_n_block += Int32(32) + (sol_attn_selector.sol_attn_bfind_b32(first_lowbit)) exact_mask1 = mask1 & (mask1 - Int32(1)) elif mask2 != Int32(0): first_lowbit = mask2 & (Int32(0) - mask2) - first_exact_n_block += Int32(64) + ( - sol_attn_selector.sol_attn_bfind_b32(first_lowbit) - ) + first_exact_n_block += Int32(64) + (sol_attn_selector.sol_attn_bfind_b32(first_lowbit)) exact_mask2 = mask2 & (mask2 - Int32(1)) elif mask3 != Int32(0): first_lowbit = mask3 & (Int32(0) - mask3) - first_exact_n_block += Int32(96) + ( - sol_attn_selector.sol_attn_bfind_b32(first_lowbit) - ) + first_exact_n_block += Int32(96) + (sol_attn_selector.sol_attn_bfind_b32(first_lowbit)) exact_mask3 = mask3 & (mask3 - Int32(1)) if first_exact_exists and warp_idx == Int32(0): pipeline_k.producer_acquire(kv_producer_state) @@ -1911,9 +1994,8 @@ def mma_one_warpgroup_sol_attn_route_tma( valid_bits1 = Int32(-1) if valid1 < Int32(32): valid_bits1 = (Int32(1) << valid1) - Int32(1) - route_has_approx = ( - ((mask0 & valid_bits0) != valid_bits0) - or ((mask1 & valid_bits1) != valid_bits1) + route_has_approx = ((mask0 & valid_bits0) != valid_bits0) or ( + (mask1 & valid_bits1) != valid_bits1 ) self.sol_attn_mask_route_approx_columns( acc_S, @@ -1934,17 +2016,13 @@ def mma_one_warpgroup_sol_attn_route_tma( ) if route_has_approx: row_sum_prev = None - if const_expr( - self.sol_attn_fast_route_lens - and not self.sol_attn_full_block_row_sum_prescale - ): + if const_expr(self.sol_attn_fast_route_lens and not self.sol_attn_full_block_row_sum_prescale): row_sum_prev = cute.make_fragment_like(softmax.row_sum, Float32) row_sum_prev.store(softmax.row_sum.load()) + row_scale = cute.make_fragment_like(softmax.row_sum, Float32) if O_should_accumulate: if const_expr(self.sol_attn_full_block_row_sum_prescale): - for r in cutlass.range( - cute.size(softmax.row_sum), unroll_full=True - ): + for r in cutlass.range(cute.size(softmax.row_sum), unroll_full=True): softmax.row_sum[r] *= Float32(1.0 / self.tile_n) row_scale = softmax.online_softmax( acc_S, @@ -1953,9 +2031,7 @@ def mma_one_warpgroup_sol_attn_route_tma( ) softmax.rescale_O(acc_O, row_scale) if const_expr(self.sol_attn_full_block_row_sum_prescale): - for r in cutlass.range( - cute.size(softmax.row_sum), unroll_full=True - ): + for r in cutlass.range(cute.size(softmax.row_sum), unroll_full=True): softmax.row_sum[r] *= Float32(self.tile_n) elif const_expr(self.sol_attn_fast_route_lens): self.sol_attn_apply_route_current_lens_to_row_sum_fast( @@ -1987,9 +2063,7 @@ def mma_one_warpgroup_sol_attn_route_tma( check_inf=not self.sol_attn_assume_nonempty_rows, ) if const_expr(self.sol_attn_full_block_row_sum_prescale): - for r in cutlass.range( - cute.size(softmax.row_sum), unroll_full=True - ): + for r in cutlass.range(cute.size(softmax.row_sum), unroll_full=True): softmax.row_sum[r] *= Float32(self.tile_n) elif const_expr(self.sol_attn_fast_route_lens): self.sol_attn_apply_route_current_lens_to_row_sum_fast( @@ -2014,13 +2088,20 @@ def mma_one_warpgroup_sol_attn_route_tma( seqlen, softmax, ) + if const_expr(self.fp8_inputs and self.sol_attn_num_splits == 1): + self.sol_attn_scale_route_probabilities( + acc_S, + tScS_route_mn, + route_n_block, + seqlen, + ) tOrP_acc = layout_utils.reshape_acc_to_frgA(acc_S) tOrP_cur = ( tOrP if const_expr(self.mma_pv_is_rs) - else cute.make_rmem_tensor_like(tOrP_acc, self.dtype) + else cute.make_rmem_tensor_like(tOrP_acc, self.p_dtype) ) - utils.cvt_f16(tOrP_acc, tOrP_cur) + self.sol_attn_convert_probability(tOrP_acc, tOrP_cur) if const_expr(not self.mma_pv_is_rs): tPrP = smem_copy_params.smem_thr_copy_P.retile(tOrP_cur) cute.copy( @@ -2030,39 +2111,48 @@ def mma_one_warpgroup_sol_attn_route_tma( ) cute.arch.fence_view_async_shared() cute.arch.sync_warp() - if O_should_accumulate: + if const_expr(self.fp8_inputs and self.sol_attn_num_splits == 1): + sm90_utils.gemm_w_idx( + tiled_mma_pv, + tile_acc_O, + tOrP_cur if const_expr(self.mma_pv_is_rs) else tOrP, + tOrVt, + zero_init=True, + B_idx=kv_consumer_state.index, + wg_wait=0, + ) + acc_O.store(acc_O.load() + tile_acc_O.load()) + elif O_should_accumulate: sm90_utils.gemm_w_idx( tiled_mma_pv, acc_O, - tOrP_cur, + tOrP_cur if const_expr(self.mma_pv_is_rs) else tOrP, tOrVt, zero_init=False, B_idx=kv_consumer_state.index, wg_wait=-1, ) + warpgroup.wait_group(0) else: sm90_utils.gemm_w_idx( tiled_mma_pv, acc_O, - tOrP_cur, + tOrP_cur if const_expr(self.mma_pv_is_rs) else tOrP, tOrVt, zero_init=True, B_idx=kv_consumer_state.index, wg_wait=-1, ) - warpgroup.wait_group(0) + warpgroup.wait_group(0) O_should_accumulate = True pipeline_v.consumer_release(kv_consumer_state) kv_consumer_state.advance() last_n_block = Int32(-1) if const_expr( - (not self.sol_attn_assume_full_k_exact_blocks) - or self.sol_attn_exact_mask_seqlen_last_only + (not self.sol_attn_assume_full_k_exact_blocks) or self.sol_attn_exact_mask_seqlen_last_only ): - last_n_block = ( - (seqlen.seqlen_k + Int32(self.tile_n - 1)) // Int32(self.tile_n) - ) - Int32(1) + last_n_block = ((seqlen.seqlen_k + Int32(self.tile_n - 1)) // Int32(self.tile_n)) - Int32(1) if O_should_accumulate: ( kv_producer_state, @@ -2141,6 +2231,13 @@ def mma_one_warpgroup_sol_attn_route_tma( pipeline_q.consumer_release_w_index(0) final_scale = softmax.finalize(sink_val=None) softmax.rescale_O(acc_O, final_scale) + if const_expr(self.fp8_inputs): + self.sol_attn_apply_v_scale( + acc_O, + tiled_mma_pv, + tidx, + mVScale_cur, + ) if const_expr(self.use_tma_O): self.epilogue_one_warpgroup_tma_o( acc_O, @@ -2222,12 +2319,8 @@ def mma_one_n_block( row_scale = softmax.online_softmax(acc_S, is_first=is_first_n_block, check_inf=check_inf) tOrP_acc = layout_utils.reshape_acc_to_frgA(acc_S) - tOrP_cur = ( - tOrP - if const_expr(self.mma_pv_is_rs) - else cute.make_rmem_tensor_like(tOrP_acc, self.dtype) - ) - utils.cvt_f16(tOrP_acc, tOrP_cur) + tOrP_cur = tOrP if const_expr(self.mma_pv_is_rs) else cute.make_rmem_tensor_like(tOrP_acc, self.p_dtype) + self.sol_attn_convert_probability(tOrP_acc, tOrP_cur) if const_expr(not self.mma_pv_is_rs): tPrP = smem_copy_params.smem_thr_copy_P.retile(tOrP_cur) cute.copy(smem_copy_params.smem_thr_copy_P, tPrP, smem_copy_params.tPsP) @@ -2291,9 +2384,7 @@ def apply_score_mod( def warp_scheduler_barrier_sync(self): if const_expr(self.use_scheduler_barrier): cute.arch.barrier( - barrier_id=int(NamedBarrierFwd.WarpSchedulerWG1) - - 1 - + utils.canonical_warp_group_idx(sync=False), + barrier_id=int(NamedBarrierFwd.WarpSchedulerWG1) - 1 + utils.canonical_warp_group_idx(sync=False), number_of_threads=2 * self.num_threads_per_warp_group, ) diff --git a/telefuser/kernel/sol_attn/triton_ref/preprocess.py b/telefuser/kernel/sol_attn/triton_ref/preprocess.py index 06f3a8b..13f60c8 100644 --- a/telefuser/kernel/sol_attn/triton_ref/preprocess.py +++ b/telefuser/kernel/sol_attn/triton_ref/preprocess.py @@ -13,11 +13,7 @@ @triton.autotune( - configs=[ - triton.Config({}, num_warps=warps, num_stages=stages) - for warps in (4, 8) - for stages in (1, 2) - ], + configs=[triton.Config({}, num_warps=warps, num_stages=stages) for warps in (4, 8) for stages in (1, 2)], key=["T"], ) @triton.jit @@ -28,6 +24,9 @@ def _reduce_kv_kernel( v_scale, kc, vc, + kc_fp8, + vc_fp8, + kc_out_scale, T, TP, NPAD, @@ -36,28 +35,54 @@ def _reduce_kv_kernel( D: tl.constexpr, BLOCK: tl.constexpr, FP8: tl.constexpr, + TOKEN_SCALES: tl.constexpr, + V_CHANNEL_SCALE: tl.constexpr, + V_TOKEN_CONTIGUOUS: tl.constexpr, + SM90_FP8_OUTPUTS: tl.constexpr, ): block, batch_head = tl.program_id(0), tl.program_id(1) batch, head = batch_head // H, batch_head % H tokens = block * BLOCK + tl.arange(0, BLOCK) dims = tl.arange(0, D) valid = tokens < T - offsets = ( - ((batch * TP + tokens[:, None]).to(tl.int64) * H + head) * D - + dims[None, :] - ) + offsets = ((batch * TP + tokens[:, None]).to(tl.int64) * H + head) * D + dims[None, :] + v_offsets = offsets + if V_TOKEN_CONTIGUOUS: + v_offsets = ((batch * H + head) * D + dims[None, :]) * TP + tokens[:, None] k_values = tl.load(k + offsets, mask=valid[:, None], other=0.0) - v_values = tl.load(v + offsets, mask=valid[:, None], other=0.0) + v_raw = tl.load(v + v_offsets, mask=valid[:, None], other=0.0) + v_values = v_raw if FP8: scale_offset = (batch * N + block) * H + head - k_values = k_values.to(tl.float32) * tl.load(k_scale + scale_offset) - v_values = v_values.to(tl.float32) * tl.load(v_scale + scale_offset) + if TOKEN_SCALES: + token_scale_offsets = (batch * TP + tokens) * H + head + k_values = ( + k_values.to(tl.float32) + * tl.load( + k_scale + token_scale_offsets, + mask=valid, + other=0.0, + )[:, None] + ) + else: + k_values = k_values.to(tl.float32) * tl.load(k_scale + scale_offset) + if V_CHANNEL_SCALE: + channel_offsets = (batch * H + head) * D + dims + v_values = v_values.to(tl.float32) * tl.load(v_scale + channel_offsets)[None, :] + else: + v_values = v_values.to(tl.float32) * tl.load(v_scale + scale_offset) block_len = tl.minimum(BLOCK, T - block * BLOCK).to(tl.float32) - summary_offsets = ( - ((batch * NPAD + block) * H + head) * D + dims - ) - tl.store(kc + summary_offsets, tl.sum(k_values, axis=0) / block_len) - tl.store(vc + summary_offsets, tl.sum(v_values, axis=0)) + summary_offsets = ((batch * NPAD + block) * H + head) * D + dims + k_summary = tl.sum(k_values, axis=0) / block_len + tl.store(kc + summary_offsets, k_summary) + if SM90_FP8_OUTPUTS: + kc_s = tl.maximum(tl.max(tl.abs(k_summary), axis=0), 1.0e-6) / 448.0 + tl.store(kc_out_scale + (batch * NPAD + block) * H + head, kc_s) + tl.store(kc_fp8 + summary_offsets, k_summary / kc_s) + vc_offsets = ((batch * H + head) * D + dims) * NPAD + block + tl.store(vc_fp8 + vc_offsets, tl.sum(v_raw.to(tl.float32), axis=0) / block_len) + else: + tl.store(vc + summary_offsets, tl.sum(v_values, axis=0)) @triton.jit @@ -81,10 +106,7 @@ def _reduce_kc_stats_kernel( for start in range(0, N, GROUP): block_indices = start + blocks valid = block_indices < N - offsets = ( - ((batch * NPAD + block_indices[:, None]) * H + head) * D - + dims[None, :] - ) + offsets = ((batch * NPAD + block_indices[:, None]) * H + head) * D + dims[None, :] values = tl.load( kc + offsets, mask=valid[:, None], @@ -115,19 +137,28 @@ def _diag_threshold_kernel( BLOCK: tl.constexpr, TAU: tl.constexpr, FP8: tl.constexpr, + TOKEN_SCALES: tl.constexpr, ): q_block, batch_head = tl.program_id(0), tl.program_id(1) batch, head = batch_head // H, batch_head % H tokens = q_block * BLOCK + tl.arange(0, BLOCK) dims = tl.arange(0, D) valid = tokens < T - offsets = ( - ((batch * TP + tokens[:, None]).to(tl.int64) * H + head) * D - + dims[None, :] - ) + offsets = ((batch * TP + tokens[:, None]).to(tl.int64) * H + head) * D + dims[None, :] q_values = tl.load(q + offsets, mask=valid[:, None], other=0.0) if FP8: - q_values = q_values.to(tl.float32) * tl.load(q_scale + (batch * N + q_block) * H + head) + if TOKEN_SCALES: + token_scale_offsets = (batch * TP + tokens) * H + head + q_values = ( + q_values.to(tl.float32) + * tl.load( + q_scale + token_scale_offsets, + mask=valid, + other=0.0, + )[:, None] + ) + else: + q_values = q_values.to(tl.float32) * tl.load(q_scale + (batch * N + q_block) * H + head) q_len = tl.minimum(BLOCK, T - q_block * BLOCK).to(tl.float32) q_centroid = tl.sum(q_values.to(tl.float32), axis=0) / q_len mean_kc = tl.load(kc_mean + batch_head * D + dims) @@ -157,19 +188,28 @@ def _pool_query_kernel( D: tl.constexpr, BLOCK: tl.constexpr, FP8: tl.constexpr, + TOKEN_SCALES: tl.constexpr, ): q_block, batch_head = tl.program_id(0), tl.program_id(1) batch, head = batch_head // H, batch_head % H tokens = q_block * BLOCK + tl.arange(0, BLOCK) dims = tl.arange(0, D) valid = tokens < T - offsets = ( - ((batch * TP + tokens[:, None]).to(tl.int64) * H + head) * D - + dims[None, :] - ) + offsets = ((batch * TP + tokens[:, None]).to(tl.int64) * H + head) * D + dims[None, :] values = tl.load(q + offsets, mask=valid[:, None], other=0.0) if FP8: - values = values.to(tl.float32) * tl.load(q_scale + (batch * N + q_block) * H + head) + if TOKEN_SCALES: + token_scale_offsets = (batch * TP + tokens) * H + head + values = ( + values.to(tl.float32) + * tl.load( + q_scale + token_scale_offsets, + mask=valid, + other=0.0, + )[:, None] + ) + else: + values = values.to(tl.float32) * tl.load(q_scale + (batch * N + q_block) * H + head) q_len = tl.minimum(BLOCK, T - q_block * BLOCK).to(tl.float32) centroid = tl.sum(values.to(tl.float32), axis=0) / q_len tl.store(q_bar + (batch_head * N + q_block) * D + dims, centroid) @@ -198,12 +238,7 @@ def _exact_fused_threshold_kernel( other=0.0, ) mean_kc = tl.load(kc_mean + batch_head * D + dims) - second_moment = tl.load( - kc_second_moment - + batch_head * D * D - + dims[:, None] * D - + dims[None, :] - ) + second_moment = tl.load(kc_second_moment + batch_head * D * D + dims[:, None] * D + dims[None, :]) raw_mean = tl.sum(q_centroid.to(tl.float32) * mean_kc[None, :], axis=1) projected = tl.dot(q_centroid, second_moment, out_dtype=tl.float32) raw_second_moment = tl.sum( @@ -244,6 +279,8 @@ def _reduce_kv( ) vc = torch.zeros_like(kc) fp8_inputs = k.dtype == torch.float8_e4m3fn + v_channel_scale = fp8_inputs and v_scale is not None and v_scale.shape == (batch, heads, head_dim) + v_token_contiguous = v.stride(1) == 1 dummy_scale = torch.ones((1,), device=k.device, dtype=torch.float32) _reduce_kv_kernel[(blocks, batch * heads)]( k, @@ -252,6 +289,9 @@ def _reduce_kv( v_scale if fp8_inputs else dummy_scale, kc, vc, + kc, + vc, + dummy_scale, tokens, padded_tokens, padded_blocks, @@ -260,10 +300,93 @@ def _reduce_kv( head_dim, BLOCK_SIZE, FP8=fp8_inputs, + TOKEN_SCALES=False, + V_CHANNEL_SCALE=v_channel_scale, + V_TOKEN_CONTIGUOUS=v_token_contiguous, + SM90_FP8_OUTPUTS=False, ) return kc, vc +def prepare_sm90_fp8( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + *, + tau: float, + scale: float, + thresh_type: str = "diag", + tokens: int | None = None, + q_scale: torch.Tensor, + k_scale: torch.Tensor, + v_scale: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Build BF16 threshold stats and TMA-ready FP8 summaries in one reduction.""" + + batch, padded_tokens, heads, head_dim = k.shape + tokens = padded_tokens if tokens is None else int(tokens) + blocks = triton.cdiv(tokens, BLOCK_SIZE) + padded_blocks = triton.cdiv(blocks, SUMMARY_PAD) * SUMMARY_PAD + kc_stats = torch.zeros( + (batch, padded_blocks, heads, head_dim), + device=k.device, + dtype=torch.bfloat16, + ) + kc_fp8 = torch.zeros_like(kc_stats, dtype=torch.float8_e4m3fn) + vc_storage = torch.zeros( + (batch, heads, head_dim, padded_blocks), + device=k.device, + dtype=torch.float8_e4m3fn, + ) + kc_out_scale = torch.ones( + (batch, padded_blocks, heads), + device=k.device, + dtype=torch.float32, + ) + _reduce_kv_kernel[(blocks, batch * heads)]( + k, + v, + k_scale, + v_scale, + kc_stats, + kc_stats, + kc_fp8, + vc_storage, + kc_out_scale, + tokens, + padded_tokens, + padded_blocks, + heads, + blocks, + head_dim, + BLOCK_SIZE, + FP8=True, + TOKEN_SCALES=True, + V_CHANNEL_SCALE=True, + V_TOKEN_CONTIGUOUS=True, + SM90_FP8_OUTPUTS=True, + ) + if thresh_type == "exact": + threshold = _compute_exact_threshold( + q, + kc_stats, + tau=tau, + scale=scale, + tokens=tokens, + q_scale=q_scale, + ) + else: + threshold = _compute_diag_threshold( + q, + kc_stats, + tau=tau, + scale=scale, + tokens=tokens, + q_scale=q_scale, + ) + return kc_fp8, vc_storage.permute(0, 3, 1, 2), threshold, kc_out_scale + + def _compute_diag_threshold( q: torch.Tensor, kc: torch.Tensor, @@ -315,6 +438,7 @@ def _compute_diag_threshold( BLOCK_SIZE, tau, FP8=q.dtype == torch.float8_e4m3fn, + TOKEN_SCALES=q_scale is not None and q_scale.shape[1] == padded_tokens, num_warps=4, num_stages=2, ) @@ -362,13 +486,12 @@ def _compute_exact_threshold( head_dim, BLOCK_SIZE, FP8=q.dtype == torch.float8_e4m3fn, + TOKEN_SCALES=q_scale is not None and q_scale.shape[1] == padded_tokens, num_warps=4, num_stages=1, ) block_m = 64 - _exact_fused_threshold_kernel[ - (triton.cdiv(blocks, block_m), batch_heads) - ]( + _exact_fused_threshold_kernel[(triton.cdiv(blocks, block_m), batch_heads)]( q_bar, kc_mean, kc_second_moment, @@ -420,4 +543,4 @@ def prepare( return kc, vc, threshold -__all__ = ["prepare"] +__all__ = ["prepare", "prepare_sm90_fp8"] diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index dcbc535..e58218e 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -37,7 +37,7 @@ from telefuser.offload.async_offload import AsyncOffloadManager from telefuser.ops.attention import MaskMap, SparseAttentionState from telefuser.ops.attention import attention as attn_func -from telefuser.ops.fp8_attention import quantize_fp8_per_block +from telefuser.ops.fp8_attention import quantize_fp8_per_block, quantize_fp8_qkv from telefuser.ops.normalization import LayerNorm, RMSNorm, fused_scale_shift, modulate from telefuser.ops.rotary import apply_rotary_emb from telefuser.utils.logging import logger @@ -148,16 +148,10 @@ def _prepare_sol_qkv( v: torch.Tensor, sparse_state: SparseAttentionState | None, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None]: - # H100 currently has no native FP8 CuTe Sol-Attn mainloop. Keeping - # QKV in BF16 lets the production SM90 kernel run without paying the - # quantize/dequantize overhead that otherwise makes this path slower. - native_fp8_sol = not (q.is_cuda and torch.cuda.get_device_capability(q.device) == (9, 0)) - if ( - self._is_sol_active(sparse_state) - and sparse_state is not None - and sparse_state.config.sol_fp8 - and native_fp8_sol - ): + if self._is_sol_active(sparse_state) and sparse_state is not None and sparse_state.config.sol_fp8: + if q.is_cuda and torch.cuda.get_device_capability(q.device) == (9, 0): + q, k, v, q_scale, k_scale, v_scale = quantize_fp8_qkv(q, k, v) + return q, k, v, (q_scale, k_scale, v_scale) q, q_scale = quantize_fp8_per_block(q) k, k_scale = quantize_fp8_per_block(k) v, v_scale = quantize_fp8_per_block(v) diff --git a/telefuser/ops/attention/attention_impl.py b/telefuser/ops/attention/attention_impl.py index d295b8e..5a2a79e 100755 --- a/telefuser/ops/attention/attention_impl.py +++ b/telefuser/ops/attention/attention_impl.py @@ -166,8 +166,11 @@ def _resolve_sol_kv_splits(q: Tensor, kv_splits: int | str) -> int: """Match the official Sol-Engine automatic split policy.""" if kv_splits != "auto": return int(kv_splits) - if torch.cuda.get_device_capability(q.device) == (9, 0) and q.shape[1] >= 65536: - return 4 + if torch.cuda.get_device_capability(q.device) == (9, 0): + if q.dtype == torch.float8_e4m3fn and q.shape[1] >= 16384: + return 4 + if q.shape[1] >= 65536: + return 4 return 1 @@ -427,7 +430,7 @@ def attention( output = sol_attn( q.contiguous(), k.contiguous(), - v.contiguous(), + v if q.dtype == torch.float8_e4m3fn else v.contiguous(), scale=scale, tau=sparse_config.sol_tau, thresh_type=sparse_config.sol_threshold_type, @@ -442,11 +445,26 @@ def attention( _warned_attn_fallback.add(msg) logger.warning("%s: %s", msg, error) if q.dtype == torch.float8_e4m3fn: - from telefuser.ops.fp8_attention import dequantize_fp8_per_block - - q = dequantize_fp8_per_block(q, q_scale, torch.bfloat16) - k = dequantize_fp8_per_block(k, k_scale, torch.bfloat16) - v = dequantize_fp8_per_block(v, v_scale, torch.bfloat16) + from telefuser.ops.fp8_attention import ( + dequantize_fp8_per_block, + dequantize_fp8_per_channel, + dequantize_fp8_per_token, + ) + + if ( + q_scale.shape[0] == q.shape[0] + and q_scale.shape[1] >= q.shape[1] + and q_scale.shape[2] == q.shape[2] + ): + q = dequantize_fp8_per_token(q, q_scale, torch.bfloat16) + k = dequantize_fp8_per_token(k, k_scale, torch.bfloat16) + else: + q = dequantize_fp8_per_block(q, q_scale, torch.bfloat16) + k = dequantize_fp8_per_block(k, k_scale, torch.bfloat16) + if v_scale.shape == (v.shape[0], v.shape[2], v.shape[3]): + v = dequantize_fp8_per_channel(v, v_scale, torch.bfloat16) + else: + v = dequantize_fp8_per_block(v, v_scale, torch.bfloat16) # Fallback to SDPA if output is None: diff --git a/telefuser/ops/fp8_attention.py b/telefuser/ops/fp8_attention.py index 6777dfc..656967e 100644 --- a/telefuser/ops/fp8_attention.py +++ b/telefuser/ops/fp8_attention.py @@ -4,10 +4,131 @@ import torch import torch.nn.functional as F +import triton +import triton.language as tl FP8_ATTENTION_BLOCK_SIZE = 64 +@triton.jit +def _quantize_qkv_fp8_stage1( + q, + k, + v, + q_out, + k_out, + q_scale, + k_scale, + v_scale, + tokens: tl.constexpr, + heads: tl.constexpr, + head_dim: tl.constexpr, + block: tl.constexpr, +): + block_idx = tl.program_id(0) + batch_head = tl.program_id(1) + batch = batch_head // heads + head = batch_head % heads + token_offsets = block_idx * block + tl.arange(0, block) + dim_offsets = tl.arange(0, head_dim) + valid = token_offsets < tokens + offsets = ((batch * tokens + token_offsets[:, None]) * heads + head) * head_dim + dim_offsets[None, :] + q_values = tl.load(q + offsets, mask=valid[:, None], other=0.0).to(tl.float32) + k_values = tl.load(k + offsets, mask=valid[:, None], other=0.0).to(tl.float32) + v_values = tl.load(v + offsets, mask=valid[:, None], other=0.0).to(tl.float32) + + q_s = tl.maximum(tl.max(tl.max(tl.abs(q_values), axis=1), axis=0), 1.0e-6) / 448.0 + k_s = tl.maximum(tl.max(tl.max(tl.abs(k_values), axis=1), axis=0), 1.0e-6) / 448.0 + scale_offsets = (batch * tl.cdiv(tokens, block) * block + token_offsets) * heads + head + tl.store(q_scale + scale_offsets, q_s, mask=valid) + tl.store(k_scale + scale_offsets, k_s, mask=valid) + tl.store(q_out + offsets, q_values / q_s, mask=valid[:, None]) + tl.store(k_out + offsets, k_values / k_s, mask=valid[:, None]) + + v_s = tl.max(tl.abs(v_values), axis=0) / 448.0 + v_scale_offsets = (batch * heads + head) * head_dim + dim_offsets + tl.atomic_max(v_scale + v_scale_offsets, v_s) + + +@triton.jit +def _quantize_qkv_fp8_stage2_v( + v, + v_out, + v_scale, + tokens: tl.constexpr, + heads: tl.constexpr, + head_dim: tl.constexpr, + block: tl.constexpr, +): + block_idx = tl.program_id(0) + batch_head = tl.program_id(1) + batch = batch_head // heads + head = batch_head % heads + token_offsets = block_idx * block + tl.arange(0, block) + dim_offsets = tl.arange(0, head_dim) + valid = token_offsets < tokens + input_offsets = ((batch * tokens + token_offsets[:, None]) * heads + head) * head_dim + dim_offsets[None, :] + output_offsets = ((batch * heads + head) * head_dim + dim_offsets[None, :]) * tokens + token_offsets[:, None] + scale_offsets = (batch * heads + head) * head_dim + dim_offsets + scale = tl.maximum(tl.load(v_scale + scale_offsets), 1.0e-6 / 448.0) + values = tl.load(v + input_offsets, mask=valid[:, None], other=0.0).to(tl.float32) + tl.store(v_out + output_offsets, values / scale[None, :], mask=valid[:, None]) + + +def quantize_fp8_qkv( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Fused SM90 block-scaled Q/K and layout-aware V-channel E4M3 quantization.""" + + if q.shape != k.shape or q.shape != v.shape or q.ndim != 4: + raise ValueError("q, k, and v must share shape [B, T, H, D]") + if not (q.is_cuda and q.is_contiguous() and k.is_contiguous() and v.is_contiguous()): + raise ValueError("fused FP8 QKV quantization requires contiguous CUDA tensors") + batch, tokens, heads, head_dim = q.shape + if head_dim != 128: + raise ValueError("fused FP8 QKV quantization requires head dimension 128") + blocks = triton.cdiv(tokens, FP8_ATTENTION_BLOCK_SIZE) + q_out = torch.empty(q.shape, device=q.device, dtype=torch.float8_e4m3fn) + k_out = torch.empty_like(q_out) + v_storage = torch.empty((batch, heads, head_dim, tokens), device=q.device, dtype=torch.float8_e4m3fn) + padded_tokens = blocks * FP8_ATTENTION_BLOCK_SIZE + q_scale = torch.ones((batch, padded_tokens, heads), device=q.device, dtype=torch.float32) + k_scale = torch.ones_like(q_scale) + v_scale = torch.zeros((batch, heads, head_dim), device=q.device, dtype=torch.float32) + grid = (blocks, batch * heads) + _quantize_qkv_fp8_stage1[grid]( + q, + k, + v, + q_out, + k_out, + q_scale, + k_scale, + v_scale, + tokens, + heads, + head_dim, + FP8_ATTENTION_BLOCK_SIZE, + num_warps=8, + num_stages=1, + ) + _quantize_qkv_fp8_stage2_v[grid]( + v, + v_storage, + v_scale, + tokens, + heads, + head_dim, + FP8_ATTENTION_BLOCK_SIZE, + num_warps=8, + num_stages=1, + ) + v_out = v_storage.permute(0, 3, 1, 2) + return q_out, k_out, v_out, q_scale, k_scale, v_scale + + def quantize_fp8_per_block( x: torch.Tensor, block_size: int = FP8_ATTENTION_BLOCK_SIZE, @@ -38,4 +159,59 @@ def dequantize_fp8_per_block( return x.to(dtype) * token_scale.to(dtype).unsqueeze(-1) -__all__ = ["FP8_ATTENTION_BLOCK_SIZE", "dequantize_fp8_per_block", "quantize_fp8_per_block"] +def dequantize_fp8_per_token( + x: torch.Tensor, + scale: torch.Tensor, + dtype: torch.dtype, +) -> torch.Tensor: + """Restore token-scaled FP8 [B, T, H, D] activations.""" + if x.dtype != torch.float8_e4m3fn: + raise TypeError("expected torch.float8_e4m3fn activations") + if scale.shape[0] != x.shape[0] or scale.shape[1] < x.shape[1] or scale.shape[2] != x.shape[2]: + raise ValueError("scale must have shape [B, padded_T, H] with padded_T >= T") + return x.to(dtype) * scale[:, : x.shape[1]].to(dtype).unsqueeze(-1) + + +def quantize_fp8_per_channel( + x: torch.Tensor, + *, + token_contiguous: bool = False, +) -> tuple[torch.Tensor, torch.Tensor]: + """Quantize BTHD with one E4M3 scale per head/channel. + + ``token_contiguous`` stores the same BTHD view over B,H,D,T-contiguous + backing memory, matching the SM90 K-major PV WGMMA operand. + """ + if x.ndim != 4 or not x.is_floating_point(): + raise ValueError("FP8 attention quantization expects a floating-point [B, T, H, D] tensor") + scale = x.detach().abs().amax(dim=1).float().clamp_min(1e-6) / 448.0 + quantized = (x / scale.to(x.dtype).unsqueeze(1)).to(torch.float8_e4m3fn) + if token_contiguous: + quantized = quantized.permute(0, 2, 3, 1).contiguous().permute(0, 3, 1, 2) + else: + quantized = quantized.contiguous() + return quantized, scale.contiguous() + + +def dequantize_fp8_per_channel( + x: torch.Tensor, + scale: torch.Tensor, + dtype: torch.dtype, +) -> torch.Tensor: + """Restore channel-scaled FP8 [B, T, H, D] activations.""" + if x.dtype != torch.float8_e4m3fn: + raise TypeError("expected torch.float8_e4m3fn activations") + if scale.shape != (x.shape[0], x.shape[2], x.shape[3]): + raise ValueError("scale must have shape [B, H, D]") + return x.to(dtype) * scale.to(dtype).unsqueeze(1) + + +__all__ = [ + "FP8_ATTENTION_BLOCK_SIZE", + "dequantize_fp8_per_block", + "dequantize_fp8_per_channel", + "dequantize_fp8_per_token", + "quantize_fp8_qkv", + "quantize_fp8_per_block", + "quantize_fp8_per_channel", +] diff --git a/tests/unit/models/test_wan_video_sol_attention.py b/tests/unit/models/test_wan_video_sol_attention.py index 6e5a700..c62b342 100644 --- a/tests/unit/models/test_wan_video_sol_attention.py +++ b/tests/unit/models/test_wan_video_sol_attention.py @@ -6,7 +6,13 @@ from telefuser.core.config import AttentionConfig, AttnImplType, SparseAttentionConfig from telefuser.models.wan_video_dit import SelfAttention, WanModel, precompute_freqs_cis_3d from telefuser.ops.attention import SparseAttentionState, attention_impl -from telefuser.ops.fp8_attention import dequantize_fp8_per_block, quantize_fp8_per_block +from telefuser.ops.fp8_attention import ( + dequantize_fp8_per_block, + dequantize_fp8_per_channel, + dequantize_fp8_per_token, + quantize_fp8_per_block, + quantize_fp8_qkv, +) def test_wan_model_enables_sol_attention_state() -> None: @@ -183,3 +189,128 @@ def tracked_sol_attn(*args, **kwargs): assert output.shape == x.shape assert torch.isfinite(output).all() assert kernel_calls == 1 + + +@pytest.mark.gpu +def test_wan_self_attention_executes_native_fp8_sol_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): + pytest.skip("Wan FP8 Sol-Attn execution test requires H100") + + assert attention_impl.SOL_ATTN_AVAILABLE + assert attention_impl.sol_attn is not None + captured = {} + sol_attn = attention_impl.sol_attn + + def tracked_sol_attn(q, k, v, **kwargs): + captured.update({"q": q, "k": k, "v": v, **kwargs}) + return sol_attn(q, k, v, **kwargs) + + monkeypatch.setattr(attention_impl, "sol_attn", tracked_sol_attn) + + module = SelfAttention(dim=128, num_heads=1).eval().cuda().to(torch.bfloat16) + x = torch.randn(1, 256, 128, device="cuda", dtype=torch.bfloat16) + freqs = precompute_freqs_cis_3d(128) + freqs_cos = torch.cat([freq.real for freq in freqs], dim=-1)[:256].cuda() + freqs_sin = torch.cat([freq.imag for freq in freqs], dim=-1)[:256].cuda() + sparse_config = SparseAttentionConfig( + sparse_impl="sol", + dense_timesteps=0, + sol_tau=-1000.0, + sol_fp8=True, + ) + module.attention_config = AttentionConfig(attn_impl=AttnImplType.SOL_ATTN, sparse_config=sparse_config) + state = SparseAttentionState(sparse_config, mask_map=None) + + output = module(x, freqs_cos, freqs_sin, sparse_state=state) + + assert output.shape == x.shape + assert torch.isfinite(output).all() + assert captured["q"].dtype is torch.float8_e4m3fn + assert captured["k"].dtype is torch.float8_e4m3fn + assert captured["v"].dtype is torch.float8_e4m3fn + assert captured["v"].stride(1) == 1 + assert captured["q_scale"].shape == (1, 256, 1) + assert captured["k_scale"].shape == (1, 256, 1) + assert captured["v_scale"].shape == (1, 1, 128) + + +@pytest.mark.gpu +def test_fused_fp8_qkv_quantization_on_h100() -> None: + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): + pytest.skip("fused FP8 QKV quantization test requires H100") + + q = torch.randn(1, 130, 2, 128, device="cuda", dtype=torch.bfloat16) + k = torch.randn_like(q) + v = torch.randn_like(q) + q_fp8, k_fp8, v_fp8, q_scale, k_scale, v_scale = quantize_fp8_qkv(q, k, v) + + assert q_fp8.shape == q.shape + assert k_fp8.shape == k.shape + assert v_fp8.shape == v.shape + assert q_scale.shape == (1, 192, 2) + assert k_scale.shape == (1, 192, 2) + assert v_scale.shape == (1, 2, 128) + assert v_fp8.stride(1) == 1 + torch.testing.assert_close( + dequantize_fp8_per_token(q_fp8, q_scale, torch.bfloat16), + q, + rtol=0.15, + atol=0.05, + ) + torch.testing.assert_close( + dequantize_fp8_per_channel(v_fp8, v_scale, torch.bfloat16), + v, + rtol=0.15, + atol=0.05, + ) + + +@pytest.mark.gpu +def test_native_fp8_sol_handles_partial_tail_on_h100() -> None: + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): + pytest.skip("partial-tail FP8 Sol-Attn test requires H100") + + q = torch.randn(1, 130, 1, 128, device="cuda", dtype=torch.bfloat16) + k = torch.randn_like(q) + v = torch.randn_like(q) + q_fp8, k_fp8, v_fp8, q_scale, k_scale, v_scale = quantize_fp8_qkv(q, k, v) + output = attention_impl.sol_attn( + q_fp8, + k_fp8, + v_fp8, + tau=-1000.0, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, + ) + reference = attention_impl.sol_attn( + dequantize_fp8_per_token(q_fp8, q_scale, torch.bfloat16).contiguous(), + dequantize_fp8_per_token(k_fp8, k_scale, torch.bfloat16).contiguous(), + dequantize_fp8_per_channel(v_fp8, v_scale, torch.bfloat16).contiguous(), + tau=-1000.0, + ) + + cosine = torch.nn.functional.cosine_similarity(output.float().flatten(), reference.float().flatten(), dim=0) + assert cosine > 0.99 + + +@pytest.mark.gpu +def test_native_fp8_sol_preserves_constant_values_at_long_sequence_on_h100() -> None: + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): + pytest.skip("long-sequence FP8 Sol-Attn test requires H100") + + q = torch.randn(1, 2048, 1, 128, device="cuda", dtype=torch.bfloat16) + k = torch.randn_like(q) + v = torch.ones_like(q) + q, k, v, q_scale, k_scale, v_scale = quantize_fp8_qkv(q, k, v) + output = attention_impl.sol_attn( + q, + k, + v, + tau=-1000.0, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, + ) + + torch.testing.assert_close(output.float(), torch.ones_like(output, dtype=torch.float32), rtol=0.02, atol=0.02) From 2a6e823f21292a11e3fb80cc4a56b38b7c78ae21 Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Fri, 14 Aug 2026 07:44:28 +0000 Subject: [PATCH 10/15] fix: preserve Wan FP8 Sol generation quality --- docs/en/attention.md | 6 +++ docs/zh/attention.md | 6 +++ examples/wan_video/README.md | 23 ++++++++- ...wan21_1_3b_text_to_video_optimized_h100.py | 35 +++++++++++++- telefuser/core/config.py | 10 ++++ telefuser/kernel/sol_attn/sm90/mainloop.py | 2 +- telefuser/models/wan_video_dit.py | 6 +++ .../models/test_wan_video_sol_attention.py | 47 +++++++++++++++++++ .../wan_video/test_optimized_example.py | 21 ++++++++- 9 files changed, 150 insertions(+), 6 deletions(-) diff --git a/docs/en/attention.md b/docs/en/attention.md index 20d1ab4..56a21b2 100644 --- a/docs/en/attention.md +++ b/docs/en/attention.md @@ -72,6 +72,8 @@ class SparseAttentionConfig: sol_threshold_type: str = "diag" # "diag" or "exact" sol_kv_splits: int | str = "auto" # "auto", 1, 2, or 4 sol_fp8: bool = False # Native FP8 Q/K/V Sol-Attn on SM90 + sol_fp8_layer_start: int = 0 # First layer using FP8 Sol-Attn + sol_fp8_layer_end: int | None = None # Exclusive end; None means all remaining layers ``` ## Calling Flow @@ -205,6 +207,10 @@ dense warmup layers or timesteps, and kernel runtime failures fall back to the existing dense attention path. Ring/USP remains dense because its online merge requires log-sum-exp output. +`sol_fp8_layer_start` and `sol_fp8_layer_end` restrict E4M3 Q/K/V to a half-open +transformer-layer range. Sparse layers outside that range continue to use BF16 +Sol-Attn. This controls accumulated FP8 routing error in diffusion models. + ### QwenImagePipeline / ZImagePipeline Supports only dense attention (image generation doesn't have temporal dimension): diff --git a/docs/zh/attention.md b/docs/zh/attention.md index bc8cdec..0d5a13b 100644 --- a/docs/zh/attention.md +++ b/docs/zh/attention.md @@ -72,6 +72,8 @@ class SparseAttentionConfig: sol_threshold_type: str = "diag" # "diag" 或 "exact" sol_kv_splits: int | str = "auto" # "auto"、1、2 或 4 sol_fp8: bool = False # SM90 原生 FP8 Q/K/V Sol-Attn + sol_fp8_layer_start: int = 0 # 启用 FP8 Sol-Attn 的首层 + sol_fp8_layer_end: int | None = None # 结束层（不包含）；None 表示其余所有层 ``` ## 调用流程 @@ -203,6 +205,10 @@ self-attention。各架构内核支持 BF16，SM90 还支持使用 FP32 累加 其他调用、dense 预热层/时间步以及内核运行失败都会回退到现有密集路径。 Ring/USP 需要 LSE 做在线合并，因此仍使用支持 LSE 的密集后端。 +`sol_fp8_layer_start` 和 `sol_fp8_layer_end` 用半开区间限制使用 E4M3 Q/K/V +的 transformer 层，区间外的稀疏层继续使用 BF16 Sol-Attn，以控制扩散模型中 +逐层累积的 FP8 路由误差。 + ### QwenImagePipeline / ZImagePipeline 仅支持密集注意力（图像生成没有时序维度）： diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 360a751..f96b201 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -184,11 +184,23 @@ Attention choices: - `sol`: Sol-Attn with dense warm-up and fallback calls - `sol-fp8`: native E4M3 Q/K/V Sol-Attn on H100 +For `sol-fp8`, `--sol-fp8-layer-start` and `--sol-fp8-layer-end` select the +half-open transformer-layer range that uses FP8 Q/K/V. Other sparse layers use +BF16 Sol-Attn. Restricting FP8 Q/K/V to middle layers avoids accumulating small +routing changes across the full denoiser. + Quantization choices: - `none`: BF16 DiT - `tf-kernel-fp8`: TeleFuser dynamic W8A8 FP8 GEMM +`--fp8-linear-scope all` quantizes every transformer-block Linear layer. +`--fp8-linear-scope ffn` keeps self/cross-attention projections in BF16 and +quantizes the 60 FFN Linear layers. The latter is recommended when Linear FP8 +and FP8 Sol-Attn are enabled together. The default `auto` selects `all` for +dense attention and `ffn` for `sol-fp8`; the default FP8 Sol layer range is +10-19 for this 30-layer Wan2.1 model. + Only DiT transformer-block Linear layers are quantized; the VAE and text encoder remain BF16. Select the two optimization axes independently: @@ -213,6 +225,9 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ --attention sol-fp8 \ --quantization tf-kernel-fp8 \ + --fp8-linear-scope ffn \ + --sol-fp8-layer-start 10 \ + --sol-fp8-layer-end 20 \ --dense-timesteps 10 \ --dense-layers 1 \ --tau 1.0 \ @@ -226,7 +241,9 @@ SM90 WGMMA mainloop. Q/K use one scale per 64-token block, V uses per-channel scales and a K-major layout, and FP32 accumulators are used throughout. `auto` selects four KV splits for long FP8 sequences to bound Hopper FP8 accumulation error. Partial tiles are physically padded while the original sequence length -remains masked in the kernel. The Sol tuning options apply to both modes. +remains masked in the kernel. FP8 split execution restores the represented N64 +route length before PV, matching the BF16 summed-centroid contract. The Sol +tuning options apply to both modes. The final log reports generation time, frames per second, and peak allocated and reserved CUDA memory. @@ -244,7 +261,8 @@ same generation interval. | BF16 | Dense | 0.8491 | 16.147 | | BF16 | Sol-Attn | 1.1090 | 17.023 | | tf-kernel FP8 | Dense | 0.8771 | 14.855 | -| tf-kernel FP8 | Sol-FP8 (native SM90 QKV) | 1.1879 | 15.730 | +| tf-kernel FP8 | Sol-FP8 (all Linear/QKV layers, aggressive) | 1.1879 | 15.730 | +| tf-kernel FP8 FFN | Sol-FP8 (QKV layers 10-19, precision) | 1.0347 | 16.256 | The benchmark prompt is: @@ -259,6 +277,7 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ --attention sol \ --quantization tf-kernel-fp8 \ + --fp8-linear-scope all \ --width 832 --height 480 \ --num-frames 81 --num-inference-steps 50 \ --sample-solver unipc --cfg-scale 5.0 --sigma-shift 5.0 --seed 42 diff --git a/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py index 68b653b..d7fb036 100644 --- a/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py +++ b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py @@ -53,8 +53,10 @@ def configure_attention_backends() -> None: torch.backends.cuda.enable_mem_efficient_sdp(True) -def make_quant_config(quantization: str) -> QuantConfig: +def make_quant_config(quantization: str, *, fp8_linear_scope: str = "all") -> QuantConfig: """Build the online quantization config used for the Wan DiT.""" + if fp8_linear_scope not in ("all", "ffn"): + raise ValueError("fp8_linear_scope must be 'all' or 'ffn'") if quantization == "none": return QuantConfig() if quantization == "tf-kernel-fp8": @@ -62,6 +64,7 @@ def make_quant_config(quantization: str) -> QuantConfig: enabled=True, quant_type=QuantType.FP8, kernel_backend=QuantKernelBackend.TF_KERNEL, + quantize_modules=(".ffn.",) if fp8_linear_scope == "ffn" else None, ) if quantization == "torchao-fp8": return QuantConfig( @@ -78,6 +81,15 @@ def make_quant_config(quantization: str) -> QuantConfig: raise ValueError("quantization must be 'none', 'tf-kernel-fp8', 'torchao-fp8', or 'bnb-nf4'") +def resolve_fp8_linear_scope(attention: str, fp8_linear_scope: str) -> str: + """Resolve the quality-safe scope for the selected attention mode.""" + if fp8_linear_scope == "auto": + return "ffn" if attention == "sol-fp8" else "all" + if fp8_linear_scope not in ("all", "ffn"): + raise ValueError("fp8_linear_scope must be 'auto', 'all', or 'ffn'") + return fp8_linear_scope + + def make_attention_config( attention: str, *, @@ -86,6 +98,8 @@ def make_attention_config( tau: float = 1.0, threshold_type: str = "diag", kv_splits: int | str = "auto", + fp8_layer_start: int = 10, + fp8_layer_end: int | None = 20, ) -> AttentionConfig: """Build the selected dense or Sol-Attn configuration.""" if attention == "dense": @@ -99,6 +113,8 @@ def make_attention_config( threshold_type=threshold_type, kv_splits=kv_splits, sol_fp8=attention == "sol-fp8", + sol_fp8_layer_start=fp8_layer_start, + sol_fp8_layer_end=fp8_layer_end, ) @@ -107,15 +123,19 @@ def get_pipeline( model_root: str = PPL_CONFIG["model_root"], attention: str = "dense", quantization: str = "none", + fp8_linear_scope: str = "auto", dense_timesteps: int = 10, dense_layers: int = 1, tau: float = 1.0, threshold_type: str = "diag", kv_splits: int | str = "auto", + sol_fp8_layer_start: int = 10, + sol_fp8_layer_end: int | None = 20, sample_solver: str = "euler", ) -> Wan21VideoPipeline: """Load Wan2.1 with independently selectable attention and quantization.""" - quant_config = make_quant_config(quantization) + fp8_linear_scope = resolve_fp8_linear_scope(attention, fp8_linear_scope) + quant_config = make_quant_config(quantization, fp8_linear_scope=fp8_linear_scope) module_manager = ModuleManager(torch_dtype=torch.bfloat16, device="cpu") module_manager.load_model(f"{model_root}/Wan2.1_VAE.pth", device="cpu", torch_dtype=torch.bfloat16) module_manager.load_model( @@ -135,6 +155,8 @@ def get_pipeline( tau=tau, threshold_type=threshold_type, kv_splits=kv_splits, + fp8_layer_start=sol_fp8_layer_start, + fp8_layer_end=sol_fp8_layer_end, ) config.dit_config.quant_config = quant_config config.dit_config.offload_config.offload_type = WeightOffloadType.NO_CPU_OFFLOAD @@ -196,11 +218,14 @@ def run( default="none", type=click.Choice(["none", "tf-kernel-fp8", "torchao-fp8", "bnb-nf4"]), ) +@click.option("--fp8-linear-scope", default="auto", type=click.Choice(["auto", "all", "ffn"])) @click.option("--dense-timesteps", default=10, type=int) @click.option("--dense-layers", default=1, type=int) @click.option("--tau", default=1.0, type=float) @click.option("--threshold-type", default="diag", type=click.Choice(["diag", "exact"])) @click.option("--kv-splits", default="auto", type=click.Choice(["auto", "1", "2", "4"])) +@click.option("--sol-fp8-layer-start", default=10, type=int) +@click.option("--sol-fp8-layer-end", default=20, type=int) @click.option("--output", default=get_example_name(__file__, "mp4")) def main( prompt: str, @@ -216,11 +241,14 @@ def main( model_root: str, attention: str, quantization: str, + fp8_linear_scope: str, dense_timesteps: int, dense_layers: int, tau: float, threshold_type: str, kv_splits: str, + sol_fp8_layer_start: int, + sol_fp8_layer_end: int | None, output: str, ) -> None: """Run Wan2.1 with optional attention and quantization optimizations.""" @@ -229,11 +257,14 @@ def main( model_root=model_root, attention=attention, quantization=quantization, + fp8_linear_scope=fp8_linear_scope, dense_timesteps=dense_timesteps, dense_layers=dense_layers, tau=tau, threshold_type=threshold_type, kv_splits=kv_splits if kv_splits == "auto" else int(kv_splits), + sol_fp8_layer_start=sol_fp8_layer_start, + sol_fp8_layer_end=sol_fp8_layer_end, sample_solver=sample_solver, ) torch.cuda.reset_peak_memory_stats() diff --git a/telefuser/core/config.py b/telefuser/core/config.py index e6d4316..0c2946c 100644 --- a/telefuser/core/config.py +++ b/telefuser/core/config.py @@ -152,6 +152,8 @@ class SparseAttentionConfig: sol_threshold_type: str = "diag" # Sol-Attn threshold estimator: "diag" or "exact" sol_kv_splits: int | str = "auto" # Auto selects split 4 for long SM90 sequences sol_fp8: bool = False # Quantize post-RoPE Q/K/V activations for FP8 Sol-Attn + sol_fp8_layer_start: int = 0 # First transformer layer using FP8 Sol-Attn + sol_fp8_layer_end: int | None = None # Exclusive end; None enables all remaining layers def __post_init__(self) -> None: if self.sparse_impl != "sol": @@ -160,6 +162,10 @@ def __post_init__(self) -> None: raise ValueError("Sol-Attn threshold type must be 'diag' or 'exact'") if self.sol_kv_splits not in ("auto", 1, 2, 4): raise ValueError("Sol-Attn KV splits must be 'auto', 1, 2, or 4") + if self.sol_fp8_layer_start < 0: + raise ValueError("Sol-Attn FP8 layer start must be non-negative") + if self.sol_fp8_layer_end is not None and self.sol_fp8_layer_end <= self.sol_fp8_layer_start: + raise ValueError("Sol-Attn FP8 layer end must be greater than its start") def should_use_dense(self, numeral_timestep: int, layer_idx: int) -> bool: """Check if dense attention should be used for current step/layer. @@ -231,6 +237,8 @@ def sol_attention( threshold_type: str = "diag", kv_splits: int | str = "auto", sol_fp8: bool = False, + sol_fp8_layer_start: int = 0, + sol_fp8_layer_end: int | None = None, **kwargs: any, ) -> AttentionConfig: """Create a Sol-Attn config for dynamic sparse video self-attention.""" @@ -244,6 +252,8 @@ def sol_attention( sol_threshold_type=threshold_type, sol_kv_splits=kv_splits, sol_fp8=sol_fp8, + sol_fp8_layer_start=sol_fp8_layer_start, + sol_fp8_layer_end=sol_fp8_layer_end, ), **kwargs, ) diff --git a/telefuser/kernel/sol_attn/sm90/mainloop.py b/telefuser/kernel/sol_attn/sm90/mainloop.py index a76f47e..c1ee378 100644 --- a/telefuser/kernel/sol_attn/sm90/mainloop.py +++ b/telefuser/kernel/sol_attn/sm90/mainloop.py @@ -2088,7 +2088,7 @@ def mma_one_warpgroup_sol_attn_route_tma( seqlen, softmax, ) - if const_expr(self.fp8_inputs and self.sol_attn_num_splits == 1): + if const_expr(self.fp8_inputs): self.sol_attn_scale_route_probabilities( acc_S, tScS_route_mn, diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index e58218e..5647543 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -148,7 +148,13 @@ def _prepare_sol_qkv( v: torch.Tensor, sparse_state: SparseAttentionState | None, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None]: + fp8_layer_active = False if self._is_sol_active(sparse_state) and sparse_state is not None and sparse_state.config.sol_fp8: + layer_end = sparse_state.config.sol_fp8_layer_end + fp8_layer_active = sparse_state.layer_idx >= sparse_state.config.sol_fp8_layer_start and ( + layer_end is None or sparse_state.layer_idx < layer_end + ) + if fp8_layer_active: if q.is_cuda and torch.cuda.get_device_capability(q.device) == (9, 0): q, k, v, q_scale, k_scale, v_scale = quantize_fp8_qkv(q, k, v) return q, k, v, (q_scale, k_scale, v_scale) diff --git a/tests/unit/models/test_wan_video_sol_attention.py b/tests/unit/models/test_wan_video_sol_attention.py index c62b342..ae368f8 100644 --- a/tests/unit/models/test_wan_video_sol_attention.py +++ b/tests/unit/models/test_wan_video_sol_attention.py @@ -158,6 +158,27 @@ def fake_attention(q, k, v, **kwargs): assert torch.isfinite(restored).all() +def test_wan_self_attention_limits_fp8_sol_to_configured_layers() -> None: + module = SelfAttention(dim=128, num_heads=1) + config = SparseAttentionConfig( + sparse_impl="sol", + dense_timesteps=0, + sol_fp8=True, + sol_fp8_layer_start=1, + sol_fp8_layer_end=2, + ) + state = SparseAttentionState(config, mask_map=None) + q = torch.randn(1, 64, 1, 128) + + bf16_qkv = module._prepare_sol_qkv(q, q, q, state) + assert bf16_qkv[3] is None + + state.update(layer_idx=1) + fp8_qkv = module._prepare_sol_qkv(q, q, q, state) + assert fp8_qkv[0].dtype is torch.float8_e4m3fn + assert fp8_qkv[3] is not None + + @pytest.mark.gpu def test_wan_self_attention_executes_sol_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): @@ -314,3 +335,29 @@ def test_native_fp8_sol_preserves_constant_values_at_long_sequence_on_h100() -> ) torch.testing.assert_close(output.float(), torch.ones_like(output, dtype=torch.float32), rtol=0.02, atol=0.02) + + +@pytest.mark.gpu +def test_native_fp8_sol_split_preserves_sparse_route_weights_on_h100() -> None: + if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): + pytest.skip("split FP8 Sol-Attn test requires H100") + + torch.manual_seed(7) + q = torch.randn(1, 4160, 1, 128, device="cuda", dtype=torch.bfloat16) + k = torch.randn_like(q) + v = torch.randn_like(q) + reference = attention_impl.sol_attn(q, k, v, tau=1.0, kv_splits=1) + q, k, v, q_scale, k_scale, v_scale = quantize_fp8_qkv(q, k, v) + output = attention_impl.sol_attn( + q, + k, + v, + tau=1.0, + kv_splits=2, + q_scale=q_scale, + k_scale=k_scale, + v_scale=v_scale, + ) + + cosine = torch.nn.functional.cosine_similarity(output.float().flatten(), reference.float().flatten(), dim=0) + assert cosine > 0.98 diff --git a/tests/unit/pipelines/wan_video/test_optimized_example.py b/tests/unit/pipelines/wan_video/test_optimized_example.py index 0e531da..3917666 100644 --- a/tests/unit/pipelines/wan_video/test_optimized_example.py +++ b/tests/unit/pipelines/wan_video/test_optimized_example.py @@ -6,6 +6,7 @@ from examples.wan_video.wan21_1_3b_text_to_video_optimized_h100 import ( make_attention_config, make_quant_config, + resolve_fp8_linear_scope, run, ) from telefuser.core.config import AttnImplType, QuantConfig, QuantKernelBackend, QuantType @@ -33,10 +34,12 @@ def test_wan_optimized_example_builds_dense_attention_config() -> None: def test_wan_optimized_example_builds_fp8_sol_config() -> None: - attention = make_attention_config("sol-fp8") + attention = make_attention_config("sol-fp8", fp8_layer_start=10, fp8_layer_end=20) assert attention.attn_impl is AttnImplType.SOL_ATTN assert attention.sparse_config is not None assert attention.sparse_config.sol_fp8 + assert attention.sparse_config.sol_fp8_layer_start == 10 + assert attention.sparse_config.sol_fp8_layer_end == 20 def test_wan_optimized_example_rejects_unknown_attention() -> None: @@ -68,6 +71,22 @@ def test_wan_optimized_example_rejects_unknown_quantization() -> None: make_quant_config("int8") +def test_wan_optimized_example_builds_ffn_only_fp8_config() -> None: + config = make_quant_config("tf-kernel-fp8", fp8_linear_scope="ffn") + + assert config.quantize_modules == (".ffn.",) + + +def test_wan_optimized_example_uses_quality_safe_auto_fp8_scope() -> None: + assert resolve_fp8_linear_scope("dense", "auto") == "all" + assert resolve_fp8_linear_scope("sol-fp8", "auto") == "ffn" + + +def test_wan_optimized_example_rejects_unknown_fp8_linear_scope() -> None: + with pytest.raises(ValueError, match="fp8_linear_scope must be"): + resolve_fp8_linear_scope("sol-fp8", "attention") + + def test_wan_model_enables_tf_kernel_fp8_on_transformer_blocks(monkeypatch: pytest.MonkeyPatch) -> None: model = WanModel.__new__(WanModel) torch.nn.Module.__init__(model) From 79fc52029e1404ab276f12c04df11f217e4faf41 Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Fri, 14 Aug 2026 08:47:53 +0000 Subject: [PATCH 11/15] feat: add FP8 dense attention ablation for Wan --- docs/en/attention.md | 6 +- docs/zh/attention.md | 6 +- examples/wan_video/README.md | 73 +++++++++++-------- ...wan21_1_3b_text_to_video_optimized_h100.py | 56 ++++++++------ .../models/test_wan_video_sol_attention.py | 8 +- .../wan_video/test_optimized_example.py | 24 +++++- 6 files changed, 109 insertions(+), 64 deletions(-) diff --git a/docs/en/attention.md b/docs/en/attention.md index 56a21b2..0634ff0 100644 --- a/docs/en/attention.md +++ b/docs/en/attention.md @@ -71,7 +71,7 @@ class SparseAttentionConfig: sol_tau: float = 1.0 # Sol-Attn routing threshold sol_threshold_type: str = "diag" # "diag" or "exact" sol_kv_splits: int | str = "auto" # "auto", 1, 2, or 4 - sol_fp8: bool = False # Native FP8 Q/K/V Sol-Attn on SM90 + sol_fp8: bool = False # FP8 Q/K/V Sol-Attn on SM90 sol_fp8_layer_start: int = 0 # First layer using FP8 Sol-Attn sol_fp8_layer_end: int | None = None # Exclusive end; None means all remaining layers ``` @@ -210,6 +210,10 @@ requires log-sum-exp output. `sol_fp8_layer_start` and `sol_fp8_layer_end` restrict E4M3 Q/K/V to a half-open transformer-layer range. Sparse layers outside that range continue to use BF16 Sol-Attn. This controls accumulated FP8 routing error in diffusion models. +Setting `dense_timesteps=0`, `dense_layers=0`, and a negative `tau` forces all +KV blocks onto the exact route. The Wan optimized example exposes this as +`--attention fp8-dense`; `--attention fp8-sol` enables centroid routing with the +same FP8 Q/K/V and QK/PV kernel. ### QwenImagePipeline / ZImagePipeline diff --git a/docs/zh/attention.md b/docs/zh/attention.md index 0d5a13b..365cc41 100644 --- a/docs/zh/attention.md +++ b/docs/zh/attention.md @@ -71,7 +71,7 @@ class SparseAttentionConfig: sol_tau: float = 1.0 # Sol-Attn 路由阈值 sol_threshold_type: str = "diag" # "diag" 或 "exact" sol_kv_splits: int | str = "auto" # "auto"、1、2 或 4 - sol_fp8: bool = False # SM90 原生 FP8 Q/K/V Sol-Attn + sol_fp8: bool = False # SM90 FP8 Q/K/V Sol-Attn sol_fp8_layer_start: int = 0 # 启用 FP8 Sol-Attn 的首层 sol_fp8_layer_end: int | None = None # 结束层（不包含）；None 表示其余所有层 ``` @@ -209,6 +209,10 @@ Ring/USP 需要 LSE 做在线合并，因此仍使用支持 LSE 的密集后端 的 transformer 层，区间外的稀疏层继续使用 BF16 Sol-Attn，以控制扩散模型中 逐层累积的 FP8 路由误差。 +设置 `dense_timesteps=0`、`dense_layers=0` 和负数 `tau` 会强制所有 KV block +走 exact 路径。Wan 优化示例将其暴露为 `--attention fp8-dense`； +`--attention fp8-sol` 使用相同的 FP8 Q/K/V 与 QK/PV kernel 并启用质心路由。 + ### QwenImagePipeline / ZImagePipeline 仅支持密集注意力（图像生成没有时序维度）： diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index f96b201..290e0e2 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -180,14 +180,15 @@ run the BF16 baseline. Attention choices: -- `dense`: PyTorch SDPA -- `sol`: Sol-Attn with dense warm-up and fallback calls -- `sol-fp8`: native E4M3 Q/K/V Sol-Attn on H100 +- `dense`: BF16 PyTorch SDPA +- `sol`: BF16 Sol-Attn with dense warm-up and fallback calls +- `fp8-dense`: E4M3 Q/K/V with FP8 QK/PV WGMMA; all KV blocks are exact +- `fp8-sol`: the same FP8 kernel with Sol routing enabled -For `sol-fp8`, `--sol-fp8-layer-start` and `--sol-fp8-layer-end` select the -half-open transformer-layer range that uses FP8 Q/K/V. Other sparse layers use -BF16 Sol-Attn. Restricting FP8 Q/K/V to middle layers avoids accumulating small -routing changes across the full denoiser. +For both FP8 modes, `--fp8-layer-start` and `--fp8-layer-end` select the +half-open transformer-layer range that uses FP8 Q/K/V. Other layers use the +corresponding BF16 exact or Sol path. Restricting FP8 Q/K/V to middle layers +avoids accumulating small quantization changes across the full denoiser. Quantization choices: @@ -196,10 +197,10 @@ Quantization choices: `--fp8-linear-scope all` quantizes every transformer-block Linear layer. `--fp8-linear-scope ffn` keeps self/cross-attention projections in BF16 and -quantizes the 60 FFN Linear layers. The latter is recommended when Linear FP8 -and FP8 Sol-Attn are enabled together. The default `auto` selects `all` for -dense attention and `ffn` for `sol-fp8`; the default FP8 Sol layer range is -10-19 for this 30-layer Wan2.1 model. +quantizes the 60 FFN Linear layers. The default `auto` selects `all`; generated +video validation shows that all-Linear FP8 preserves quality. Attention Q/K/V +are more sensitive, so their default FP8 layer range is 10-19 for this 30-layer +Wan2.1 model. Only DiT transformer-block Linear layers are quantized; the VAE and text encoder remain BF16. Select the two optimization axes independently: @@ -215,19 +216,21 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ --attention sol --quantization none -# Dense + tf-kernel FP8 +# FP8 Dense: exact FP8 attention + FP8 Linear python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ - --attention dense --quantization tf-kernel-fp8 + --attention fp8-dense \ + --quantization tf-kernel-fp8 \ + --fp8-layer-start 10 \ + --fp8-layer-end 20 -# Sol-Attn + tf-kernel FP8 +# FP8 Sol: routed FP8 attention + FP8 Linear python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ - --attention sol-fp8 \ + --attention fp8-sol \ --quantization tf-kernel-fp8 \ - --fp8-linear-scope ffn \ - --sol-fp8-layer-start 10 \ - --sol-fp8-layer-end 20 \ + --fp8-layer-start 10 \ + --fp8-layer-end 20 \ --dense-timesteps 10 \ --dense-layers 1 \ --tau 1.0 \ @@ -235,15 +238,17 @@ python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --kv-splits auto ``` -`tf-kernel-fp8` is the FP8-GEMM implementation benchmarked below. On H100, -`sol-fp8` quantizes post-RoPE Q/K/V to E4M3 and runs QK and PV through the CuTe -SM90 WGMMA mainloop. Q/K use one scale per 64-token block, V uses per-channel -scales and a K-major layout, and FP32 accumulators are used throughout. `auto` -selects four KV splits for long FP8 sequences to bound Hopper FP8 accumulation -error. Partial tiles are physically padded while the original sequence length -remains masked in the kernel. FP8 split execution restores the represented N64 -route length before PV, matching the BF16 summed-centroid contract. The Sol -tuning options apply to both modes. +In this example, FP8 means the E4M3 attention implementation rather than a +BF16-attention run with only its Linear layers quantized. Post-RoPE Q/K/V are +quantized and QK/PV run through the CuTe SM90 WGMMA mainloop. Q/K use one scale +per 64-token block, V uses per-channel scales and a K-major layout, and FP32 +accumulators are used throughout. `fp8-dense` forces every routed KV block onto +the exact path, while `fp8-sol` permits centroid approximation. `auto` selects +four KV splits for long FP8 sequences to bound Hopper FP8 accumulation error. +Partial tiles are physically padded while the original sequence length remains +masked in the kernel. FP8 split execution restores the represented N64 route +length before PV, matching the BF16 summed-centroid contract. The accompanying +FP8 Linear GEMMs use the tf-kernel backend. The final log reports generation time, frames per second, and peak allocated and reserved CUDA memory. @@ -260,9 +265,14 @@ same generation interval. | --- | --- | ---: | ---: | | BF16 | Dense | 0.8491 | 16.147 | | BF16 | Sol-Attn | 1.1090 | 17.023 | -| tf-kernel FP8 | Dense | 0.8771 | 14.855 | -| tf-kernel FP8 | Sol-FP8 (all Linear/QKV layers, aggressive) | 1.1879 | 15.730 | -| tf-kernel FP8 FFN | Sol-FP8 (QKV layers 10-19, precision) | 1.0347 | 16.256 | +| FP8 | Dense (Q/K/V layers 10-19, exact) | 0.8392 | 15.730 | +| FP8 | Sol-Attn (Q/K/V layers 10-19) | 1.0202 | 15.730 | + +Both FP8 rows quantize all 300 transformer-block Linear layers and use the same +E4M3 attention layer range. FP8 Dense therefore measures this implementation's +exact QK/PV path, not BF16 SDPA with only Linear quantization. Against the +corresponding BF16 output, FP8 Dense measures 22.3664 dB PSNR / 0.839404 SSIM, +and FP8 Sol measures 20.8865 dB PSNR / 0.795351 SSIM. The benchmark prompt is: @@ -275,9 +285,10 @@ Example command (change `--attention` and `--quantization` for each ablation): python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ --model-root /path/to/Wan2.1-T2V-1.3B \ --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ - --attention sol \ + --attention fp8-sol \ --quantization tf-kernel-fp8 \ --fp8-linear-scope all \ + --fp8-layer-start 10 --fp8-layer-end 20 \ --width 832 --height 480 \ --num-frames 81 --num-inference-steps 50 \ --sample-solver unipc --cfg-scale 5.0 --sigma-shift 5.0 --seed 42 diff --git a/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py index d7fb036..f9abafa 100644 --- a/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py +++ b/examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py @@ -1,8 +1,8 @@ """Wan2.1 1.3B T2V with optional attention and quantization optimizations. -Attention can use dense SDPA or Sol-Attn. DiT Linear layers can remain BF16 or -use tf-kernel FP8, TorchAO FP8, or bitsandbytes NF4. The example keeps the DiT -on CUDA so quantized Linear modules are not repeatedly reconstructed. +Attention can use BF16 dense/Sol or FP8 dense/Sol kernels. DiT Linear layers can +remain BF16 or use tf-kernel FP8, TorchAO FP8, or bitsandbytes NF4. The example +keeps the DiT on CUDA so quantized Linear modules are not repeatedly rebuilt. """ from __future__ import annotations @@ -40,6 +40,8 @@ "sigma_shift": 8.0, } +FP8_ATTENTION_MODES = ("fp8-dense", "fp8-sol") + def configure_attention_backends() -> None: """Configure dense attention backends used directly or by SOL fallbacks.""" @@ -82,9 +84,9 @@ def make_quant_config(quantization: str, *, fp8_linear_scope: str = "all") -> Qu def resolve_fp8_linear_scope(attention: str, fp8_linear_scope: str) -> str: - """Resolve the quality-safe scope for the selected attention mode.""" + """Resolve the Linear FP8 scope for the selected attention mode.""" if fp8_linear_scope == "auto": - return "ffn" if attention == "sol-fp8" else "all" + return "all" if fp8_linear_scope not in ("all", "ffn"): raise ValueError("fp8_linear_scope must be 'auto', 'all', or 'ffn'") return fp8_linear_scope @@ -101,18 +103,22 @@ def make_attention_config( fp8_layer_start: int = 10, fp8_layer_end: int | None = 20, ) -> AttentionConfig: - """Build the selected dense or Sol-Attn configuration.""" + """Build the selected BF16 or FP8 dense/Sol attention configuration.""" if attention == "dense": return AttentionConfig.dense_attention(AttnImplType.TORCH_SDPA) - if attention not in ("sol", "sol-fp8"): - raise ValueError("attention must be 'dense', 'sol', or 'sol-fp8'") + if attention not in ("sol", *FP8_ATTENTION_MODES): + raise ValueError("attention must be 'dense', 'sol', 'fp8-dense', or 'fp8-sol'") + + fp8_dense = attention == "fp8-dense" return AttentionConfig.sol_attention( - dense_timesteps=dense_timesteps, - dense_layers=dense_layers, - tau=tau, + dense_timesteps=0 if fp8_dense else dense_timesteps, + dense_layers=0 if fp8_dense else dense_layers, + # A negative threshold makes every routed KV block exact. This runs the + # FP8 CuTe mainloop without introducing Sol's centroid approximation. + tau=-1000.0 if fp8_dense else tau, threshold_type=threshold_type, kv_splits=kv_splits, - sol_fp8=attention == "sol-fp8", + sol_fp8=attention in FP8_ATTENTION_MODES, sol_fp8_layer_start=fp8_layer_start, sol_fp8_layer_end=fp8_layer_end, ) @@ -129,8 +135,8 @@ def get_pipeline( tau: float = 1.0, threshold_type: str = "diag", kv_splits: int | str = "auto", - sol_fp8_layer_start: int = 10, - sol_fp8_layer_end: int | None = 20, + fp8_layer_start: int = 10, + fp8_layer_end: int | None = 20, sample_solver: str = "euler", ) -> Wan21VideoPipeline: """Load Wan2.1 with independently selectable attention and quantization.""" @@ -155,8 +161,8 @@ def get_pipeline( tau=tau, threshold_type=threshold_type, kv_splits=kv_splits, - fp8_layer_start=sol_fp8_layer_start, - fp8_layer_end=sol_fp8_layer_end, + fp8_layer_start=fp8_layer_start, + fp8_layer_end=fp8_layer_end, ) config.dit_config.quant_config = quant_config config.dit_config.offload_config.offload_type = WeightOffloadType.NO_CPU_OFFLOAD @@ -212,7 +218,11 @@ def run( @click.option("--sigma-shift", default=PPL_CONFIG["sigma_shift"], type=float) @click.option("--sample-solver", default="euler", type=click.Choice(["euler", "unipc"])) @click.option("--model-root", default=PPL_CONFIG["model_root"]) -@click.option("--attention", default="dense", type=click.Choice(["dense", "sol", "sol-fp8"])) +@click.option( + "--attention", + default="dense", + type=click.Choice(["dense", "sol", "fp8-dense", "fp8-sol"]), +) @click.option( "--quantization", default="none", @@ -224,8 +234,8 @@ def run( @click.option("--tau", default=1.0, type=float) @click.option("--threshold-type", default="diag", type=click.Choice(["diag", "exact"])) @click.option("--kv-splits", default="auto", type=click.Choice(["auto", "1", "2", "4"])) -@click.option("--sol-fp8-layer-start", default=10, type=int) -@click.option("--sol-fp8-layer-end", default=20, type=int) +@click.option("--fp8-layer-start", "--sol-fp8-layer-start", default=10, type=int) +@click.option("--fp8-layer-end", "--sol-fp8-layer-end", default=20, type=int) @click.option("--output", default=get_example_name(__file__, "mp4")) def main( prompt: str, @@ -247,8 +257,8 @@ def main( tau: float, threshold_type: str, kv_splits: str, - sol_fp8_layer_start: int, - sol_fp8_layer_end: int | None, + fp8_layer_start: int, + fp8_layer_end: int | None, output: str, ) -> None: """Run Wan2.1 with optional attention and quantization optimizations.""" @@ -263,8 +273,8 @@ def main( tau=tau, threshold_type=threshold_type, kv_splits=kv_splits if kv_splits == "auto" else int(kv_splits), - sol_fp8_layer_start=sol_fp8_layer_start, - sol_fp8_layer_end=sol_fp8_layer_end, + fp8_layer_start=fp8_layer_start, + fp8_layer_end=fp8_layer_end, sample_solver=sample_solver, ) torch.cuda.reset_peak_memory_stats() diff --git a/tests/unit/models/test_wan_video_sol_attention.py b/tests/unit/models/test_wan_video_sol_attention.py index ae368f8..aec8bbc 100644 --- a/tests/unit/models/test_wan_video_sol_attention.py +++ b/tests/unit/models/test_wan_video_sol_attention.py @@ -213,7 +213,7 @@ def tracked_sol_attn(*args, **kwargs): @pytest.mark.gpu -def test_wan_self_attention_executes_native_fp8_sol_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: +def test_wan_self_attention_executes_fp8_sol_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): pytest.skip("Wan FP8 Sol-Attn execution test requires H100") @@ -287,7 +287,7 @@ def test_fused_fp8_qkv_quantization_on_h100() -> None: @pytest.mark.gpu -def test_native_fp8_sol_handles_partial_tail_on_h100() -> None: +def test_fp8_sol_handles_partial_tail_on_h100() -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): pytest.skip("partial-tail FP8 Sol-Attn test requires H100") @@ -316,7 +316,7 @@ def test_native_fp8_sol_handles_partial_tail_on_h100() -> None: @pytest.mark.gpu -def test_native_fp8_sol_preserves_constant_values_at_long_sequence_on_h100() -> None: +def test_fp8_sol_preserves_constant_values_at_long_sequence_on_h100() -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): pytest.skip("long-sequence FP8 Sol-Attn test requires H100") @@ -338,7 +338,7 @@ def test_native_fp8_sol_preserves_constant_values_at_long_sequence_on_h100() -> @pytest.mark.gpu -def test_native_fp8_sol_split_preserves_sparse_route_weights_on_h100() -> None: +def test_fp8_sol_split_preserves_sparse_route_weights_on_h100() -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): pytest.skip("split FP8 Sol-Attn test requires H100") diff --git a/tests/unit/pipelines/wan_video/test_optimized_example.py b/tests/unit/pipelines/wan_video/test_optimized_example.py index 3917666..38b3eb0 100644 --- a/tests/unit/pipelines/wan_video/test_optimized_example.py +++ b/tests/unit/pipelines/wan_video/test_optimized_example.py @@ -34,10 +34,24 @@ def test_wan_optimized_example_builds_dense_attention_config() -> None: def test_wan_optimized_example_builds_fp8_sol_config() -> None: - attention = make_attention_config("sol-fp8", fp8_layer_start=10, fp8_layer_end=20) + attention = make_attention_config("fp8-sol", fp8_layer_start=10, fp8_layer_end=20) assert attention.attn_impl is AttnImplType.SOL_ATTN assert attention.sparse_config is not None assert attention.sparse_config.sol_fp8 + assert attention.sparse_config.sol_tau == 1.0 + assert attention.sparse_config.sol_fp8_layer_start == 10 + assert attention.sparse_config.sol_fp8_layer_end == 20 + + +def test_wan_optimized_example_builds_fp8_dense_config() -> None: + attention = make_attention_config("fp8-dense", fp8_layer_start=10, fp8_layer_end=20) + + assert attention.attn_impl is AttnImplType.SOL_ATTN + assert attention.sparse_config is not None + assert attention.sparse_config.sol_fp8 + assert attention.sparse_config.dense_timesteps == 0 + assert attention.sparse_config.dense_layers == 0 + assert attention.sparse_config.sol_tau == -1000.0 assert attention.sparse_config.sol_fp8_layer_start == 10 assert attention.sparse_config.sol_fp8_layer_end == 20 @@ -77,14 +91,16 @@ def test_wan_optimized_example_builds_ffn_only_fp8_config() -> None: assert config.quantize_modules == (".ffn.",) -def test_wan_optimized_example_uses_quality_safe_auto_fp8_scope() -> None: +def test_wan_optimized_example_uses_all_linear_layers_for_auto_fp8_scope() -> None: assert resolve_fp8_linear_scope("dense", "auto") == "all" - assert resolve_fp8_linear_scope("sol-fp8", "auto") == "ffn" + assert resolve_fp8_linear_scope("sol", "auto") == "all" + assert resolve_fp8_linear_scope("fp8-dense", "auto") == "all" + assert resolve_fp8_linear_scope("fp8-sol", "auto") == "all" def test_wan_optimized_example_rejects_unknown_fp8_linear_scope() -> None: with pytest.raises(ValueError, match="fp8_linear_scope must be"): - resolve_fp8_linear_scope("sol-fp8", "attention") + resolve_fp8_linear_scope("fp8-sol", "attention") def test_wan_model_enables_tf_kernel_fp8_on_transformer_blocks(monkeypatch: pytest.MonkeyPatch) -> None: From ce35916deab976967b1eb7dd85035a584e829aef Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Fri, 14 Aug 2026 10:22:19 +0000 Subject: [PATCH 12/15] perf: accelerate Wan FP8 Sol attention --- examples/wan_video/README.md | 19 +++-- telefuser/core/config.py | 2 +- telefuser/kernel/sol_attn/interface.py | 11 ++- telefuser/kernel/sol_attn/sm90/mainloop.py | 8 +- .../kernel/sol_attn/triton_ref/preprocess.py | 6 +- telefuser/models/wan_video_dit.py | 24 ++++-- telefuser/ops/attention/attention_impl.py | 12 ++- telefuser/ops/fp8_attention.py | 9 +-- telefuser/ops/fp8_gemm.py | 75 +++++++++++++------ .../models/test_wan_video_sol_attention.py | 33 ++++++-- tests/unit/ops/test_fp8_gemm.py | 48 ++++++++++++ tests/unit/ops/test_sol_attention.py | 54 +++++++++++++ 12 files changed, 236 insertions(+), 65 deletions(-) diff --git a/examples/wan_video/README.md b/examples/wan_video/README.md index 290e0e2..64852c9 100644 --- a/examples/wan_video/README.md +++ b/examples/wan_video/README.md @@ -187,7 +187,7 @@ Attention choices: For both FP8 modes, `--fp8-layer-start` and `--fp8-layer-end` select the half-open transformer-layer range that uses FP8 Q/K/V. Other layers use the -corresponding BF16 exact or Sol path. Restricting FP8 Q/K/V to middle layers +corresponding BF16 dense or Sol path. Restricting FP8 Q/K/V to middle layers avoids accumulating small quantization changes across the full denoiser. Quantization choices: @@ -244,11 +244,16 @@ quantized and QK/PV run through the CuTe SM90 WGMMA mainloop. Q/K use one scale per 64-token block, V uses per-channel scales and a K-major layout, and FP32 accumulators are used throughout. `fp8-dense` forces every routed KV block onto the exact path, while `fp8-sol` permits centroid approximation. `auto` selects -four KV splits for long FP8 sequences to bound Hopper FP8 accumulation error. +two KV splits for long FP8 sequences, which is faster at Wan's sequence length +without changing the FP32 accumulation contract. Partial tiles are physically padded while the original sequence length remains masked in the kernel. FP8 split execution restores the represented N64 route length before PV, matching the BF16 summed-centroid contract. The accompanying -FP8 Linear GEMMs use the tf-kernel backend. +FP8 Linear GEMMs use the tf-kernel backend. Self-attention Q/K/V projections +share one dynamic activation quantization instead of quantizing the same input +three times. With a partial FP8 layer range, FP8 Dense sends unquantized layers +to SDPA and FP8 Sol sends unquantized sparse layers to Triton, avoiding a second +CuTe specialization in the cold-start path. The final log reports generation time, frames per second, and peak allocated and reserved CUDA memory. @@ -265,14 +270,14 @@ same generation interval. | --- | --- | ---: | ---: | | BF16 | Dense | 0.8491 | 16.147 | | BF16 | Sol-Attn | 1.1090 | 17.023 | -| FP8 | Dense (Q/K/V layers 10-19, exact) | 0.8392 | 15.730 | -| FP8 | Sol-Attn (Q/K/V layers 10-19) | 1.0202 | 15.730 | +| FP8 | Dense (Q/K/V layers 10-19, exact) | 0.8739 | 15.730 | +| FP8 | Sol-Attn (Q/K/V layers 10-19) | 1.1565 | 15.730 | Both FP8 rows quantize all 300 transformer-block Linear layers and use the same E4M3 attention layer range. FP8 Dense therefore measures this implementation's exact QK/PV path, not BF16 SDPA with only Linear quantization. Against the -corresponding BF16 output, FP8 Dense measures 22.3664 dB PSNR / 0.839404 SSIM, -and FP8 Sol measures 20.8865 dB PSNR / 0.795351 SSIM. +corresponding BF16 output, FP8 Dense measures 22.0257 dB PSNR / 0.828783 SSIM, +and FP8 Sol measures 20.8502 dB PSNR / 0.792656 SSIM. The benchmark prompt is: diff --git a/telefuser/core/config.py b/telefuser/core/config.py index 0c2946c..7bd1f5f 100644 --- a/telefuser/core/config.py +++ b/telefuser/core/config.py @@ -150,7 +150,7 @@ class SparseAttentionConfig: use_sage_attention: bool = False # Use sage attention backend sol_tau: float = 1.0 # Sol-Attn routing threshold multiplier sol_threshold_type: str = "diag" # Sol-Attn threshold estimator: "diag" or "exact" - sol_kv_splits: int | str = "auto" # Auto selects split 4 for long SM90 sequences + sol_kv_splits: int | str = "auto" # Auto selects split 2 for long FP8 SM90 sequences sol_fp8: bool = False # Quantize post-RoPE Q/K/V activations for FP8 Sol-Attn sol_fp8_layer_start: int = 0 # First transformer layer using FP8 Sol-Attn sol_fp8_layer_end: int | None = None # Exclusive end; None enables all remaining layers diff --git a/telefuser/kernel/sol_attn/interface.py b/telefuser/kernel/sol_attn/interface.py index 0800ca4..1843222 100644 --- a/telefuser/kernel/sol_attn/interface.py +++ b/telefuser/kernel/sol_attn/interface.py @@ -400,6 +400,7 @@ def sol_attn( q_scale: torch.Tensor | None = None, k_scale: torch.Tensor | None = None, v_scale: torch.Tensor | None = None, + force_triton: bool = False, ) -> torch.Tensor: """Compute noncausal Sol-Attn for contiguous BF16 or FP8 BTHD tensors. @@ -410,6 +411,8 @@ def sol_attn( fp8_inputs = any(x.dtype == torch.float8_e4m3fn for x in (q, k, v)) if fp8_inputs: + if force_triton: + raise ValueError("force_triton is only supported for BF16 Sol-Attn") if kv_splits not in (1, 2, 4): raise ValueError("kv_splits must be 1, 2, or 4") if not all(x.dtype == torch.float8_e4m3fn for x in (q, k, v)): @@ -420,11 +423,7 @@ def sol_attn( arch = tuple(torch.cuda.get_device_capability(q.device)) native_sm90_fp8 = arch == (9, 0) and _cute_runtime_available() blocks = (q.shape[1] + BLOCK_SIZE - 1) // BLOCK_SIZE - expected_scale_shape = ( - (q.shape[0], blocks * BLOCK_SIZE, q.shape[2]) - if native_sm90_fp8 - else (q.shape[0], blocks, q.shape[2]) - ) + expected_scale_shape = (q.shape[0], blocks, q.shape[2]) for name, tensor in (("q_scale", q_scale), ("k_scale", k_scale)): if tensor.shape != expected_scale_shape or tensor.device != q.device: raise ValueError(f"{name} must have shape {expected_scale_shape} on the Q/K/V device") @@ -483,7 +482,7 @@ def sol_attn( ) if kv_splits not in (1, 2, 4): raise ValueError("kv_splits must be 1, 2, or 4") - backend = _backend_for_arch(arch) + backend = "triton" if force_triton else _backend_for_arch(arch) scale = q.shape[-1] ** -0.5 if scale is None else float(scale) tau = float(tau) diff --git a/telefuser/kernel/sol_attn/sm90/mainloop.py b/telefuser/kernel/sol_attn/sm90/mainloop.py index c1ee378..047dbb1 100644 --- a/telefuser/kernel/sol_attn/sm90/mainloop.py +++ b/telefuser/kernel/sol_attn/sm90/mainloop.py @@ -72,7 +72,7 @@ def __init__( self.fp8_probability_scale = 1.0 self.sol_attn_group_size = 64 self.sol_attn_group_words = 2 - self.mma_pv_is_rs = True + self.mma_pv_is_rs = not fp8_inputs self.sol_attn_mma_regs_override = 128 self.sol_attn_warp_route_mask = True self.sol_attn_fast_route_lens = True @@ -232,9 +232,7 @@ def sol_attn_scale_exact_scores( """Apply one Q/K scale per N64 tile to an FP8 QK accumulator.""" acc_S_mn = layout_utils.reshape_acc_to_mn(acc_S) - factor = Float32(q_scale[q_block * Int32(self.tile_m)]) * Float32( - k_scale[n_block * Int32(self.tile_n)] - ) + factor = Float32(q_scale[q_block]) * Float32(k_scale[n_block]) for i in cutlass.range_constexpr(cute.size(acc_S_mn)): acc_S_mn[i] = Float32(acc_S_mn[i]) * factor @@ -251,7 +249,7 @@ def sol_attn_scale_route_scores( """Apply block-scaled Q and per-centroid K dequantization.""" acc_S_mn = layout_utils.reshape_acc_to_mn(acc_S) - q_factor = Float32(q_scale[q_block * Int32(self.tile_m)]) + q_factor = Float32(q_scale[q_block]) for i in cutlass.range_constexpr(cute.size(acc_S_mn)): col = tScS_mn[i][1] factor = q_factor * Float32(kc_scale[route_n_block + col]) diff --git a/telefuser/kernel/sol_attn/triton_ref/preprocess.py b/telefuser/kernel/sol_attn/triton_ref/preprocess.py index 13f60c8..bec3bb8 100644 --- a/telefuser/kernel/sol_attn/triton_ref/preprocess.py +++ b/telefuser/kernel/sol_attn/triton_ref/preprocess.py @@ -361,7 +361,7 @@ def prepare_sm90_fp8( head_dim, BLOCK_SIZE, FP8=True, - TOKEN_SCALES=True, + TOKEN_SCALES=False, V_CHANNEL_SCALE=True, V_TOKEN_CONTIGUOUS=True, SM90_FP8_OUTPUTS=True, @@ -438,7 +438,7 @@ def _compute_diag_threshold( BLOCK_SIZE, tau, FP8=q.dtype == torch.float8_e4m3fn, - TOKEN_SCALES=q_scale is not None and q_scale.shape[1] == padded_tokens, + TOKEN_SCALES=False, num_warps=4, num_stages=2, ) @@ -486,7 +486,7 @@ def _compute_exact_threshold( head_dim, BLOCK_SIZE, FP8=q.dtype == torch.float8_e4m3fn, - TOKEN_SCALES=q_scale is not None and q_scale.shape[1] == padded_tokens, + TOKEN_SCALES=False, num_warps=4, num_stages=1, ) diff --git a/telefuser/models/wan_video_dit.py b/telefuser/models/wan_video_dit.py index 5647543..9748408 100755 --- a/telefuser/models/wan_video_dit.py +++ b/telefuser/models/wan_video_dit.py @@ -38,6 +38,7 @@ from telefuser.ops.attention import MaskMap, SparseAttentionState from telefuser.ops.attention import attention as attn_func from telefuser.ops.fp8_attention import quantize_fp8_per_block, quantize_fp8_qkv +from telefuser.ops.fp8_gemm import FP8Linear, fp8_linear_forward_many from telefuser.ops.normalization import LayerNorm, RMSNorm, fused_scale_shift, modulate from telefuser.ops.rotary import apply_rotary_emb from telefuser.utils.logging import logger @@ -135,9 +136,15 @@ def _prepare_sol_projection_input( x: torch.Tensor, sparse_state: SparseAttentionState | None, ) -> torch.Tensor: - # TorchAO preserves Wan's FP32 residual dtype, unlike autocast nn.Linear. - # Cast once before q/k/v instead of casting all three projected tensors. - if self._is_sol_active(sparse_state) and x.dtype != torch.bfloat16 and torch.is_autocast_enabled(x.device.type): + # Cast once before q/k/v instead of inside all three FP8 projections. + # Keeping the outer projection dtype in sync also avoids a redundant + # BF16 -> FP32 -> BF16 round trip before the output FP8 Linear. + shared_fp8_qkv = all(isinstance(projection, FP8Linear) for projection in (self.q, self.k, self.v)) + if ( + (self._is_sol_active(sparse_state) or shared_fp8_qkv) + and x.dtype != torch.bfloat16 + and torch.is_autocast_enabled(x.device.type) + ): return x.to(torch.bfloat16) return x @@ -233,9 +240,14 @@ def default_forward( input_dtype = x.dtype x = self._prepare_sol_projection_input(x, sparse_state) projection_dtype = x.dtype - q = self.norm_q(self.q(x)) - k = self.norm_k(self.k(x)) - v = self.v(x) + if all(isinstance(projection, FP8Linear) for projection in (self.q, self.k, self.v)): + q, k, v = fp8_linear_forward_many((self.q, self.k, self.v), x) + q = self.norm_q(q) + k = self.norm_k(k) + else: + q = self.norm_q(self.q(x)) + k = self.norm_k(self.k(x)) + v = self.v(x) q = rope_apply(q, freqs_cos, freqs_sin, self.num_heads) k = rope_apply(k, freqs_cos, freqs_sin, self.num_heads) q = rearrange(q, "b s (n d) -> b s n d", n=self.num_heads) diff --git a/telefuser/ops/attention/attention_impl.py b/telefuser/ops/attention/attention_impl.py index 5a2a79e..22162e7 100755 --- a/telefuser/ops/attention/attention_impl.py +++ b/telefuser/ops/attention/attention_impl.py @@ -168,7 +168,7 @@ def _resolve_sol_kv_splits(q: Tensor, kv_splits: int | str) -> int: return int(kv_splits) if torch.cuda.get_device_capability(q.device) == (9, 0): if q.dtype == torch.float8_e4m3fn and q.shape[1] >= 16384: - return 4 + return 2 if q.shape[1] >= 65536: return 4 return 1 @@ -243,6 +243,11 @@ def attention( elif attn_impl == AttnImplType.SOL_ATTN: if sparse_state.should_use_dense(): attn_impl = AttnImplType.FLASH_ATTN_2 + elif sparse_state.config.sol_fp8 and sparse_state.config.sol_tau < 0.0 and q.dtype == torch.bfloat16: + # FP8 Dense only needs the CuTe exact mainloop in quantized + # layers. Unquantized layers use the faster dense backend and + # avoid compiling a second BF16 CuTe specialization. + attn_impl = AttnImplType.TORCH_SDPA else: if sparse_state.mask_map is None: raise RuntimeError("Radial attention requires a mask map") @@ -438,6 +443,11 @@ def attention( q_scale=q_scale, k_scale=k_scale, v_scale=v_scale, + # A partial FP8 layer range otherwise compiles both BF16 + # and FP8 CuTe specializations on the first sparse step. + # Triton is a better cold-start tradeoff for the remaining + # sparse BF16 layers; exact FP8 Dense keeps CuTe throughout. + force_triton=(sparse_config.sol_fp8 and q.dtype == torch.bfloat16 and sparse_config.sol_tau >= 0.0), ) except (RuntimeError, TypeError, ValueError) as error: msg = "Sol-Attn execution failed, falling back to TORCH_SDPA" diff --git a/telefuser/ops/fp8_attention.py b/telefuser/ops/fp8_attention.py index 656967e..7a2f36d 100644 --- a/telefuser/ops/fp8_attention.py +++ b/telefuser/ops/fp8_attention.py @@ -39,9 +39,9 @@ def _quantize_qkv_fp8_stage1( q_s = tl.maximum(tl.max(tl.max(tl.abs(q_values), axis=1), axis=0), 1.0e-6) / 448.0 k_s = tl.maximum(tl.max(tl.max(tl.abs(k_values), axis=1), axis=0), 1.0e-6) / 448.0 - scale_offsets = (batch * tl.cdiv(tokens, block) * block + token_offsets) * heads + head - tl.store(q_scale + scale_offsets, q_s, mask=valid) - tl.store(k_scale + scale_offsets, k_s, mask=valid) + scale_offset = (batch * tl.cdiv(tokens, block) + block_idx) * heads + head + tl.store(q_scale + scale_offset, q_s) + tl.store(k_scale + scale_offset, k_s) tl.store(q_out + offsets, q_values / q_s, mask=valid[:, None]) tl.store(k_out + offsets, k_values / k_s, mask=valid[:, None]) @@ -93,8 +93,7 @@ def quantize_fp8_qkv( q_out = torch.empty(q.shape, device=q.device, dtype=torch.float8_e4m3fn) k_out = torch.empty_like(q_out) v_storage = torch.empty((batch, heads, head_dim, tokens), device=q.device, dtype=torch.float8_e4m3fn) - padded_tokens = blocks * FP8_ATTENTION_BLOCK_SIZE - q_scale = torch.ones((batch, padded_tokens, heads), device=q.device, dtype=torch.float32) + q_scale = torch.empty((batch, blocks, heads), device=q.device, dtype=torch.float32) k_scale = torch.ones_like(q_scale) v_scale = torch.zeros((batch, heads, head_dim), device=q.device, dtype=torch.float32) grid = (blocks, batch * heads) diff --git a/telefuser/ops/fp8_gemm.py b/telefuser/ops/fp8_gemm.py index 0cd04e3..5603161 100644 --- a/telefuser/ops/fp8_gemm.py +++ b/telefuser/ops/fp8_gemm.py @@ -239,45 +239,56 @@ def forward(self, x: torch.Tensor) -> torch.Tensor: "Use fp16_weight_storage='cpu_offload' (or 'keep') for CPU fallback." ) - # tf-kernel FP8 GEMM only supports fp16/bf16 outputs. + if x.dtype not in (torch.float16, torch.bfloat16) and not self.options.cast_inputs: + if self.linear is not None: + return self.linear(x) + if self._fp16_weight_cpu is not None: + weight = self._fp16_weight_cpu.to(device=x.device, dtype=x.dtype) + bias = self._fp16_bias_cpu + bias = bias.to(device=x.device, dtype=x.dtype) if bias is not None else None + return torch.nn.functional.linear(x, weight, bias) + raise RuntimeError("cast_inputs=False requires FP16 weights for fallback, but they were discarded.") + + x_fp, in_dtype, out_dtype = self._prepare_cuda_input(x) + x_shape = x_fp.shape + x_2d = x_fp.reshape(-1, x_shape[-1]).contiguous() + qinput = torch.empty_like(x_2d, dtype=torch.float8_e4m3fn) + input_scale = torch.empty((x_2d.shape[0], 1), dtype=torch.float32, device=x_fp.device) + self._tf_kernel.tf_per_token_quant_fp8(x_2d, qinput, input_scale) + return self._forward_quantized(qinput, input_scale, x_shape, in_dtype, out_dtype) + + def _prepare_cuda_input(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.dtype, torch.dtype]: + """Cast an activation to a dtype supported by tf-kernel FP8 GEMM.""" in_dtype = x.dtype if in_dtype not in (torch.float16, torch.bfloat16): - if not self.options.cast_inputs: - # Fall back if we still have FP16 weights. - if self.linear is not None: - return self.linear(x) - if self._fp16_weight_cpu is not None: - w = self._fp16_weight_cpu.to(device=x.device, dtype=in_dtype) - b = self._fp16_bias_cpu - b = b.to(device=x.device, dtype=in_dtype) if b is not None else None - return torch.nn.functional.linear(x, w, b) - raise RuntimeError("cast_inputs=False requires FP16 weights for fallback, but they were discarded.") - # import nvtx - # nvtx.push_range(f"cast_input") x_fp = x.to(torch.bfloat16) - # nvtx.pop_range() out_dtype = torch.bfloat16 else: x_fp = x out_dtype = in_dtype - - self._maybe_requantize_weight(x_fp.device) + return x_fp, in_dtype, out_dtype + + def _forward_quantized( + self, + qinput: torch.Tensor, + input_scale: torch.Tensor, + x_shape: torch.Size, + in_dtype: torch.dtype, + out_dtype: torch.dtype, + ) -> torch.Tensor: + """Run this Linear using an already quantized shared activation.""" + self._maybe_requantize_weight(qinput.device) if self.linear is not None: bias = self.linear.bias else: bias = self.bias if bias is not None: - if bias.device != x_fp.device: - bias = bias.to(device=x_fp.device, non_blocking=True) + if bias.device != qinput.device: + bias = bias.to(device=qinput.device, non_blocking=True) if bias.dtype != out_dtype: bias = bias.to(dtype=out_dtype) - x_shape = x_fp.shape - x_2d = x_fp.reshape(-1, x_shape[-1]).contiguous() - qinput = torch.empty_like(x_2d, dtype=torch.float8_e4m3fn) - input_scale = torch.empty((x_2d.shape[0], 1), dtype=torch.float32, device=x_fp.device) - self._tf_kernel.tf_per_token_quant_fp8(x_2d, qinput, input_scale) y = self._tf_kernel.fp8_scaled_mm( qinput, self._fp8_weight, @@ -293,6 +304,24 @@ def forward(self, x: torch.Tensor) -> torch.Tensor: return y +def fp8_linear_forward_many(linears: tuple[FP8Linear, ...], x: torch.Tensor) -> tuple[torch.Tensor, ...]: + """Reuse one dynamic activation quantization across compatible FP8 Linears.""" + if not linears: + return () + first = linears[0] + unsupported_no_cast = x.dtype not in (torch.float16, torch.bfloat16) and not first.options.cast_inputs + if not x.is_cuda or unsupported_no_cast or any(linear.options != first.options for linear in linears[1:]): + return tuple(linear(x) for linear in linears) + + x_fp, in_dtype, out_dtype = first._prepare_cuda_input(x) + x_shape = x_fp.shape + x_2d = x_fp.reshape(-1, x_shape[-1]).contiguous() + qinput = torch.empty_like(x_2d, dtype=torch.float8_e4m3fn) + input_scale = torch.empty((x_2d.shape[0], 1), dtype=torch.float32, device=x_fp.device) + first._tf_kernel.tf_per_token_quant_fp8(x_2d, qinput, input_scale) + return tuple(linear._forward_quantized(qinput, input_scale, x_shape, in_dtype, out_dtype) for linear in linears) + + def enable_fp8_gemm( model: nn.Module, *, diff --git a/tests/unit/models/test_wan_video_sol_attention.py b/tests/unit/models/test_wan_video_sol_attention.py index aec8bbc..08532f6 100644 --- a/tests/unit/models/test_wan_video_sol_attention.py +++ b/tests/unit/models/test_wan_video_sol_attention.py @@ -9,10 +9,10 @@ from telefuser.ops.fp8_attention import ( dequantize_fp8_per_block, dequantize_fp8_per_channel, - dequantize_fp8_per_token, quantize_fp8_per_block, quantize_fp8_qkv, ) +from telefuser.ops.fp8_gemm import FP8Linear def test_wan_model_enables_sol_attention_state() -> None: @@ -137,6 +137,23 @@ def test_wan_self_attention_casts_projection_input_once_under_autocast() -> None assert sol_x.dtype is torch.bfloat16 +def test_wan_self_attention_casts_shared_fp8_qkv_input_during_dense_warmup() -> None: + module = SelfAttention(dim=128, num_heads=1) + for name in ("q", "k", "v"): + projection = FP8Linear.__new__(FP8Linear) + torch.nn.Module.__init__(projection) + setattr(module, name, projection) + config = SparseAttentionConfig(sparse_impl="sol", dense_timesteps=1, dense_layers=0) + state = SparseAttentionState(config, mask_map=None) + x = torch.randn(1, 4, 128) + + with patch("telefuser.models.wan_video_dit.torch.is_autocast_enabled", return_value=True): + prepared = module._prepare_sol_projection_input(x, state) + + assert state.should_use_dense() + assert prepared.dtype is torch.bfloat16 + + def test_wan_self_attention_quantizes_qkv_for_fp8_sol() -> None: module = SelfAttention(dim=128, num_heads=1).to(torch.bfloat16) config = SparseAttentionConfig(sparse_impl="sol", dense_timesteps=0, sol_fp8=True) @@ -250,8 +267,8 @@ def tracked_sol_attn(q, k, v, **kwargs): assert captured["k"].dtype is torch.float8_e4m3fn assert captured["v"].dtype is torch.float8_e4m3fn assert captured["v"].stride(1) == 1 - assert captured["q_scale"].shape == (1, 256, 1) - assert captured["k_scale"].shape == (1, 256, 1) + assert captured["q_scale"].shape == (1, 4, 1) + assert captured["k_scale"].shape == (1, 4, 1) assert captured["v_scale"].shape == (1, 1, 128) @@ -268,12 +285,12 @@ def test_fused_fp8_qkv_quantization_on_h100() -> None: assert q_fp8.shape == q.shape assert k_fp8.shape == k.shape assert v_fp8.shape == v.shape - assert q_scale.shape == (1, 192, 2) - assert k_scale.shape == (1, 192, 2) + assert q_scale.shape == (1, 3, 2) + assert k_scale.shape == (1, 3, 2) assert v_scale.shape == (1, 2, 128) assert v_fp8.stride(1) == 1 torch.testing.assert_close( - dequantize_fp8_per_token(q_fp8, q_scale, torch.bfloat16), + dequantize_fp8_per_block(q_fp8, q_scale, torch.bfloat16), q, rtol=0.15, atol=0.05, @@ -305,8 +322,8 @@ def test_fp8_sol_handles_partial_tail_on_h100() -> None: v_scale=v_scale, ) reference = attention_impl.sol_attn( - dequantize_fp8_per_token(q_fp8, q_scale, torch.bfloat16).contiguous(), - dequantize_fp8_per_token(k_fp8, k_scale, torch.bfloat16).contiguous(), + dequantize_fp8_per_block(q_fp8, q_scale, torch.bfloat16).contiguous(), + dequantize_fp8_per_block(k_fp8, k_scale, torch.bfloat16).contiguous(), dequantize_fp8_per_channel(v_fp8, v_scale, torch.bfloat16).contiguous(), tau=-1000.0, ) diff --git a/tests/unit/ops/test_fp8_gemm.py b/tests/unit/ops/test_fp8_gemm.py index 256fb3f..d6324f1 100644 --- a/tests/unit/ops/test_fp8_gemm.py +++ b/tests/unit/ops/test_fp8_gemm.py @@ -51,3 +51,51 @@ def test_fp8_linear_tf_kernel_forward() -> None: assert actual.dtype == expected.dtype assert torch.isfinite(actual).all() torch.testing.assert_close(actual.float(), expected.float(), atol=0.1, rtol=0.1) + + +@pytest.mark.gpu +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA is required") +@pytest.mark.skipif(fp8_gemm.tf_kernel is None, reason="tf-kernel is required") +def test_fp8_linear_preserves_cuda_fallback_when_casting_is_disabled() -> None: + linear = nn.Linear(8, 4, device="cuda", dtype=torch.float32) + inputs = torch.randn(2, 8, device="cuda", dtype=torch.float32) + wrapped = fp8_gemm.FP8Linear( + linear, + options=fp8_gemm.FP8GemmOptions( + cast_inputs=False, + fp16_weight_storage="keep", + materialize_fp8_on_wrap=False, + ), + ) + + torch.testing.assert_close(wrapped(inputs), linear(inputs)) + + +@pytest.mark.gpu +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA is required") +@pytest.mark.skipif(fp8_gemm.tf_kernel is None, reason="tf-kernel is required") +def test_fp8_linear_forward_many_reuses_activation_quantization(monkeypatch: pytest.MonkeyPatch) -> None: + torch.manual_seed(0) + modules = tuple( + fp8_gemm.FP8Linear( + nn.Linear(64, 128, device="cuda", dtype=torch.bfloat16), + options=fp8_gemm.FP8GemmOptions(fp16_weight_storage="keep"), + ) + for _ in range(3) + ) + inputs = torch.randn(2, 3, 64, device="cuda", dtype=torch.bfloat16) + expected = tuple(module(inputs) for module in modules) + quantization_calls = 0 + quantize = modules[0]._tf_kernel.tf_per_token_quant_fp8 + + def tracked_quantize(*args, **kwargs): + nonlocal quantization_calls + quantization_calls += 1 + return quantize(*args, **kwargs) + + monkeypatch.setattr(modules[0]._tf_kernel, "tf_per_token_quant_fp8", tracked_quantize) + actual = fp8_gemm.fp8_linear_forward_many(modules, inputs) + + assert quantization_calls == 1 + for result, reference in zip(actual, expected): + torch.testing.assert_close(result.float(), reference.float(), atol=0.1, rtol=0.1) diff --git a/tests/unit/ops/test_sol_attention.py b/tests/unit/ops/test_sol_attention.py index ab663fd..f87414b 100644 --- a/tests/unit/ops/test_sol_attention.py +++ b/tests/unit/ops/test_sol_attention.py @@ -94,6 +94,60 @@ def test_sol_attention_dense_guard_does_not_call_kernel() -> None: kernel.assert_not_called() +@pytest.mark.gpu +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA is required") +def test_fp8_sol_uses_triton_for_unquantized_bf16_layers() -> None: + q = torch.randn(1, 64, 1, 128, device="cuda", dtype=torch.bfloat16) + kernel = MagicMock(side_effect=lambda q, _k, _v, **_kwargs: q) + config = AttentionConfig.sol_attention( + dense_timesteps=0, + dense_layers=0, + sol_fp8=True, + sol_fp8_layer_start=10, + sol_fp8_layer_end=20, + ) + assert config.sparse_config is not None + state = SparseAttentionState(config.sparse_config, mask_map=None) + + with ( + patch.object(attention_impl, "SOL_ATTN_AVAILABLE", True), + patch.object(attention_impl, "sol_attn", kernel), + ): + output = attention_impl.attention(q, q, q, attention_config=config, sparse_state=state) + + assert output.shape == q.shape + assert kernel.call_args.kwargs["force_triton"] is True + + +def test_fp8_dense_uses_sdpa_for_unquantized_bf16_layers() -> None: + q = torch.randn(1, 64, 1, 128, dtype=torch.bfloat16) + kernel = MagicMock() + config = AttentionConfig.sol_attention( + dense_timesteps=0, + dense_layers=0, + tau=-1000.0, + sol_fp8=True, + sol_fp8_layer_start=10, + sol_fp8_layer_end=20, + ) + assert config.sparse_config is not None + state = SparseAttentionState(config.sparse_config, mask_map=None) + + with ( + patch.object(attention_impl, "SOL_ATTN_AVAILABLE", True), + patch.object(attention_impl, "sol_attn", kernel), + ): + output = attention_impl.attention(q, q, q, attention_config=config, sparse_state=state) + + expected = torch.nn.functional.scaled_dot_product_attention( + q.transpose(1, 2), + q.transpose(1, 2), + q.transpose(1, 2), + ).transpose(1, 2) + torch.testing.assert_close(output, expected) + kernel.assert_not_called() + + @pytest.mark.gpu def test_sol_attention_public_ops_matches_sdpa_on_h100(monkeypatch: pytest.MonkeyPatch) -> None: if not torch.cuda.is_available() or torch.cuda.get_device_capability() != (9, 0): From 9879de2c45d688e1fbc0bf1cf25f16176093661b Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Mon, 17 Aug 2026 02:22:02 +0000 Subject: [PATCH 13/15] test: cover MiniMax H3 Turbo LoRA service parity --- tests/unit/service/test_example_service_parity.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/unit/service/test_example_service_parity.py b/tests/unit/service/test_example_service_parity.py index 8de6f59..587d008 100644 --- a/tests/unit/service/test_example_service_parity.py +++ b/tests/unit/service/test_example_service_parity.py @@ -30,6 +30,7 @@ "wan21_i2v_service": (Path("examples/wan_video/wan21_14b_image_to_video_480p_service.py"), "i2v", True), "minimax_h3_fl2va": (Path("examples/minimax_h3/minimax_h3_fl2va_h100.py"), "t2v", True), "minimax_h3_ref2va": (Path("examples/minimax_h3/minimax_h3_ref2va_h100.py"), "s2v", True), + "minimax_h3_turbo_lora": (Path("examples/minimax_h3/minimax_h3_turbo_lora_h100.py"), "i2v", True), "wan22_i2v_distill": (Path("examples/wan_video/wan22_14b_image_to_video_distill_h100.py"), "i2v", True), "lingbot_video_dense": (Path("examples/lingbot_video/lingbot_video_dense_1_3b.py"), "t2i", True), "lingbot_video_moe": (Path("examples/lingbot_video/lingbot_video_moe_30b.py"), "t2i", True), From b649f0ea65b5e12bd89e128ad478d3fce2624d44 Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Wed, 19 Aug 2026 03:01:47 +0000 Subject: [PATCH 14/15] feat: add MiniMax H3 FP8 Sol attention Add sparse runtime state and packed-sequence sink handling for MiniMax H3, with FP8 QKV dispatch on active Sol layers and dense FA4 guards for protected steps and layers. Reuse tf-kernel FP8 Linear GEMMs, expose unified example flags, and extend the benchmark matrix and documentation. Verification: ruff format and lint checks passed; the full unit suite passed with 1639 tests and 11 skips; the H100 BF16/FP8 Dense/Sol generation matrix completed. --- docs/en/quantization.md | 2 +- examples/minimax_h3/README.md | 43 +++++- examples/minimax_h3/common.py | 27 +++- examples/minimax_h3/minimax_h3_fl2va_h100.py | 33 ++++- examples/minimax_h3/minimax_h3_ref2va_h100.py | 2 +- telefuser/models/minimax_h3_dit.py | 130 ++++++++++++++++-- telefuser/ops/attention/attention_impl.py | 6 + telefuser/pipelines/minimax_h3/denoising.py | 7 +- tests/unit/models/test_minimax_h3_dit.py | 101 ++++++++++++++ .../pipelines/minimax_h3/test_examples.py | 54 ++++++++ .../pipelines/minimax_h3/test_parallelism.py | 16 ++- .../benchmark_minimax_h3_quantization.py | 27 +++- 12 files changed, 418 insertions(+), 30 deletions(-) diff --git a/docs/en/quantization.md b/docs/en/quantization.md index 0f56858..41a6f09 100644 --- a/docs/en/quantization.md +++ b/docs/en/quantization.md @@ -94,7 +94,7 @@ quant_config = QuantConfig( ) ``` -For MiniMax H3, use `quantization="tf-kernel-fp8"` with +For MiniMax H3, use `quantization="fp8"` (`"tf-kernel-fp8"` remains an alias) with `examples/minimax_h3/minimax_h3_fl2va_h100.py`. This backend is single-GPU only and keeps the FP8 weights resident after first-use conversion. It is distinct from the scaled-FP8 checkpoint path below: the latter expects weights and scales already serialized in the checkpoint. diff --git a/examples/minimax_h3/README.md b/examples/minimax_h3/README.md index 4269ad6..6084467 100644 --- a/examples/minimax_h3/README.md +++ b/examples/minimax_h3/README.md @@ -317,7 +317,7 @@ MiniMax H3 supports three single-GPU online quantization backends for the DiT tr | CLI value | Backend | Weight/activation path | |---|---|---| | torchao-fp8 | TorchAO | FP8 dynamic activation and FP8 weight when supported, otherwise TorchAO's FP8 weight-only path | -| tf-kernel-fp8 | TeleFuser tf-kernel | Per-token activation and per-output-channel weight FP8 (W8A8), BF16 output | +| fp8 (`tf-kernel-fp8` alias) | TeleFuser tf-kernel | Per-token activation and per-output-channel weight FP8 (W8A8), BF16 output | | bnb-nf4 | bitsandbytes | NF4 weight-only with BF16 compute | All three paths convert the 258 Linear layers in the main and token-refiner transformer blocks. The FP32 video/audio @@ -342,12 +342,13 @@ python examples/minimax_h3/minimax_h3_fl2va_h100.py \ --output outputs/minimax_h3_bnb_nf4.mp4 python examples/minimax_h3/minimax_h3_fl2va_h100.py \ --mode t2va \ - --quantization tf-kernel-fp8 \ + --quantization fp8 \ --duration 5 \ --output outputs/minimax_h3_tf_kernel_fp8.mp4 ~~~ -The FL2VA CLI accepts `--quantization` with `torchao-fp8`, `tf-kernel-fp8`, or `bnb-nf4`; omit it for BF16. The Python +The FL2VA CLI accepts `--quantization` with `fp8`, `torchao-fp8`, or `bnb-nf4`; `tf-kernel-fp8` remains a compatibility +alias and omitting the option keeps BF16. The Python loader accepts the same names: ~~~python @@ -356,7 +357,7 @@ from examples.minimax_h3.common import load_minimax_h3_pipeline pipeline = load_minimax_h3_pipeline( "/path/to/MiniMaxAI_MiniMax-H3", partition="FL2VA", - quantization="tf-kernel-fp8", + quantization="fp8", ) ~~~ @@ -364,12 +365,42 @@ Online quantization currently requires ulysses_degree=1, tp_degree=1, and FSDP d would invalidate those wrappers' BF16 parameter-sharding contract, so unsupported combinations fail before checkpoint loading. +### FP8 Sol-Attn + +The same FL2VA example exposes dense/Sol and BF16/FP8 as independent switches. `--quantization fp8` applies +tf-kernel W8A8 Linear GEMMs to the transformer blocks. `--attn-impl SOL_ATTN` enables the MiniMax-H3 Sol policy: +the first 10 denoising steps and first 2 DiT layers remain dense, the full condition prefix is an exact KV sink, and +prefix queries are recomputed with BF16 dense attention. Adding `--sol-fp8` quantizes post-RoPE Q/K/V in active sparse +layers and dispatches the SM90 CuTe FP8 Sol mainloop. + +~~~bash +# BF16 dense +python -m examples.minimax_h3.minimax_h3_fl2va_h100 --mode t2va --output outputs/h3_bf16.mp4 + +# BF16 Sol +python -m examples.minimax_h3.minimax_h3_fl2va_h100 \ + --mode t2va --attn-impl SOL_ATTN --output outputs/h3_bf16_sol.mp4 + +# FP8 Linear + dense attention +python -m examples.minimax_h3.minimax_h3_fl2va_h100 \ + --mode t2va --quantization fp8 --output outputs/h3_fp8.mp4 + +# FP8 Linear + FP8 Sol attention +python -m examples.minimax_h3.minimax_h3_fl2va_h100 \ + --mode t2va --quantization fp8 --attn-impl SOL_ATTN --sol-fp8 \ + --output outputs/h3_fp8_sol.mp4 +~~~ + +Use `--sol-dense-steps`, `--sol-dense-layers`, `--sol-tau`, `--sol-threshold-type`, +`--sol-fp8-layer-start`, and `--sol-fp8-layer-end` to override the policy for controlled ablations. The defaults +match the released H100 MiniMax-H3 Sol profile. + For matched BF16/TorchAO-FP8/tf-kernel-FP8/NF4 profiling, use the validation benchmark. It writes the synchronized MP4 plus a JSON report containing load time, end-to-end generation time, stage timings, and denoising allocator peaks: ~~~bash -python tools/validation/benchmark_minimax_h3_quantization.py \ - --backend tf-kernel-fp8 \ +python -m tools.validation.benchmark_minimax_h3_quantization \ + --backend fp8-sol \ --duration 5 \ --steps 50 \ --output outputs/minimax_h3_tf_kernel_fp8_50step.mp4 diff --git a/examples/minimax_h3/common.py b/examples/minimax_h3/common.py index 253f4c9..0883a67 100644 --- a/examples/minimax_h3/common.py +++ b/examples/minimax_h3/common.py @@ -138,6 +138,7 @@ def minimax_h3_quant_config(quantization: str | QuantType | None) -> QuantConfig if isinstance(quantization, str): normalized = quantization.strip().lower().replace("_", "-") names = { + "fp8": QuantType.FP8, "torchao-fp8": QuantType.TORCHAO_FP8, "bnb-nf4": QuantType.BNB_NF4, "tf-kernel-fp8": QuantType.FP8, @@ -145,7 +146,7 @@ def minimax_h3_quant_config(quantization: str | QuantType | None) -> QuantConfig try: quant_type = names[normalized] except KeyError as exc: - raise ValueError("quantization must be 'torchao-fp8', 'tf-kernel-fp8', 'bnb-nf4', or None") from exc + raise ValueError("quantization must be 'fp8', 'torchao-fp8', 'tf-kernel-fp8', 'bnb-nf4', or None") from exc elif isinstance(quantization, QuantType): quant_type = quantization else: @@ -172,6 +173,13 @@ def load_minimax_h3_pipeline( text_encoder_tp_degree: int | None = None, enable_fsdp: bool | None = None, attn_impl: AttnImplType | str = AttnImplType.FLASH_ATTN_4, + sol_fp8: bool = False, + sol_dense_steps: int = 10, + sol_dense_layers: int = 2, + sol_tau: float = 1.0, + sol_threshold_type: str = "exact", + sol_fp8_layer_start: int = 0, + sol_fp8_layer_end: int | None = None, feature_cache_config: FeatureCacheConfig | None = None, adaln_cache_path: str | Path | None = None, online_adaln_cache: bool = False, @@ -209,6 +217,8 @@ def load_minimax_h3_pipeline( attn_impl = AttnImplType[attn_impl] except KeyError as exc: raise ValueError(f"unsupported attention implementation: {attn_impl}") from exc + if sol_fp8 and attn_impl != AttnImplType.SOL_ATTN: + raise ValueError("sol_fp8 requires attn_impl=SOL_ATTN") component_root = Path(model_root) / partition if not component_root.is_dir(): raise FileNotFoundError(f"MiniMax H3 partition not found: {component_root}") @@ -248,12 +258,25 @@ def load_minimax_h3_pipeline( offload_config=resident_offload, parallel_config=text_parallel, ) + attention_config = ( + AttentionConfig.sol_attention( + dense_timesteps=sol_dense_steps, + dense_layers=sol_dense_layers, + tau=sol_tau, + threshold_type=sol_threshold_type, + sol_fp8=sol_fp8, + sol_fp8_layer_start=sol_fp8_layer_start, + sol_fp8_layer_end=sol_fp8_layer_end, + ) + if attn_impl == AttnImplType.SOL_ATTN + else AttentionConfig.dense_attention(attn_impl) + ) dit_runtime = ModelRuntimeConfig( device_type=runtime_device.type, device_id=runtime_device.index or 0, torch_dtype=torch.bfloat16, offload_config=dit_offload, - attention_config=AttentionConfig.dense_attention(attn_impl), + attention_config=attention_config, feature_cache_config=feature_cache_config or FeatureCacheConfig(), quant_config=quant_config, lora_configs=[LoraConfig(path=str(lora_path), strength=lora_strength)] if lora_path else [], diff --git a/examples/minimax_h3/minimax_h3_fl2va_h100.py b/examples/minimax_h3/minimax_h3_fl2va_h100.py index 57af404..b833507 100644 --- a/examples/minimax_h3/minimax_h3_fl2va_h100.py +++ b/examples/minimax_h3/minimax_h3_fl2va_h100.py @@ -35,6 +35,7 @@ "enable_fsdp": None, "online_adaln_cache": True, "attn_impl": AttnImplType.FLASH_ATTN_4, + "sol_fp8": False, "feature_cache_model_type": "MiniMax-H3-Base", "feature_cache_n_derivatives": 1, "feature_cache_taylor_threshold": 2, @@ -85,6 +86,13 @@ def get_pipeline( enable_fsdp: bool | None = PPL_CONFIG["enable_fsdp"], online_adaln_cache: bool = PPL_CONFIG["online_adaln_cache"], attn_impl: AttnImplType | str = PPL_CONFIG["attn_impl"], + sol_fp8: bool = PPL_CONFIG["sol_fp8"], + sol_dense_steps: int = 10, + sol_dense_layers: int = 2, + sol_tau: float = 1.0, + sol_threshold_type: str = "exact", + sol_fp8_layer_start: int = 0, + sol_fp8_layer_end: int | None = None, enable_feature_cache: bool = False, feature_cache_model_type: str = PPL_CONFIG["feature_cache_model_type"], feature_cache_n_derivatives: int = PPL_CONFIG["feature_cache_n_derivatives"], @@ -104,6 +112,13 @@ def get_pipeline( enable_fsdp=enable_fsdp, online_adaln_cache=online_adaln_cache, attn_impl=attn_impl, + sol_fp8=sol_fp8, + sol_dense_steps=sol_dense_steps, + sol_dense_layers=sol_dense_layers, + sol_tau=sol_tau, + sol_threshold_type=sol_threshold_type, + sol_fp8_layer_start=sol_fp8_layer_start, + sol_fp8_layer_end=sol_fp8_layer_end, feature_cache_config=FeatureCacheConfig( enabled=enable_feature_cache, model_type=feature_cache_model_type, @@ -270,16 +285,23 @@ def _main(default_quantization: str | None = PPL_CONFIG["quantization"]) -> None parser.add_argument("--device", default=PPL_CONFIG["device"]) parser.add_argument( "--quantization", - choices=("torchao-fp8", "tf-kernel-fp8", "bnb-nf4"), + choices=("fp8", "torchao-fp8", "tf-kernel-fp8", "bnb-nf4"), default=default_quantization, help="Online DiT Linear quantization backend (single GPU only).", ) parser.add_argument("--gpu-num", "--ulysses-degree", dest="gpu_num", type=int, choices=(1, 2, 4), default=1) parser.add_argument( "--attn-impl", - choices=("FLASH_ATTN_4", "SAGE_ATTN_2_8_8_SM90"), + choices=("FLASH_ATTN_4", "SAGE_ATTN_2_8_8_SM90", "SOL_ATTN"), default=PPL_CONFIG["attn_impl"].name, ) + parser.add_argument("--sol-fp8", action="store_true", help="Use FP8 Q/K/V in active Sol-Attn layers.") + parser.add_argument("--sol-dense-steps", type=int, default=10) + parser.add_argument("--sol-dense-layers", type=int, default=2) + parser.add_argument("--sol-tau", type=float, default=1.0) + parser.add_argument("--sol-threshold-type", choices=("exact", "diag"), default="exact") + parser.add_argument("--sol-fp8-layer-start", type=int, default=0) + parser.add_argument("--sol-fp8-layer-end", type=int) parser.add_argument("--enable-feature-cache", action="store_true") parser.add_argument("--feature-cache-model-type", default=PPL_CONFIG["feature_cache_model_type"]) parser.add_argument( @@ -324,6 +346,13 @@ def _main(default_quantization: str | None = PPL_CONFIG["quantization"]) -> None num_inference_steps=args.steps, enable_fsdp=args.enable_fsdp, attn_impl=args.attn_impl, + sol_fp8=args.sol_fp8, + sol_dense_steps=args.sol_dense_steps, + sol_dense_layers=args.sol_dense_layers, + sol_tau=args.sol_tau, + sol_threshold_type=args.sol_threshold_type, + sol_fp8_layer_start=args.sol_fp8_layer_start, + sol_fp8_layer_end=args.sol_fp8_layer_end, enable_feature_cache=args.enable_feature_cache, feature_cache_model_type=args.feature_cache_model_type, feature_cache_n_derivatives=args.feature_cache_n_derivatives, diff --git a/examples/minimax_h3/minimax_h3_ref2va_h100.py b/examples/minimax_h3/minimax_h3_ref2va_h100.py index a7de40f..ef6971e 100644 --- a/examples/minimax_h3/minimax_h3_ref2va_h100.py +++ b/examples/minimax_h3/minimax_h3_ref2va_h100.py @@ -263,7 +263,7 @@ def main() -> None: parser.add_argument("--flow-shift", type=float, default=PPL_CONFIG["flow_shift"]) parser.add_argument("--audio-flow-shift", type=float, default=PPL_CONFIG["audio_flow_shift"]) parser.add_argument("--device", default=PPL_CONFIG["device"]) - parser.add_argument("--quantization", choices=("torchao-fp8", "tf-kernel-fp8", "bnb-nf4")) + parser.add_argument("--quantization", choices=("fp8", "torchao-fp8", "tf-kernel-fp8", "bnb-nf4")) parser.add_argument("--gpu-num", "--ulysses-degree", dest="gpu_num", type=int, choices=(1, 2, 4), default=1) fsdp_group = parser.add_mutually_exclusive_group() fsdp_group.add_argument("--enable-fsdp", dest="enable_fsdp", action="store_true") diff --git a/telefuser/models/minimax_h3_dit.py b/telefuser/models/minimax_h3_dit.py index 98522b9..fad880d 100644 --- a/telefuser/models/minimax_h3_dit.py +++ b/telefuser/models/minimax_h3_dit.py @@ -14,6 +14,7 @@ import torch import torch.distributed as dist import torch.nn as nn +import torch.nn.functional as F from telefuser.core.base_model import BaseModel from telefuser.core.config import AttentionConfig, AttnImplType, QuantConfig, QuantKernelBackend, QuantType @@ -29,7 +30,8 @@ from telefuser.distributed.ulysses_comm import ulysses_gather_heads_destination_major, ulysses_scatter_heads from telefuser.feature_cache import AdaTaylorCacheCalibrator, NoOpCache from telefuser.ops import RMSNorm, apply_qk_norm_rope_neox, indexed_gate, indexed_scale_shift, silu_and_mul_reuse_input -from telefuser.ops.attention import attention +from telefuser.ops.attention import SparseAttentionState, attention +from telefuser.ops.fp8_attention import quantize_fp8_per_block, quantize_fp8_qkv from telefuser.ops.rotary import apply_rotary_emb_neox from telefuser.utils.logging import logger @@ -446,7 +448,7 @@ def enable_tp(self, group: dist.ProcessGroup, *, rank: int, world_size: int) -> self.tp_group = group @staticmethod - def _sage_live_tokens(sequence_lengths: list[int], total_tokens: int) -> int: + def _live_tokens(sequence_lengths: list[int], total_tokens: int) -> int: if len(sequence_lengths) == 1 and sequence_lengths[0] == total_tokens: return total_tokens if ( @@ -457,7 +459,42 @@ def _sage_live_tokens(sequence_lengths: list[int], total_tokens: int) -> int: and total_tokens % 64 == 0 ): return sequence_lengths[0] - raise ValueError("MiniMax H3 SageAttention requires one live sequence with optional trailing alignment padding") + raise ValueError("MiniMax H3 optimized attention requires one live sequence with optional trailing padding") + + @staticmethod + def _is_sol_active(sparse_state: SparseAttentionState | None) -> bool: + return sparse_state is not None and not sparse_state.should_use_dense() + + @classmethod + def _prepare_sol_qkv( + cls, + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + sparse_state: SparseAttentionState, + ) -> tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None, + ]: + config = sparse_state.config + layer_end = config.sol_fp8_layer_end + fp8_layer_active = ( + cls._is_sol_active(sparse_state) + and config.sol_fp8 + and sparse_state.layer_idx >= config.sol_fp8_layer_start + and (layer_end is None or sparse_state.layer_idx < layer_end) + ) + if not fp8_layer_active: + return query, key, value, None + if query.is_cuda and torch.cuda.get_device_capability(query.device) == (9, 0): + query, key, value, q_scale, k_scale, v_scale = quantize_fp8_qkv(query, key, value) + else: + query, q_scale = quantize_fp8_per_block(query) + key, k_scale = quantize_fp8_per_block(key) + value, v_scale = quantize_fp8_per_block(value) + return query, key, value, (q_scale, k_scale, v_scale) def forward( self, @@ -467,6 +504,8 @@ def forward( rope_cos_sin_cache: torch.Tensor | None, attention_config: AttentionConfig | None, cu_seqlens: torch.Tensor | None = None, + sparse_state: SparseAttentionState | None = None, + prefix_tokens: int = 0, ) -> torch.Tensor: sequence, _ = hidden.shape qkv = self.qkv_proj(hidden).reshape(sequence, 3, self.num_heads, self.head_dim) @@ -500,16 +539,53 @@ def forward( query = query_wait() key = key_wait() value = value_wait() - if attention_config is not None and attention_config.attn_impl == AttnImplType.SAGE_ATTN_2_8_8_SM90: + optimized_impls = {AttnImplType.SAGE_ATTN_2_8_8_SM90, AttnImplType.SOL_ATTN} + if attention_config is not None and attention_config.attn_impl in optimized_impls: total_tokens = query.shape[1] - live_tokens = self._sage_live_tokens(sequence_lengths, total_tokens) + live_tokens = self._live_tokens(sequence_lengths, total_tokens) + live_query = query[:, :live_tokens].contiguous() + live_key = key[:, :live_tokens].contiguous() + live_value = value[:, :live_tokens].contiguous() + scales = None + runtime_attention_config = attention_config + runtime_sparse_state = sparse_state + if attention_config.attn_impl == AttnImplType.SOL_ATTN: + if sparse_state is None: + raise RuntimeError("MiniMax H3 Sol-Attn requires sparse runtime state") + if not 0 <= prefix_tokens <= live_tokens: + raise ValueError("MiniMax H3 Sol-Attn prefix must be within the live packed sequence") + sol_query, sol_key, sol_value, scales = self._prepare_sol_qkv( + live_query, + live_key, + live_value, + sparse_state, + ) + if sparse_state.should_use_dense(): + runtime_attention_config = AttentionConfig.dense_attention(AttnImplType.FLASH_ATTN_4) + runtime_sparse_state = None + else: + sol_query, sol_key, sol_value = live_query, live_key, live_value live_output = attention( - query[:, :live_tokens].contiguous(), - key[:, :live_tokens].contiguous(), - value[:, :live_tokens].contiguous(), - attention_config=attention_config, + sol_query, + sol_key, + sol_value, + attention_config=runtime_attention_config, + sparse_state=runtime_sparse_state, scale=self.head_dim**-0.5, + q_scale=None if scales is None else scales[0], + k_scale=None if scales is None else scales[1], + v_scale=None if scales is None else scales[2], + sink_start=0, + sink_tokens=prefix_tokens, ) + if self._is_sol_active(sparse_state) and prefix_tokens: + dense_prefix = F.scaled_dot_product_attention( + live_query[:, :prefix_tokens].transpose(1, 2), + live_key.transpose(1, 2), + live_value.transpose(1, 2), + scale=self.head_dim**-0.5, + ).transpose(1, 2) + live_output = torch.cat((dense_prefix, live_output[:, prefix_tokens:]), dim=1) if live_tokens == total_tokens: output = live_output else: @@ -683,6 +759,8 @@ def forward( rope_cos_sin_cache: torch.Tensor, attention_config: AttentionConfig | None, cu_seqlens: torch.Tensor | None = None, + sparse_state: SparseAttentionState | None = None, + prefix_tokens: int = 0, adaln_params: tuple[torch.Tensor, ...] | None = None, ) -> torch.Tensor: if adaln_params is None: @@ -696,6 +774,8 @@ def forward( rope_cos_sin_cache=rope_cos_sin_cache, attention_config=attention_config, cu_seqlens=cu_seqlens, + sparse_state=sparse_state, + prefix_tokens=prefix_tokens, ) hidden = indexed_gate(residual, gate_msa, value, combined_indices) residual = hidden @@ -759,6 +839,25 @@ def __init__(self, config: MiniMaxH3DiTConfig | None = None) -> None: self._online_adaln_rows: dict[str, tuple[float, tuple[torch.Tensor, ...], torch.Tensor]] = {} self._online_adaln_batches: list[tuple[torch.Tensor, torch.Tensor, torch.Tensor]] = [] self._online_adaln_copy_device: torch.device | None = None + self.sparse_attention_state: SparseAttentionState | None = None + + def set_attention_config(self, attention_config: AttentionConfig) -> None: + super().set_attention_config(attention_config) + if attention_config.attn_impl == AttnImplType.SOL_ATTN: + if attention_config.sparse_config is None: + raise ValueError("MiniMax H3 Sol-Attn requires sparse attention configuration") + self.sparse_attention_state = SparseAttentionState( + config=attention_config.sparse_config, + mask_map=None, + model_type="minimax_h3", + ) + else: + self.sparse_attention_state = None + + def _token_refiner_attention_config(self) -> AttentionConfig: + if self.attention_config.is_sparse(): + return AttentionConfig.dense_attention(AttnImplType.FLASH_ATTN_4) + return self.attention_config def adaln_fingerprint(self) -> str: if self.time_embedder is None: @@ -977,7 +1076,7 @@ def _static_inputs( prompt = kwargs["prompt_embeds"].to(device=device, dtype=torch.bfloat16) prompt = self.condition_proj(prompt[: text_positions.numel()]) - prompt = self.token_refiner(prompt, attention_config=self.attention_config) + prompt = self.token_refiner(prompt, attention_config=self._token_refiner_attention_config()) rope_position_ids = kwargs["img_position_ids"].to(device) rope_position_ids = rope_position_ids[:, rope_row_start:rope_row_stop] rope_frequencies = self.rope(rope_position_ids) @@ -1026,6 +1125,13 @@ def forward(self, **kwargs: Any) -> tuple[torch.Tensor, torch.Tensor]: output_positions = self._position_ids( kwargs["img_pos_for_infer_output_info"], "img_pos_for_infer_output_info" ).to(device) + sparse_state = self.sparse_attention_state + prefix_tokens = 0 + if self.attention_config.attn_impl == AttnImplType.SOL_ATTN: + if sparse_state is None: + raise RuntimeError("MiniMax H3 Sol-Attn was not initialized through set_attention_config") + sparse_state.update(numeral_timestep=int(kwargs.get("sparse_step_index", 0))) + prefix_tokens = int(kwargs.get("sol_prefix_tokens", output_positions.min().item())) local_embedding_layout = kwargs.get("local_embedding_layout") use_local_embedding = self.usp_flag and local_embedding_layout is not None @@ -1146,6 +1252,8 @@ def layout_tensor(name: str) -> torch.Tensor: block.adaln_proj.split_output(output) for block, output in zip(self.blocks, gathered_adaln) ) for index, block in enumerate(self.blocks): + if sparse_state is not None: + sparse_state.update(layer_idx=index) hidden = block( hidden, adaln_input=adaln_input, @@ -1154,6 +1262,8 @@ def layout_tensor(name: str) -> torch.Tensor: rope_cos_sin_cache=rope_cos_sin_cache, attention_config=self.attention_config, cu_seqlens=cu_seqlens, + sparse_state=sparse_state, + prefix_tokens=prefix_tokens, adaln_params=None if block_adaln_params is None else block_adaln_params[index], ) if isinstance(feature_cache, AdaTaylorCacheCalibrator): diff --git a/telefuser/ops/attention/attention_impl.py b/telefuser/ops/attention/attention_impl.py index 22162e7..f03427e 100755 --- a/telefuser/ops/attention/attention_impl.py +++ b/telefuser/ops/attention/attention_impl.py @@ -193,6 +193,8 @@ def attention( q_scale: Tensor | None = None, k_scale: Tensor | None = None, v_scale: Tensor | None = None, + sink_start: int | None = None, + sink_tokens: int = 0, **kwargs: Any, ) -> Tensor | tuple[Tensor, Tensor]: """Unified attention function. @@ -212,6 +214,8 @@ def attention( return_lse: Return log-sum-exp values. sequence_lengths: Length of each sequence packed along the sequence axis. cu_seqlens: Optional precomputed cumulative sequence lengths for varlen kernels. + sink_start: Start of the exact KV sink used by Sol-Attn. + sink_tokens: Number of exact KV sink tokens used by Sol-Attn. **kwargs: Implementation-specific arguments. Returns: @@ -443,6 +447,8 @@ def attention( q_scale=q_scale, k_scale=k_scale, v_scale=v_scale, + sink_start=sink_start, + sink_tokens=sink_tokens, # A partial FP8 layer range otherwise compiles both BF16 # and FP8 CuTe specializations on the first sparse step. # Triton is a better cold-start tradeoff for the remaining diff --git a/telefuser/pipelines/minimax_h3/denoising.py b/telefuser/pipelines/minimax_h3/denoising.py index 89d02ee..e2fa27d 100644 --- a/telefuser/pipelines/minimax_h3/denoising.py +++ b/telefuser/pipelines/minimax_h3/denoising.py @@ -126,6 +126,9 @@ def __init__( self.transformer = module_manager.fetch_module("minimax_h3_transformer") if self.transformer is None: raise ValueError("ModuleManager must contain 'minimax_h3_transformer'") + set_attention_config = getattr(self.transformer, "set_attention_config", None) + if callable(set_attention_config): + set_attention_config(model_runtime_config.attention_config) if model_runtime_config.lora_configs: MiniMaxH3LoraAdapter.apply(self.transformer, model_runtime_config.lora_configs) step_update = "training_euler" if model_runtime_config.lora_configs else "reference_blend" @@ -159,7 +162,6 @@ def parallel_models(self) -> None: raise NotImplementedError(f"MiniMax H3 does not support these parallel degrees yet: {invalid}") device_mesh = create_device_mesh_from_config(parallel_config) self.transformer.device_mesh = device_mesh - self.transformer.set_attention_config(self.model_runtime_config.attention_config) if parallel_config.tp_degree > 1: if parallel_config.enable_fsdp: raise ValueError("MiniMax H3 DiT tensor parallelism cannot be combined with FSDP") @@ -374,6 +376,7 @@ def denoise( text_pos_cpu = packed["text_pos"] text_pos = text_pos_cpu.to(device) target_img_pos = img_pos[video_update] + sol_prefix_tokens = int(img_pos_cpu[video_update_cpu][0]) target_video_row_start = int((~video_update_cpu).sum()) target_audio_row_start = int((~audio_update_cpu).sum()) condition_img_pos = img_pos_cpu[~video_update_cpu] @@ -496,6 +499,8 @@ def denoise( block_combined_indices=block_combined_indices, local_embedding_layout=local_embedding_layout, static_cache_key=static_cache_key, + sparse_step_index=step, + sol_prefix_tokens=sol_prefix_tokens, skip_mask_out_condition=True, ) audio_target_velocity = audio_velocity[audio_target_slice] diff --git a/tests/unit/models/test_minimax_h3_dit.py b/tests/unit/models/test_minimax_h3_dit.py index 35147de..8bc228a 100644 --- a/tests/unit/models/test_minimax_h3_dit.py +++ b/tests/unit/models/test_minimax_h3_dit.py @@ -14,6 +14,7 @@ MiniMaxH3DiTConfig, _reorder_grouped_qkv_to_qkv, ) +from telefuser.ops.attention import SparseAttentionState from telefuser.ops.rotary import apply_qk_norm_rope_neox, apply_rotary_emb_neox @@ -159,6 +160,106 @@ def sage_output(query: torch.Tensor, *_: torch.Tensor, **__: object) -> torch.Te assert torch.count_nonzero(output[61:]) == 0 +def test_sol_attention_preserves_prefix_sink_and_dense_prefix_queries() -> None: + module = MiniMaxH3Attention(_small_config()).eval() + hidden = torch.randn(64, 32, dtype=torch.bfloat16) + config = AttentionConfig.sol_attention(dense_timesteps=0, dense_layers=0, threshold_type="exact") + state = SparseAttentionState(config.sparse_config, mask_map=None, model_type="minimax_h3") + + with ( + patch("telefuser.models.minimax_h3_dit.attention", side_effect=lambda query, *_args, **_kwargs: query) as sol, + patch( + "telefuser.models.minimax_h3_dit.F.scaled_dot_product_attention", + side_effect=lambda query, *_args, **_kwargs: query, + ) as dense_prefix, + ): + output = module( + hidden, + sequence_lengths=[61, 3], + rope_cos_sin_cache=None, + attention_config=config, + sparse_state=state, + prefix_tokens=13, + ) + + assert sol.call_args.args[0].shape == (1, 61, 4, 8) + assert sol.call_args.kwargs["sparse_state"] is state + assert sol.call_args.kwargs["sink_start"] == 0 + assert sol.call_args.kwargs["sink_tokens"] == 13 + assert dense_prefix.call_args.args[0].shape == (1, 4, 13, 8) + assert dense_prefix.call_args.args[1].shape == (1, 4, 61, 8) + assert torch.count_nonzero(output[61:]) == 0 + + +def test_sol_dense_guard_uses_packed_flash_attention_4() -> None: + module = MiniMaxH3Attention(_small_config()).eval() + hidden = torch.randn(64, 32, dtype=torch.bfloat16) + config = AttentionConfig.sol_attention(dense_timesteps=10, dense_layers=2, threshold_type="exact") + state = SparseAttentionState(config.sparse_config, mask_map=None, model_type="minimax_h3") + + with patch("telefuser.models.minimax_h3_dit.attention", side_effect=lambda query, *_args, **_kwargs: query) as call: + module( + hidden, + sequence_lengths=[61, 3], + rope_cos_sin_cache=None, + attention_config=config, + sparse_state=state, + prefix_tokens=13, + ) + + runtime_config = call.call_args.kwargs["attention_config"] + assert runtime_config.attn_impl == AttnImplType.FLASH_ATTN_4 + assert call.call_args.kwargs["sparse_state"] is None + + +def test_sol_fp8_passes_quantized_qkv_scales_to_attention() -> None: + module = MiniMaxH3Attention(_small_config()).eval() + hidden = torch.randn(64, 32, dtype=torch.bfloat16) + config = AttentionConfig.sol_attention( + dense_timesteps=0, + dense_layers=0, + threshold_type="exact", + sol_fp8=True, + ) + state = SparseAttentionState(config.sparse_config, mask_map=None, model_type="minimax_h3") + scales = [torch.ones(1, 1, 4) * value for value in (1, 2, 3)] + + def quantize(value: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + return value.to(torch.float8_e4m3fn), scales.pop(0) + + with ( + patch("telefuser.models.minimax_h3_dit.quantize_fp8_per_block", side_effect=quantize), + patch( + "telefuser.models.minimax_h3_dit.attention", + side_effect=lambda query, *_args, **_kwargs: query.float(), + ) as attention_call, + ): + module( + hidden, + sequence_lengths=[61, 3], + rope_cos_sin_cache=None, + attention_config=config, + sparse_state=state, + ) + + assert attention_call.call_args.args[0].dtype == torch.float8_e4m3fn + assert attention_call.call_args.kwargs["q_scale"].flatten()[0].item() == 1 + assert attention_call.call_args.kwargs["k_scale"].flatten()[0].item() == 2 + assert attention_call.call_args.kwargs["v_scale"].flatten()[0].item() == 3 + + +def test_minimax_h3_initializes_sol_runtime_state() -> None: + model = MiniMaxH3DiT(_small_config()) + config = AttentionConfig.sol_attention(dense_timesteps=10, dense_layers=2, threshold_type="exact") + + model.set_attention_config(config) + + assert model.sparse_attention_state is not None + assert model.sparse_attention_state.config is config.sparse_config + assert model.sparse_attention_state.model_type == "minimax_h3" + assert model._token_refiner_attention_config().attn_impl == AttnImplType.FLASH_ATTN_4 + + def test_ulysses_overlaps_strided_value_scatter_with_qk_preprocessing() -> None: module = MiniMaxH3Attention(_small_config()).eval() module.ulysses_group = MagicMock() diff --git a/tests/unit/pipelines/minimax_h3/test_examples.py b/tests/unit/pipelines/minimax_h3/test_examples.py index 4655763..336113d 100644 --- a/tests/unit/pipelines/minimax_h3/test_examples.py +++ b/tests/unit/pipelines/minimax_h3/test_examples.py @@ -134,6 +134,13 @@ def fake_loader(model_root: str, **kwargs: object) -> object: "enable_fsdp": True, "online_adaln_cache": True, "attn_impl": AttnImplType.FLASH_ATTN_4, + "sol_fp8": False, + "sol_dense_steps": 10, + "sol_dense_layers": 2, + "sol_tau": 1.0, + "sol_threshold_type": "exact", + "sol_fp8_layer_start": 0, + "sol_fp8_layer_end": None, "feature_cache_config": FeatureCacheConfig( enabled=True, model_type="MiniMax-H3-Base", @@ -166,6 +173,7 @@ def test_cache_calibration_applies_validated_h3_profile(tmp_path: Path) -> None: [ ("torchao-fp8", QuantType.TORCHAO_FP8, QuantKernelBackend.TORCHAO), ("torchao_fp8", QuantType.TORCHAO_FP8, QuantKernelBackend.TORCHAO), + ("fp8", QuantType.FP8, QuantKernelBackend.TF_KERNEL), ("tf-kernel-fp8", QuantType.FP8, QuantKernelBackend.TF_KERNEL), ("bnb-nf4", QuantType.BNB_NF4, QuantKernelBackend.BITSANDBYTES), ], @@ -221,6 +229,52 @@ def fake_get_pipeline(*args: object, **kwargs: object) -> object: assert fl2va_example.PIPELINE_MANIFEST["pipeline_name"] == fl2va_example.PPL_CONFIG["name"] +def test_standard_example_forwards_fp8_sol_configuration(monkeypatch: pytest.MonkeyPatch) -> None: + calls = [] + sentinel = object() + + def fake_get_pipeline(*args: object, **kwargs: object) -> object: + calls.append((args, kwargs)) + return sentinel + + monkeypatch.setattr(fl2va_example, "load_minimax_h3_pipeline", fake_get_pipeline) + result = fl2va_example.get_pipeline( + 1, + "/models/h3", + attn_impl="SOL_ATTN", + sol_fp8=True, + sol_dense_steps=10, + sol_dense_layers=2, + sol_tau=0.9, + sol_threshold_type="diag", + sol_fp8_layer_start=2, + sol_fp8_layer_end=40, + quantization="tf-kernel-fp8", + ) + + assert result is sentinel + options = calls[0][1] + assert options["attn_impl"] == "SOL_ATTN" + assert options["sol_fp8"] is True + assert options["sol_dense_steps"] == 10 + assert options["sol_dense_layers"] == 2 + assert options["sol_tau"] == 0.9 + assert options["sol_threshold_type"] == "diag" + assert options["sol_fp8_layer_start"] == 2 + assert options["sol_fp8_layer_end"] == 40 + assert options["quantization"] == "tf-kernel-fp8" + + +def test_sol_fp8_rejects_dense_attention(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="sol_fp8 requires"): + load_minimax_h3_pipeline( + tmp_path, + partition="FL2VA", + attn_impl=AttnImplType.FLASH_ATTN_4, + sol_fp8=True, + ) + + def test_fl2va_run_maps_standard_service_tasks_to_model_conditions() -> None: calls = [] marker = object() diff --git a/tests/unit/pipelines/minimax_h3/test_parallelism.py b/tests/unit/pipelines/minimax_h3/test_parallelism.py index 379366a..da9537e 100644 --- a/tests/unit/pipelines/minimax_h3/test_parallelism.py +++ b/tests/unit/pipelines/minimax_h3/test_parallelism.py @@ -4,6 +4,8 @@ import torch from telefuser.core.config import ( + AttentionConfig, + AttnImplType, ModelRuntimeConfig, OffloadConfig, ParallelConfig, @@ -19,7 +21,10 @@ from telefuser.pipelines.minimax_h3.vae import MiniMaxH3VideoVAEStage -def _stage(parallel_config: ParallelConfig) -> tuple[MiniMaxH3DenoisingStage, MagicMock]: +def _stage( + parallel_config: ParallelConfig, + attention_config: AttentionConfig | None = None, +) -> tuple[MiniMaxH3DenoisingStage, MagicMock]: transformer = MagicMock() transformer.parameters.return_value = [torch.nn.Parameter(torch.zeros(1, dtype=torch.float32))] transformer.get_fsdp_module_names.return_value = ["blocks"] @@ -31,10 +36,19 @@ def _stage(parallel_config: ParallelConfig) -> tuple[MiniMaxH3DenoisingStage, Ma torch_dtype=torch.bfloat16, parallel_config=parallel_config, offload_config=OffloadConfig(offload_type=WeightOffloadType.NO_CPU_OFFLOAD), + attention_config=attention_config or AttentionConfig.dense_attention(), ) return MiniMaxH3DenoisingStage(manager, runtime), transformer +def test_single_gpu_stage_applies_sol_attention_config() -> None: + config = AttentionConfig.sol_attention(dense_timesteps=10, dense_layers=2, threshold_type="exact", sol_fp8=True) + + _, transformer = _stage(ParallelConfig(device_ids=[0]), config) + + transformer.set_attention_config.assert_called_once_with(config) + + def test_local_embedding_layout_selects_only_rank_owned_rows() -> None: layout = _build_local_embedding_layout( seq_len=12, diff --git a/tools/validation/benchmark_minimax_h3_quantization.py b/tools/validation/benchmark_minimax_h3_quantization.py index 9ec5ea1..50d43a7 100644 --- a/tools/validation/benchmark_minimax_h3_quantization.py +++ b/tools/validation/benchmark_minimax_h3_quantization.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: Apache-2.0 -"""Benchmark MiniMax H3 BF16 and online-quantized single-GPU profiles.""" +"""Benchmark MiniMax H3 dense/Sol and BF16/FP8 single-GPU profiles.""" from __future__ import annotations @@ -23,8 +23,13 @@ def _package_version(name: str) -> str | None: def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--model-root", default="/hhb-data/aigc/model_zoo/MiniMaxAI_MiniMax-H3") - parser.add_argument("--backend", choices=("bf16", "torchao-fp8", "tf-kernel-fp8", "bnb-nf4"), required=True) + parser.add_argument( + "--backend", + choices=("bf16", "bf16-sol", "fp8", "fp8-sol", "torchao-fp8", "bnb-nf4"), + required=True, + ) parser.add_argument("--prompt", default="Steam rises from the ramen while the family talks in the background.") + parser.add_argument("--prompt-file", type=Path, help="JSON file containing a top-level prompt string.") parser.add_argument("--duration", type=float, default=5.0) parser.add_argument("--steps", type=int, default=50) parser.add_argument("--seed", type=int, default=0) @@ -33,15 +38,22 @@ def main() -> None: parser.add_argument("--output", type=Path, required=True) parser.add_argument("--metrics-json", type=Path) args = parser.parse_args() + prompt = args.prompt + if args.prompt_file is not None: + prompt = json.loads(args.prompt_file.read_text(encoding="utf-8"))["prompt"] - quantization = None if args.backend == "bf16" else args.backend + uses_sol = args.backend.endswith("-sol") + quantization = "tf-kernel-fp8" if args.backend in {"fp8", "fp8-sol"} else args.backend + if args.backend in {"bf16", "bf16-sol"}: + quantization = None load_started = time.perf_counter() pipeline = load_minimax_h3_pipeline( args.model_root, partition="FL2VA", device=args.device, num_inference_steps=args.steps, - attn_impl=AttnImplType.FLASH_ATTN_4, + attn_impl=AttnImplType.SOL_ATTN if uses_sol else AttnImplType.FLASH_ATTN_4, + sol_fp8=args.backend == "fp8-sol", quantization=quantization, ) load_seconds = time.perf_counter() - load_started @@ -49,7 +61,7 @@ def main() -> None: generation_started = time.perf_counter() result = pipeline( task="t2va", - prompt=args.prompt, + prompt=prompt, conditions=[], target={ "short_edge": 768, @@ -65,17 +77,20 @@ def main() -> None: finally: pipeline.stop() + denoising_seconds = float(result.runtime_metrics["denoising_seconds"]) report = { "backend": args.backend, "model_root": str(Path(args.model_root)), "output": str(args.output), - "prompt": args.prompt, + "prompt": prompt, "duration_seconds": args.duration, "num_inference_steps": args.steps, "seed": args.seed, "aspect_ratio": args.aspect_ratio, "load_seconds": load_seconds, "generation_seconds": generation_seconds, + "denoising_steps_per_second": args.steps / denoising_seconds, + "generated_video_seconds_per_second": args.duration / generation_seconds, "save_seconds": save_seconds, "runtime_metrics": result.runtime_metrics, "versions": { From 6c42ed368ed0ef49d8ac3bfc4c8f513f59b3f54b Mon Sep 17 00:00:00 2001 From: Uxtio-Ada <414416158@qq.com> Date: Wed, 19 Aug 2026 05:41:06 +0000 Subject: [PATCH 15/15] docs: add FP8 Sol-Attn technical article Document the FP8 Linear and attention ownership boundary, post-RoPE QKV quantization, SM90 CuTe routed/exact mainloop, model-specific quality guards, and validated Wan and MiniMax-H3 ablations. Add English and Chinese pages, indexes, navigation, and attention-guide cross-links. Verification: bilingual MkDocs build passed; git diff --check passed. --- docs/en/attention.md | 4 + docs/en/blog/fp8_sol_attention.md | 351 ++++++++++++++++++++++++++++++ docs/en/blog/index.md | 1 + docs/zh/attention.md | 4 + docs/zh/blog/fp8_sol_attention.md | 336 ++++++++++++++++++++++++++++ docs/zh/blog/index.md | 1 + mkdocs.yml | 2 + 7 files changed, 699 insertions(+) create mode 100644 docs/en/blog/fp8_sol_attention.md create mode 100644 docs/zh/blog/fp8_sol_attention.md diff --git a/docs/en/attention.md b/docs/en/attention.md index 0634ff0..757e53c 100644 --- a/docs/en/attention.md +++ b/docs/en/attention.md @@ -166,6 +166,7 @@ else: | Pipeline | Dense Attention | Radial | Sol-Attn | Notes | |----------|-----------------|--------|----------|-------| | `Wan21VideoPipeline` | Yes | Yes | Experimental | Sol-Attn covers eligible self-attention calls | +| `MiniMaxH3Pipeline` | Yes | No | Experimental | FL2VA supports exact prefix sinks and FP8 Q/K/V on SM90 | | `Wan22VideoPipeline` | Yes | Yes | No | Sol-Attn is not wired into Wan2.2 yet | | `QwenImagePipeline` | Yes | No | No | Image generation doesn't need temporal sparse attention | | `ZImagePipeline` | Yes | No | No | Image generation doesn't need temporal sparse attention | @@ -214,6 +215,9 @@ Setting `dense_timesteps=0`, `dense_layers=0`, and a negative `tau` forces all KV blocks onto the exact route. The Wan optimized example exposes this as `--attention fp8-dense`; `--attention fp8-sol` enables centroid routing with the same FP8 Q/K/V and QK/PV kernel. +For the kernel data flow, precision boundaries, and H100 ablations, see the +[FP8 Sol-Attn technical article](blog/fp8_sol_attention.md). + ### QwenImagePipeline / ZImagePipeline diff --git a/docs/en/blog/fp8_sol_attention.md b/docs/en/blog/fp8_sol_attention.md new file mode 100644 index 0000000..8638ad9 --- /dev/null +++ b/docs/en/blog/fp8_sol_attention.md @@ -0,0 +1,351 @@ +--- +title: "FP8 Sol-Attn: Quantized Sparse Attention for Video DiTs on H100" +description: Combining tf-kernel W8A8 Linear GEMMs with block-scaled FP8 QKV and a routed SM90 CuTe attention mainloop. +date: 2026-08-19 +status: validated +validated_revision: b649f0e +hardware: 1 x NVIDIA H100 80 GB HBM3 +tags: + - fp8 + - sol-attn + - sparse-attention + - cute + - video-dit +--- + +# FP8 Sol-Attn: Quantized Sparse Attention for Video DiTs on H100 + +Video diffusion transformers spend most of their denoising time in two different matrix-multiplication families: +Linear layers in projections and feed-forward networks, and the QK/PV products inside attention. Quantizing only +Linear layers reduces weight traffic and model memory, but leaves long-sequence attention in BF16. Enabling sparse +attention reduces the number of exact KV blocks, but does not by itself use Hopper FP8 Tensor Cores. + +TeleFuser combines these optimizations without treating them as one interchangeable backend: + +- source-built `tf-kernel` provides dynamic W8A8 E4M3 Linear GEMMs; +- TeleFuser quantizes post-RoPE Q/K/V with attention-specific scale and layout policies; and +- the built-in SM90 CuTe Sol-Attn mainloop executes routed E4M3 QK and PV WGMMA with FP32 accumulation. + +The result is an independently configurable path for Wan2.1 and MiniMax-H3. BF16 Dense remains the default. This +article explains the ownership boundary, kernel data flow, quality protections, and the measured single-H100 result. + +Sol-Attn itself is prior work from NVIDIA's Sol-Engine. TeleFuser does not claim a new sparse-attention algorithm, +FP8 format, or Tensor Core primitive. The contribution described here is the framework and kernel engineering needed +to carry block-scaled FP8 operands through Sol routing and exact attention, while preserving model-specific dense +regions and existing fallbacks. + +## Validation Snapshot + +| Field | Value | +|---|---| +| Status | `validated` | +| Implementation revision | `b649f0e` | +| Validation date | 2026-08-19 | +| GPU | 1 x NVIDIA H100 80 GB HBM3 (SM90) | +| Software | Python 3.11.13, PyTorch 2.11.0+cu128, CUDA 12.8 | +| Optional extension | Source-built SM90 `tf-kernel` wheel for FP8 Linear GEMMs | +| Attention kernel | Built-in TeleFuser CuTe DSL SM90 Sol-Attn | +| Validated models | Wan2.1-T2V-1.3B and MiniMax-H3 FL2VA | + +These are point measurements for the stated hardware, revisions, prompts, and cold-start policy. They are not +performance or quality guarantees for another model, sequence length, GPU, or software stack. + +## The Boundary: Linear GEMM Is Not Attention GEMM + +The existing `tf_kernel.fp8_scaled_mm` operator accepts two-dimensional matrices and their scales. TeleFuser uses it +to replace selected `nn.Linear` modules: + +1. cache each weight matrix in E4M3 with one scale per output channel; +2. quantize each activation row to E4M3 at runtime; and +3. run the scaled GEMM with a BF16 output. + +That operator accelerates projections and feed-forward layers. It cannot directly execute +`softmax(QK^T)V`, build a dynamic Sol route, maintain online-softmax state, or merge exact and approximate KV blocks. +The FP8 Sol-Attn kernel is therefore not a duplicate implementation of `tf-kernel` FP8 GEMM. It consumes four-dimensional attention operands and owns the QK, softmax, PV, and sparse-route data flow. + +| Path | Owner | Input contract | Work performed | +|---|---|---|---| +| FP8 Linear | `tf-kernel` through `telefuser.ops.fp8_gemm` | 2D E4M3 activation and weight matrices | Projection and FFN GEMM, BF16 output | +| FP8 QKV preparation | `telefuser.ops.fp8_attention` | Post-RoPE BF16 `[B,T,H,128]` | Scale calculation, E4M3 conversion, V relayout | +| FP8 Sol-Attn | `telefuser.kernel.sol_attn` | E4M3 Q/K/V plus FP32 scales | Routing, exact/approx attention, online softmax, BF16 output | + +Keeping these layers separate also allows controlled ablations: FP8 Linear can run with dense BF16 attention, and +BF16 Linear can run with BF16 Sol-Attn. + +## Design Goals, Non-Goals, and Alternatives + +The implementation was designed to: + +- preserve BF16 Dense as the unchanged default and expose FP8 Linear and FP8 attention independently; +- keep model code on `telefuser.ops` while architecture-specific dispatch stays below the public ops boundary; +- use native Hopper FP8 Tensor Cores for QK and PV, with FP32 accumulation and BF16 output; +- avoid materializing the full attention matrix or a global route mask; +- support exact KV sinks, dense step/layer guards, partial FP8 layer ranges, and non-aligned token tails; and +- retain a BF16 fallback for unsupported contracts and runtime failures. + +It was not intended to replace `tf-kernel` Linear GEMM, change checkpoint serialization, quantize text encoders or VAEs, make every attention variant FP8, or claim that every FP8 configuration must be faster. + +Several alternatives were measured or rejected during development: + +- **Reuse `tf_kernel.fp8_scaled_mm` for attention.** Its 2D GEMM contract cannot express online softmax, dynamic + routing, block sinks, or exact/summary merging. +- **Keep Q/K/V in BF16 after FP8 Linear.** This is a useful memory and Linear-throughput ablation, but leaves QK/PV + on the BF16 path and does not satisfy the attention optimization goal. +- **Route FP8 Q/K/V through the Triton reference path on H100.** It provides portability and a fallback, but its + conversion, launch, and Tensor Core utilization were slower than the specialized CuTe mainloop at production shapes. +- **Quantize every attention layer.** This maximized FP8 coverage but caused visible video degradation. Partial-layer + controls retained the measured speedup with a better quality boundary. + +## End-to-End Data Flow + +```mermaid +flowchart LR + H[BF16 hidden states] --> LQ[Dynamic activation quantization] + W[Cached E4M3 Linear weights] --> LG + LQ --> LG[tf-kernel FP8 Linear GEMMs] + LG --> P[BF16 Q/K/V projections] + P --> R[Q/K norm and RoPE] + R --> FQ[Fused Q/K/V FP8 preparation] + FQ --> QK[E4M3 Q/K
one scale per N64 head block] + FQ --> V[E4M3 V
per-channel scale and PV layout] + QK --> C[Block summaries and route thresholds] + V --> C + QK --> M[SM90 CuTe Sol mainloop] + V --> M + C --> M + M --> O[BF16 attention output] +``` + +This is a fused attention mainloop, not a claim that the full graph is one CUDA kernel. QKV quantization and centroid +preprocessing remain separate Triton kernels. The mainloop fuses the expensive routed/exact QK, online softmax, and PV +work so it does not materialize a full attention matrix or a global dense routing mask. + +## Attention-Specific FP8 Preparation + +Q, K, and V are quantized after Q/K normalization and RoPE. Quantizing earlier would require the following operators +to understand FP8 scales and would move the quantization boundary away from the values actually consumed by +attention. + +For every batch, head, and 64-token Q or K block, TeleFuser computes one E4M3 scale: + +$$ +s_{q,bh} = \frac{\max |Q_{b,h,64\text{-token block},:}|}{448}, \qquad +s_{k,bh} = \frac{\max |K_{b,h,64\text{-token block},:}|}{448}. +$$ + +V uses one scale per batch, head, and channel across the token dimension: + +$$ +s_{v,bhd} = \frac{\max_t |V_{b,t,h,d}|}{448}. +$$ + +The fused SM90 preparation path uses two Triton launches. The first reads BF16 Q/K/V once, writes E4M3 Q/K and +their block scales, and accumulates V-channel maxima. The second quantizes V directly into token-contiguous backing +storage. That layout is still exposed as `[B,T,H,D]`, but makes the token dimension contiguous for the K-major PV +WGMMA operand and avoids a separate transpose before attention. + +The fallback preparation path uses the same public scale contract with PyTorch operations. This keeps model code on +the public ops layer and makes unsupported devices testable without importing the CuTe backend. + +## Sol Routing + +Sol-Attn partitions the sequence into 64-token Q and KV blocks. Preprocessing builds a K summary and a V summary for +each KV block. For each Q block and head, the `diag` or `exact` estimator derives a threshold of the form + +$$ +\theta = \mu + \tau\sigma, +$$ + +where the exact estimator retains the full second moment and the diagonal estimator uses only per-channel variance. +The CuTe mainloop evaluates Q against groups of K summaries, reduces the distributed WGMMA accumulator into route +scores, and creates a CTA-local exact-block bitmask. + +- Important blocks take the **exact route** and execute full QK, online softmax, and PV. +- Remaining blocks take the **summary route**, using the K/V summaries with block-length correction. +- Configured sink blocks are always exact, regardless of their route score. + +The summary route is not equivalent to dropping a KV block. It retains a compressed contribution in the same online-softmax state. The threshold controls how much work is promoted back to exact attention. + +## The SM90 CuTe Mainloop + +The Hopper specialization uses 64x64 QK tiles, head dimension 128, one 128-thread warpgroup, TMA K/V movement, and +WGMMA Tensor Core instructions. Its FP8 path adds the following work to the upstream BF16 structure: + +1. **Scale-aware route QK.** Block-scaled Q and quantized K summaries run through E4M3 WGMMA. Their FP32 accumulator + is multiplied by the corresponding Q and K-summary scales before routing decisions. +2. **In-mainloop route-mask construction.** Warp-local reductions convert route accumulators into a compact exact-block bitmask. Full groups and static tails have separate compile-time specializations. +3. **Exact E4M3 QK.** Selected KV blocks execute QK WGMMA into FP32, followed by scale application and online + softmax. +4. **Approximate summary contribution.** Non-exact columns consume the precomputed summaries and correct both the + numerator and denominator for the current KV-block length. +5. **E4M3 PV.** Post-softmax probabilities are converted to E4M3 and multiplied by token-contiguous E4M3 V. The V + channel scale is applied to the FP32 output accumulator after all PV contributions. +6. **One online-softmax merge.** Exact and summary routes update the same row maxima, row sums, and output + accumulator. The full attention matrix is never written to HBM. +7. **Split-KV for long sequences.** On SM90, `auto` selects two splits for FP8 sequences at or above 16,384 tokens + and four splits at or above 65,536 tokens. A final log-sum-exp reduction merges the partial outputs. + +```mermaid +flowchart TB + A[Q tile: 64 x 128] --> RQK[E4M3 route QK WGMMA] + KC[K-summary group] --> RQK + RQK --> RM[Warp reductions and exact-block bitmask] + RM -->|exact bit| EQK[E4M3 exact QK WGMMA] + K[Selected K tile] --> EQK + RM -->|summary bit| AP[Summary score and V-summary contribution] + VC[V summaries] --> AP + EQK --> OS[Shared FP32 online-softmax state] + AP --> OS + OS --> P[Probabilities converted to E4M3] + P --> PV[E4M3 PV WGMMA] + V[Token-contiguous V tile] --> PV + PV --> S[Apply V channel scales] + S --> O[BF16 output tile] +``` + +The kernel cache key includes device, architecture, batch, token count, head count, KV splits, and input dtype. This +prevents a BF16 specialization from being reused for E4M3 inputs and makes the first-execution compilation cost +explicit in cold-start measurements. + +## Diffusion Quality Protections + +FP8 error and sparse-routing error accumulate across many denoising layers and steps. TeleFuser exposes three +orthogonal controls instead of forcing one all-layer policy: + +- `dense_timesteps` keeps early, high-noise-sensitive denoising steps dense; +- `dense_layers` keeps the first transformer layers dense at every sparse step; and +- `sol_fp8_layer_start` / `sol_fp8_layer_end` restrict E4M3 Q/K/V to a half-open layer range. + +Wan2.1 uses FP8 attention only in layers 10-19 in the validated profile. This retained the measured performance while +avoiding the visible degradation observed when every attention layer used FP8 Q/K/V. + +MiniMax-H3 has a packed multimodal sequence, so it needs two additional protections. The complete conditioning +prefix is registered as an exact KV sink, and prefix queries are recomputed with BF16 dense attention. The first ten +steps and first two DiT layers also use matched packed FlashAttention-4. Token-refiner attention remains dense. + +Unsupported shapes, dtypes, devices, or runtime kernel failures retain the public attention fallback. FP8 operands +are dequantized before the BF16 fallback. Ring/USP attention remains dense because its online distributed merge needs +log-sum-exp behavior outside the current Sol contract. + +## Performance Results + +### MiniMax-H3 FL2VA + +Each configuration ran in an independent clean process on one H100 80 GB with no other GPU processes. The workload +used the official complex starship T2VA prompt, 1344x768 output, 124 frames at 24 FPS, a five-second request, 50 +denoising steps, and seed 0. Timing includes first-execution kernel/JIT costs. `denoising_steps_per_second` is +`50 / runtime_metrics["denoising_seconds"]`; peak memory is `torch.cuda.max_memory_allocated()` during generation. +End-to-end generation excludes MP4 saving. + +In this table, **FP8 Dense** means FP8 Linear GEMMs with BF16 FlashAttention-4. Only **FP8 Sol** quantizes Q/K/V. + +| Linear | Attention | Denoising time | Throughput | Peak allocated | Generation time | +|---|---|---:|---:|---:|---:| +| BF16 | Dense FA4 | 310.409 s | 0.1611 step/s | 65.67 GiB | 457.5 s | +| BF16 | Sol-Attn | 213.442 s | 0.2343 step/s | 67.21 GiB | 321.3 s | +| FP8 | Dense FA4 | 276.836 s | 0.1806 step/s | 35.94 GiB | 397.5 s | +| FP8 | FP8 Sol-Attn | **188.185 s** | **0.2657 step/s** | **38.14 GiB** | **308.6 s** | + +FP8 Sol-Attn improves denoising throughput by **65.0%** over BF16 Dense while reducing peak allocated memory by +**41.9%**. Against FP8 Dense, Sol routing adds **47.1%** throughput for a **6.1%** memory increase. The ablation shows +that Sol provides most of the compute reduction, while FP8 Linear provides most of the model-memory reduction. + +The Sol rows use more memory than their matching dense rows because centroids, thresholds, route state, output/LSE, +and optional split-KV workspaces are live in addition to Q/K/V. Sol is a compute optimization, not a guarantee of +lower attention workspace. + +### Wan2.1-T2V-1.3B + +The Wan cold-start run used 832x480, 81 frames, 50 UniPC steps, CFG 5.0, sigma shift 5.0, seed 42, and the official +boxing-cats prompt. Timing starts after model loading, includes first-execution kernel/JIT cost, and excludes MP4 +encoding. Both FP8 rows quantize all 300 transformer-block Linear layers and restrict E4M3 Q/K/V to layers 10-19. + +Here **FP8 Exact** is the exact QK/PV CuTe path with routing disabled; it is not BF16 dense attention. + +| Linear | Attention | Throughput | Peak allocated | +|---|---|---:|---:| +| BF16 | Dense | 0.8491 frames/s | 16.147 GiB | +| BF16 | Sol-Attn | 1.1090 frames/s | 17.023 GiB | +| FP8 | FP8 Exact, layers 10-19 | 0.8739 frames/s | 15.730 GiB | +| FP8 | FP8 Sol-Attn, layers 10-19 | **1.1565 frames/s** | **15.730 GiB** | + +FP8 Sol-Attn is **36.2%** faster than BF16 Dense and uses **2.6%** less peak allocated memory. The small 2.9% FP8 Exact gain also shows why quantization overhead must be measured: FP8 is not automatically faster when the matrices +are small or conversion and launch costs dominate. + +## Output Validation + +All four MiniMax-H3 profiles produced valid 1344x768 H.264 videos with 124 frames and synchronized AAC audio. Manual +midpoint-frame inspection found coherent content and no black frames or obvious numerical failure. This is a smoke +test, not a perceptual-quality study; structural similarity between independently diverging diffusion trajectories +must not be interpreted as an absolute video-quality score. + +For Wan, matching-attention comparisons measured 22.0257 dB PSNR / 0.828783 SSIM for FP8 Exact and 20.8502 dB PSNR / +0.792656 SSIM for FP8 Sol. The partial attention-layer range was selected after an all-layer FP8 run showed visible +degradation. + +Correctness coverage includes scale forwarding, dense guards, exact sinks, non-aligned token tails, constant-value +preservation, split-KV route weights, public-op fallback, and real H100 FP8 Sol execution. The full unit suite at the +validated revision completed with **1,639 passed and 11 skipped** tests. + +## Reproduction + +Build and install an SM90 `tf-kernel` wheel using its repository Makefile, then run from the TeleFuser repository +root. The CuTe Sol-Attn implementation is already packaged with TeleFuser. + +MiniMax-H3 ablation: + +```bash +python -m tools.validation.benchmark_minimax_h3_quantization \ + --model-root /path/to/MiniMax-H3 \ + --backend fp8-sol \ + --prompt-file /path/to/demo_prompt.json \ + --duration 5 --steps 50 --seed 0 --aspect-ratio 16:9 \ + --output outputs/minimax_h3_fp8_sol.mp4 \ + --metrics-json outputs/minimax_h3_fp8_sol.metrics.json +``` + +Repeat with `--backend bf16`, `bf16-sol`, `fp8`, and `fp8-sol`. Use a fresh process for every profile if comparing +the cold path. + +Wan2.1 FP8 Sol: + +```bash +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ + --attention fp8-sol --quantization tf-kernel-fp8 \ + --fp8-linear-scope all --fp8-layer-start 10 --fp8-layer-end 20 \ + --width 832 --height 480 --num-frames 81 --num-inference-steps 50 \ + --sample-solver unipc --cfg-scale 5.0 --sigma-shift 5.0 --seed 42 +``` + +## Limitations + +- The validated FP8 attention mainloop targets SM90, noncausal self-attention, equal Q/K/V shapes, and head dimension + 128. BF16 Sol has broader architecture fallbacks, but the performance result does not transfer to them. +- MiniMax-H3 online `tf-kernel` FP8 Linear quantization is currently single-GPU only. Its TP/FSDP loading contract + remains BF16. +- QKV quantization and centroid preprocessing are separate kernels. Further fusion may reduce launch and memory-traffic overhead, but would increase specialization and register pressure. +- CuTe compilation is shape- and dtype-specific. Cold-start latency includes compilation; persistent services should + evaluate warm steady state separately. +- The best FP8 layer range is model- and checkpoint-dependent. An all-layer setting should not be treated as the + default quality/performance point. +- Peak allocated memory is a CUDA allocator metric, not total process or device memory. The experiments report one + run per configuration and do not establish variance bounds. + +## Related Work + +[Sol-Attn](https://arxiv.org/abs/2607.24027) and the +[Sol-Engine implementation](https://github.com/NVlabs/Sana/tree/sol-engine) define the dynamic summary/exact routing +algorithm and architecture-specific sparse-attention kernels used as the starting point. TeleFuser adapts that work +to its public attention dispatch, model runtime state, packed multimodal sequences, exact sinks, and independent +quantization configuration. + +[FlashAttention](https://arxiv.org/abs/2205.14135) established tiled IO-aware exact attention with online softmax. +The CuTe mainloop here retains that execution structure while adding Sol routing and FP8 scale handling. NVIDIA +Hopper WGMMA and TMA provide the hardware primitives; TeleFuser does not claim those primitives or E4M3 arithmetic as +new. + +The narrower contribution is a validated composition: dynamic W8A8 Linear GEMMs, post-RoPE block-scaled FP8 QKV, +layout-aware V preparation, and a scale-aware routed/exact SM90 mainloop, exposed behind reversible model +configuration and guarded by full-pipeline quality checks. diff --git a/docs/en/blog/index.md b/docs/en/blog/index.md index 1608a76..436d113 100644 --- a/docs/en/blog/index.md +++ b/docs/en/blog/index.md @@ -13,6 +13,7 @@ alternatives, implementation tradeoffs, and hardware-specific results behind tha | Date | Article | Status | Validated platform | |---|---|---|---| +| 2026-08-19 | [FP8 Sol-Attn: Quantized Sparse Attention for Video DiTs on H100](fp8_sol_attention.md) | Validated | 1 x H100 80 GB | | 2026-08-06 | [CUDA IPC Ulysses: Overlapping Attention Communication on H100](cuda_ipc_ulysses.md) | Validated | 4 x H100 80 GB | ## Publication Contract diff --git a/docs/zh/attention.md b/docs/zh/attention.md index 365cc41..7722b79 100644 --- a/docs/zh/attention.md +++ b/docs/zh/attention.md @@ -166,6 +166,7 @@ else: | Pipeline | 密集注意力 | Radial | Sol-Attn | 说明 | |----------|-----------|--------|----------|------| | `Wan21VideoPipeline` | 支持 | 支持 | 实验性 | Sol-Attn 用于满足约束的 self-attention | +| `MiniMaxH3Pipeline` | 支持 | 不支持 | 实验性 | FL2VA 支持 exact prefix sink 与 SM90 FP8 Q/K/V | | `Wan22VideoPipeline` | 支持 | 支持 | 不支持 | 尚未接入 Wan2.2 | | `QwenImagePipeline` | 支持 | 不支持 | 不支持 | 图像生成不需要时序稀疏注意力 | | `ZImagePipeline` | 支持 | 不支持 | 不支持 | 图像生成不需要时序稀疏注意力 | @@ -212,6 +213,9 @@ Ring/USP 需要 LSE 做在线合并，因此仍使用支持 LSE 的密集后端 设置 `dense_timesteps=0`、`dense_layers=0` 和负数 `tau` 会强制所有 KV block 走 exact 路径。Wan 优化示例将其暴露为 `--attention fp8-dense`； `--attention fp8-sol` 使用相同的 FP8 Q/K/V 与 QK/PV kernel 并启用质心路由。 +关于 kernel 数据流、精度边界与 H100 消融结果，参见 +[FP8 Sol-Attn 技术文章](blog/fp8_sol_attention.md)。 + ### QwenImagePipeline / ZImagePipeline diff --git a/docs/zh/blog/fp8_sol_attention.md b/docs/zh/blog/fp8_sol_attention.md new file mode 100644 index 0000000..af838e9 --- /dev/null +++ b/docs/zh/blog/fp8_sol_attention.md @@ -0,0 +1,336 @@ +--- +title: "FP8 Sol-Attn：H100 视频 DiT 的量化稀疏注意力" +description: 将 tf-kernel W8A8 Linear GEMM、分块量化 FP8 QKV 与 SM90 CuTe Sol-Attn mainloop 组合起来。 +date: 2026-08-19 +status: validated +validated_revision: b649f0e +hardware: 1 x NVIDIA H100 80 GB HBM3 +tags: + - fp8 + - sol-attn + - sparse-attention + - cute + - video-dit +--- + +# FP8 Sol-Attn：H100 视频 DiT 的量化稀疏注意力 + +视频扩散 Transformer 的去噪时间主要消耗在两类矩阵乘法上：投影和 FFN 中的 Linear，以及注意力内部的 +QK/PV。只量化 Linear 可以减少权重访存和模型显存，但长序列注意力仍然使用 BF16；只启用稀疏注意力 +可以减少精确计算的 KV block 数量，但不会自动使用 Hopper 的 FP8 Tensor Core。 + +TeleFuser 将二者组合起来，同时保留清晰的实现边界： + +- 源码构建的 `tf-kernel` 提供动态 W8A8 E4M3 Linear GEMM； +- TeleFuser 按注意力需求量化 post-RoPE Q/K/V，并生成对应 scale 和 V layout； +- 内置的 SM90 CuTe Sol-Attn mainloop 使用 E4M3 WGMMA 完成 route QK、exact QK 和 PV，累加使用 FP32。 + +Wan2.1 和 MiniMax-H3 可以分别开关 Linear FP8 与 Sol-Attn，BF16 Dense 仍是默认路径。本文说明这两类 +FP8 GEMM 为什么不能互相替代、kernel 数据流如何组织、如何保护扩散生成精度，以及单张 H100 上的实测结果。 + +Sol-Attn 算法来自 NVIDIA Sol-Engine。TeleFuser 不声称提出了新的稀疏注意力算法、FP8 格式或 Tensor +Core 指令。这里的工作重点是把分块缩放的 FP8 operand 接入 Sol routing 和 exact attention，并在框架层 +保留模型特定的 dense 区域、精确 sink、回退路径与可控配置。 + +## 验证快照 + +| 项目 | 值 | +|---|---| +| 状态 | `validated` | +| 实现 revision | `b649f0e` | +| 验证日期 | 2026-08-19 | +| GPU | 1 x NVIDIA H100 80 GB HBM3（SM90） | +| 软件 | Python 3.11.13、PyTorch 2.11.0+cu128、CUDA 12.8 | +| 可选扩展 | 为 SM90 源码构建的 `tf-kernel` wheel，用于 FP8 Linear GEMM | +| 注意力 kernel | TeleFuser 内置 CuTe DSL SM90 Sol-Attn | +| 验证模型 | Wan2.1-T2V-1.3B、MiniMax-H3 FL2VA | + +下文数据是指定硬件、revision、prompt 和冷启动策略下的单点测量，不代表其他模型、序列长度、GPU 或 +软件栈上的性能与质量保证。 + +## 边界：Linear GEMM 不等于 Attention GEMM + +已有的 `tf_kernel.fp8_scaled_mm` 接收二维矩阵及其 scale。TeleFuser 用它替换选定的 `nn.Linear`： + +1. 将权重按输出通道量化成 E4M3 并缓存； +2. 在运行时按 activation row 量化输入； +3. 执行 scaled GEMM，输出回到 BF16。 + +这个算子可以加速投影与 FFN，却不能直接执行 `softmax(QK^T)V`、动态构建 Sol route、维护 online +softmax 状态，或合并 exact 与 summary KV block。因此 FP8 Sol-Attn 不是对 `tf-kernel` FP8 GEMM 的 +重复实现。它接收四维注意力 operand，并负责 QK、softmax、PV 与稀疏 route 的完整数据流。 + +| 路径 | 所属模块 | 输入约束 | 负责的计算 | +|---|---|---|---| +| FP8 Linear | `tf-kernel`，通过 `telefuser.ops.fp8_gemm` 调用 | 二维 E4M3 activation/weight | Projection 和 FFN GEMM，输出 BF16 | +| FP8 QKV preparation | `telefuser.ops.fp8_attention` | Post-RoPE BF16 `[B,T,H,128]` | 计算 scale、转换 E4M3、调整 V layout | +| FP8 Sol-Attn | `telefuser.kernel.sol_attn` | E4M3 Q/K/V 与 FP32 scale | Routing、exact/summary attention、online softmax、输出 BF16 | + +这个边界也支持严格消融：FP8 Linear 可以搭配 BF16 Dense Attention，BF16 Linear 也可以搭配 BF16 +Sol-Attn。 + +## 设计目标、非目标与备选方案 + +实现目标包括： + +- 保持 BF16 Dense 默认行为不变，并让 FP8 Linear 与 FP8 attention 能够独立开关； +- 模型代码只调用 `telefuser.ops`，architecture-specific dispatch 位于 public ops 边界以下； +- QK 与 PV 都使用 Hopper 原生 FP8 Tensor Core，累加使用 FP32，输出回到 BF16； +- 不落盘完整 attention matrix 或全局 route mask； +- 支持 exact KV sink、dense step/layer guard、部分 FP8 layer range 和非对齐 token tail； +- 对不满足约束或 runtime failure 保留 BF16 fallback。 + +非目标包括替代 `tf-kernel` Linear GEMM、改变 checkpoint 格式、量化 text encoder/VAE、让所有 attention +variant 都使用 FP8，或声称任何 FP8 配置都必然更快。 + +开发过程中评估或排除了以下方案： + +- **直接复用 `tf_kernel.fp8_scaled_mm` 做 attention**：二维 GEMM contract 无法表达 online softmax、动态 + routing、block sink 或 exact/summary merge。 +- **FP8 Linear 后继续使用 BF16 Q/K/V**：这是有效的显存和 Linear 吞吐消融，但 QK/PV 仍在 BF16 路径， + 没有完成 attention 优化目标。 +- **在 H100 上让 FP8 Q/K/V 进入 Triton reference path**：它适合 portability 与 fallback，但在生产 shape + 下的 conversion、launch 和 Tensor Core 利用率不如专用 CuTe mainloop。 +- **量化所有 attention layer**：FP8 覆盖率最高，但视频出现可见退化；部分层控制保留了实测加速，并提供 + 更合理的质量边界。 + +## 端到端数据流 + +```mermaid +flowchart LR + H[BF16 hidden states] --> LQ[动态量化 activation] + W[缓存的 E4M3 Linear 权重] --> LG + LQ --> LG[tf-kernel FP8 Linear GEMM] + LG --> P[BF16 Q/K/V projection] + P --> R[Q/K norm 与 RoPE] + R --> FQ[融合 Q/K/V FP8 preparation] + FQ --> QK[E4M3 Q/K
每 N64 head block 一个 scale] + FQ --> V[E4M3 V
按通道 scale 与 PV layout] + QK --> C[Block summary 与 route threshold] + V --> C + QK --> M[SM90 CuTe Sol mainloop] + V --> M + C --> M + M --> O[BF16 attention output] +``` + +这里的“融合”特指 attention mainloop，并不是声称整张计算图只包含一个 CUDA kernel。QKV 量化与 +centroid preprocessing 仍是独立的 Triton kernel。CuTe mainloop 融合了最重的 route/exact QK、online +softmax 和 PV，因而不需要把完整注意力矩阵或全局 dense route mask 写入 HBM。 + +## 面向注意力的 FP8 Preparation + +Q、K、V 在 Q/K norm 与 RoPE 之后量化。若更早量化，后续算子也必须理解 FP8 scale，而且量化边界 +不再对应 attention 实际消费的数值。 + +每个 batch、head 和 64-token Q/K block 使用一个 E4M3 scale： + +$$ +s_{q,bh} = \frac{\max |Q_{b,h,64\text{-token block},:}|}{448}, \qquad +s_{k,bh} = \frac{\max |K_{b,h,64\text{-token block},:}|}{448}. +$$ + +V 在 token 维度上按 batch、head、channel 计算 scale： + +$$ +s_{v,bhd} = \frac{\max_t |V_{b,t,h,d}|}{448}. +$$ + +SM90 快速路径使用两个 Triton launch。第一个只读取一次 BF16 Q/K/V，写出 E4M3 Q/K 及 block scale， +并归约 V-channel 最大值；第二个把 V 直接量化到 token-contiguous backing storage。对外仍是 +`[B,T,H,D]` view，但 token 维对 K-major PV WGMMA 连续，避免 attention 前再做一次 transpose。 + +PyTorch fallback 保持相同的公共 scale contract，使模型代码始终调用 public ops，也可以在不加载 CuTe +backend 的设备上测试。 + +## Sol Routing + +Sol-Attn 将序列划分为 64-token Q/KV block。预处理为每个 KV block 构造 K summary 与 V summary。 +`diag` 或 `exact` estimator 为每个 Q block 和 head 计算如下形式的阈值： + +$$ +\theta = \mu + \tau\sigma. +$$ + +`exact` estimator 保留完整二阶矩，`diag` 只使用逐通道方差。CuTe mainloop 让 Q 与成组的 K summary +计算 route score，再把分布在 WGMMA accumulator 中的数据归约成 CTA-local exact-block bitmask。 + +- 重要 block 进入 **exact route**，执行完整 QK、online softmax 与 PV； +- 其余 block 进入 **summary route**，使用 K/V summary 并校正实际 block 长度； +- 配置为 sink 的 block 无条件走 exact route。 + +Summary route 并不是直接丢弃 KV block，而是在同一 online-softmax 状态中保留其压缩贡献。阈值决定 +多少 block 会重新提升为精确注意力。 + +## SM90 CuTe Mainloop + +Hopper specialization 使用 64x64 QK tile、128 维 head、一个 128-thread warpgroup、TMA K/V 搬运和 +WGMMA Tensor Core。FP8 路径在 BF16 Sol 结构上增加以下工作： + +1. **Scale-aware route QK**：分块量化 Q 与量化 K summary 通过 E4M3 WGMMA，FP32 accumulator 在 route + 判定前乘回对应 scale。 +2. **Mainloop 内构造 route mask**：warp 内归约将 route accumulator 转成紧凑 exact-block bitmask；完整 + group 与静态 tail 使用不同的编译期 specialization。 +3. **Exact E4M3 QK**：被选中的 KV block 执行 QK WGMMA，输出 FP32，随后应用 scale 和 online softmax。 +4. **Summary contribution**：非 exact 列消费预计算 summary，并同时校正分子、分母与当前 KV block 长度。 +5. **E4M3 PV**：softmax probability 转为 E4M3，与 token-contiguous E4M3 V 相乘；全部 PV 完成后再将 + V channel scale 应用到 FP32 output accumulator。 +6. **统一 online-softmax merge**：exact 与 summary route 更新相同的 row max、row sum 和 output + accumulator，不落盘完整 attention matrix。 +7. **长序列 Split-KV**：SM90 `auto` 策略在 FP8 序列长度达到 16,384 时选择两个 split，达到 65,536 + 时选择四个 split，最后通过 log-sum-exp reduction 合并部分结果。 + +```mermaid +flowchart TB + A[Q tile: 64 x 128] --> RQK[E4M3 route QK WGMMA] + KC[K-summary group] --> RQK + RQK --> RM[Warp 归约与 exact-block bitmask] + RM -->|exact bit| EQK[E4M3 exact QK WGMMA] + K[选中的 K tile] --> EQK + RM -->|summary bit| AP[Summary score 与 V-summary contribution] + VC[V summaries] --> AP + EQK --> OS[共享 FP32 online-softmax 状态] + AP --> OS + OS --> P[Probability 转换为 E4M3] + P --> PV[E4M3 PV WGMMA] + V[Token-contiguous V tile] --> PV + PV --> S[应用 V channel scale] + S --> O[BF16 output tile] +``` + +Kernel cache key 包含 device、architecture、batch、token 数、head 数、KV split 与 input dtype，避免 BF16 +specialization 被错误复用于 E4M3 输入。冷启动测量也因此明确包含第一次 CuTe 编译开销。 + +## 扩散模型精度保护 + +FP8 误差和 sparse routing 误差会在多层、多步去噪中累积。TeleFuser 提供三个彼此独立的控制项： + +- `dense_timesteps` 让最初、对噪声敏感的去噪步骤保持 dense； +- `dense_layers` 让每个 sparse step 的前若干 Transformer 层保持 dense； +- `sol_fp8_layer_start` / `sol_fp8_layer_end` 将 E4M3 Q/K/V 限制在半开层区间。 + +经过验证的 Wan2.1 配置只在第 10-19 层使用 FP8 attention。这个区间保留了性能收益，同时避免了所有 +attention layer 都量化时观察到的明显画质退化。 + +MiniMax-H3 使用 packed multimodal sequence，因此增加了两项保护：完整 condition prefix 被注册为 exact +KV sink，prefix query 使用 BF16 dense attention 重新计算。前十个 step、前两个 DiT layer 使用匹配的 +packed FlashAttention-4，token refiner 也始终保持 dense。 + +不支持的 shape、dtype、device 或 kernel runtime failure 会保留公共 attention fallback。FP8 operand 会先 +反量化再进入 BF16 fallback。Ring/USP 仍走 dense，因为它的分布式 online merge 需要当前 Sol contract +之外的 log-sum-exp 行为。 + +## 性能结果 + +### MiniMax-H3 FL2VA + +四个配置分别在单张 H100 80 GB 的独立干净进程中运行，GPU 上没有其他进程。Workload 使用官方复杂 +星舰 T2VA prompt、1344x768、124 帧、24 FPS、请求时长 5 秒、50 个去噪 step 和 seed 0。计时包含首次 +kernel/JIT 开销。`denoising_steps_per_second` 定义为 `50 / runtime_metrics["denoising_seconds"]`，峰值 +显存为生成期间的 `torch.cuda.max_memory_allocated()`；端到端生成时间不包含 MP4 保存。 + +下表的 **FP8 Dense** 表示 FP8 Linear GEMM + BF16 FlashAttention-4，只有 **FP8 Sol** 会量化 Q/K/V。 + +| Linear | Attention | 去噪时间 | 吞吐 | 峰值 allocated | 生成时间 | +|---|---|---:|---:|---:|---:| +| BF16 | Dense FA4 | 310.409 s | 0.1611 step/s | 65.67 GiB | 457.5 s | +| BF16 | Sol-Attn | 213.442 s | 0.2343 step/s | 67.21 GiB | 321.3 s | +| FP8 | Dense FA4 | 276.836 s | 0.1806 step/s | 35.94 GiB | 397.5 s | +| FP8 | FP8 Sol-Attn | **188.185 s** | **0.2657 step/s** | **38.14 GiB** | **308.6 s** | + +FP8 Sol-Attn 相比 BF16 Dense 将去噪吞吐提高 **65.0%**，峰值 allocated 显存降低 **41.9%**。相比 +FP8 Dense，Sol routing 以 **6.1%** 的额外显存换来 **47.1%** 的吞吐提升。消融说明 Sol 主要减少计算量， +FP8 Linear 主要降低模型显存，组合后得到最好的吞吐/显存折中。 + +Sol 相比相同 Linear 精度的 Dense 会多使用 centroids、threshold、route state、output/LSE 和可选 split-KV +workspace。因此 Sol 是计算优化，并不保证 attention workspace 更小。 + +### Wan2.1-T2V-1.3B + +Wan 冷启动实验使用 832x480、81 帧、50 个 UniPC step、CFG 5.0、sigma shift 5.0、seed 42 和官方拳击猫 +prompt。计时从模型加载完成后开始，包含首次 kernel/JIT 开销，不包含 MP4 编码。两个 FP8 配置均量化 +全部 300 个 Transformer-block Linear，并只在第 10-19 层使用 E4M3 Q/K/V。 + +这里的 **FP8 Exact** 是关闭 routing 的 exact QK/PV CuTe 路径，不是 BF16 Dense Attention。 + +| Linear | Attention | 吞吐 | 峰值 allocated | +|---|---|---:|---:| +| BF16 | Dense | 0.8491 frames/s | 16.147 GiB | +| BF16 | Sol-Attn | 1.1090 frames/s | 17.023 GiB | +| FP8 | FP8 Exact，第 10-19 层 | 0.8739 frames/s | 15.730 GiB | +| FP8 | FP8 Sol-Attn，第 10-19 层 | **1.1565 frames/s** | **15.730 GiB** | + +FP8 Sol-Attn 比 BF16 Dense 快 **36.2%**，峰值 allocated 显存少 **2.6%**。FP8 Exact 只有 2.9% 的提升， +也说明量化必须实测：矩阵较小，或 conversion 与 launch overhead 占主导时，FP8 不会自动更快。 + +## 输出验证 + +四个 MiniMax-H3 配置均生成有效的 1344x768 H.264 视频：124 帧并带同步 AAC 音频。人工检查中间帧时， +内容连贯，没有黑帧或明显数值异常。这只是 smoke test，不是感知质量研究；不同扩散轨迹之间的 SSIM +不能解释为绝对视频质量分数。 + +Wan 在 matching-attention 对比下，FP8 Exact 为 22.0257 dB PSNR / 0.828783 SSIM，FP8 Sol 为 +20.8502 dB PSNR / 0.792656 SSIM。选择部分 attention layer 的原因，是全层 FP8 Q/K/V 实验出现了可见退化。 + +正确性覆盖包括 scale forwarding、dense guard、exact sink、非对齐 token tail、constant-value +preservation、split-KV route weight、public-op fallback 和真实 H100 FP8 Sol 执行。验证 revision 的完整 +单测结果为 **1,639 passed、11 skipped**。 + +## 复现 + +先通过 `tf-kernel/` 的 Makefile 构建并安装 SM90 wheel，再从 TeleFuser 仓库根目录运行。CuTe Sol-Attn +已经随 TeleFuser 源码提供。 + +MiniMax-H3 消融： + +```bash +python -m tools.validation.benchmark_minimax_h3_quantization \ + --model-root /path/to/MiniMax-H3 \ + --backend fp8-sol \ + --prompt-file /path/to/demo_prompt.json \ + --duration 5 --steps 50 --seed 0 --aspect-ratio 16:9 \ + --output outputs/minimax_h3_fp8_sol.mp4 \ + --metrics-json outputs/minimax_h3_fp8_sol.metrics.json +``` + +依次将 `--backend` 改为 `bf16`、`bf16-sol`、`fp8` 与 `fp8-sol`。对比冷启动时，每个 profile 必须使用 +新进程。 + +Wan2.1 FP8 Sol： + +```bash +python examples/wan_video/wan21_1_3b_text_to_video_optimized_h100.py \ + --model-root /path/to/Wan2.1-T2V-1.3B \ + --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ + --attention fp8-sol --quantization tf-kernel-fp8 \ + --fp8-linear-scope all --fp8-layer-start 10 --fp8-layer-end 20 \ + --width 832 --height 480 --num-frames 81 --num-inference-steps 50 \ + --sample-solver unipc --cfg-scale 5.0 --sigma-shift 5.0 --seed 42 +``` + +## 限制 + +- 已验证的 FP8 attention mainloop 面向 SM90、noncausal self-attention、相同 Q/K/V shape 和 128 维 head。 + BF16 Sol 有更广的 architecture fallback，但本文性能数据不能直接迁移到这些路径。 +- MiniMax-H3 在线 `tf-kernel` FP8 Linear 目前只支持单 GPU；其 TP/FSDP loading contract 仍为 BF16。 +- QKV quantization 与 centroid preprocessing 是独立 kernel。进一步融合可能减少 launch 与访存开销， + 但也会提高 specialization 数量和 register pressure。 +- CuTe 编译与 shape、dtype 绑定。冷启动结果包含编译，常驻服务还应单独评估 warm steady state。 +- 最佳 FP8 layer range 依赖模型与 checkpoint，不能把全层 FP8 当作默认质量/性能点。 +- Peak allocated 是 CUDA allocator 指标，不是进程或整张 GPU 的总显存。每个配置只有一次测量，尚未给出 + 方差范围。 + +## 相关工作 + +[Sol-Attn](https://arxiv.org/abs/2607.24027) 与 +[Sol-Engine 实现](https://github.com/NVlabs/Sana/tree/sol-engine) 提出了这里作为起点的动态 summary/exact +routing 算法和多架构 sparse-attention kernel。TeleFuser 将其接入 public attention dispatch、模型 runtime +state、packed multimodal sequence、exact sink 与独立量化配置。 + +[FlashAttention](https://arxiv.org/abs/2205.14135) 建立了基于 tiled IO-aware exact attention 与 online +softmax 的执行结构。这里的 CuTe mainloop 保留这一结构并加入 Sol routing 与 FP8 scale handling。Hopper +WGMMA、TMA 和 E4M3 算术均是 NVIDIA 硬件能力，不属于 TeleFuser 的新算法声明。 + +本文工作的更准确边界是一个经过端到端验证的组合：动态 W8A8 Linear GEMM、post-RoPE 分块 FP8 QKV、 +面向 PV 的 V layout，以及 scale-aware routed/exact SM90 mainloop；这些能力通过可逆配置暴露，并受到完整 +视频生成质量检查的约束。 diff --git a/docs/zh/blog/index.md b/docs/zh/blog/index.md index cc7bc0a..622175b 100644 --- a/docs/zh/blog/index.md +++ b/docs/zh/blog/index.md @@ -12,6 +12,7 @@ description: 记录 TeleFuser 性能与运行时优化的分析、实现和验 | 日期 | 文章 | 状态 | 验证平台 | |---|---|---|---| +| 2026-08-19 | [FP8 Sol-Attn：H100 视频 DiT 的量化稀疏注意力](fp8_sol_attention.md) | 已验证 | 1 x H100 80 GB | | 2026-08-06 | [CUDA IPC Ulysses：在 H100 上重叠 Attention 通信](cuda_ipc_ulysses.md) | 已验证 | 4 x H100 80 GB | ## 发布约定 diff --git a/mkdocs.yml b/mkdocs.yml index 7beee63..c4a8616 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -127,6 +127,7 @@ plugins: Benchmarks: 基准测试 Technical Blog: 技术博客 Overview: 总览 + FP8 Sol-Attn: FP8 Sol-Attn CUDA IPC Ulysses: CUDA IPC Ulysses Parallel Inference: 并行推理 Communication Architecture: 通信架构 @@ -181,6 +182,7 @@ nav: - TeleFuser and AIPerf: benchmark_aiperf.md - Technical Blog: - Overview: blog/index.md + - FP8 Sol-Attn: blog/fp8_sol_attention.md - CUDA IPC Ulysses: blog/cuda_ipc_ulysses.md - Configuration: - configuration.md