bondscell_results_$1b768306-0e0e-414d-a501-5c62a2db720fqueued¤logsrunning¦outputbodyV

Computations as Trees

mimetext/htmlrootassigneelast_run_timestampA&nV^9persist_js_state·has_pluto_hook_features§cell_id$1b768306-0e0e-414d-a501-5c62a2db720fdepends_on_disabled_cells§runtimeXpublished_object_keysdepends_on_skipped_cells§errored$aa99b123-7164-4ed7-833f-b99b19118a0dqueued¤logsrunning¦outputbody[+] → 14 mimetext/plainrootassigneelast_run_timestampA&o,-5?persist_js_state·has_pluto_hook_features§cell_id$aa99b123-7164-4ed7-833f-b99b19118a0ddepends_on_disabled_cells§runtime?published_object_keysdepends_on_skipped_cells§errored$be693f92-de60-4b43-82b3-ee6cceea310aqueued¤logsrunning¦outputbody^

$$\frac{\partial}{\partial x} (2xy + (x-1)^2) = 2y + 2(x-1)$$

$$\frac{\partial}{\partial y} (2xy + (x-1)^2) = 2x$$

If x = 3 and y = 5, then these evaluate to 14 and 6.

mimetext/htmlrootassigneelast_run_timestampA&nZ6persist_js_state·has_pluto_hook_features§cell_id$be693f92-de60-4b43-82b3-ee6cceea310adepends_on_disabled_cells§runtimeppublished_object_keysdepends_on_skipped_cells§errored$1f1b384a-6588-45a5-9dd3-6de3face8bfbqueued¤logsrunning¦outputbody)ad_steps (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA&o5persist_js_state·has_pluto_hook_features§cell_id$1f1b384a-6588-45a5-9dd3-6de3face8bfbdepends_on_disabled_cells§runtime-+published_object_keysdepends_on_skipped_cells§errored$e55dfad2-db50-459f-ab54-fa7637fc3638queued¤logsrunning¦outputbodyq

How can we visualize both the forward and the reverse pass?

We can further visualize each steps we just took. First we do the forwards calculation, where we also build up our tree, then we go down the tree in the opposite direction to accumulate our gradient.

mimetext/htmlrootassigneelast_run_timestampA&nlpersist_js_state·has_pluto_hook_features§cell_id$e55dfad2-db50-459f-ab54-fa7637fc3638depends_on_disabled_cells§runtimeץpublished_object_keysdepends_on_skipped_cells§errored$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bqueued¤logsrunning¦outputbodyrTracked{Float64} val: Float64 3.0 name: Symbol x df: Nothing nothing deps: Array{Tracked}((0,)) Tracked[] mimetext/plainrootassigneelast_run_timestampA&o++persist_js_state·has_pluto_hook_features§cell_id$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bdepends_on_disabled_cells§runtime1published_object_keysdepends_on_skipped_cells§errored$7dca2c78-6faf-450a-8096-1c5b9f7f1295queued¤logsrunning¦outputbody+leaky_relu (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA&o-"@1persist_js_state·has_pluto_hook_features§cell_id$7dca2c78-6faf-450a-8096-1c5b9f7f1295depends_on_disabled_cells§runtimeJpublished_object_keysdepends_on_skipped_cells§errored$13487e65-5e48-4a37-9bea-f262dd7b6d56queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o+1persist_js_state·has_pluto_hook_features§cell_id$13487e65-5e48-4a37-9bea-f262dd7b6d56depends_on_disabled_cells§runtimeEpublished_object_keysdepends_on_skipped_cells§errored$96286b65-1a22-4458-a399-46579248cce4queued¤logslinemsgSaved animation to text/plaincell_id$96286b65-1a22-4458-a399-46579248cce4kwargsfn"/tmp/jl_PgxmhEjSpg.gif"text/plainidPlots_c6bd189bfile9/home/runner/.julia/packages/Plots/lW9ll/src/animation.jlgroupanimationlevelInforunning¦outputbody0mimetext/htmlrootassigneelast_run_timestampA&o3persist_js_state·has_pluto_hook_features§cell_id$96286b65-1a22-4458-a399-46579248cce4depends_on_disabled_cells§runtimeqSpublished_object_keysdepends_on_skipped_cells§errored$1a154bb7-93a3-4973-8908-788db77ac294queued¤logsrunning¦outputbody. mimetext/htmlrootassignees2last_run_timestampA&o6persist_js_state·has_pluto_hook_features§cell_id$1a154bb7-93a3-4973-8908-788db77ac294depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$8f1ec8d7-89ab-4bc5-9eef-2b0a69019854queued¤logsrunning¦outputbodyN

We'll try to approximate a sine curve:

mimetext/htmlrootassigneelast_run_timestampA&nm0persist_js_state·has_pluto_hook_features§cell_id$8f1ec8d7-89ab-4bc5-9eef-2b0a69019854depends_on_disabled_cells§runtimeepublished_object_keysdepends_on_skipped_cells§errored$795c510e-6280-42d9-8820-290ca07be88dqueued¤logsrunning¦outputbodyprefixMain.var"workspace#3".TrackedelementstypeArrayprefix_shortobjectiddff3d38f0020b412mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o,5 persist_js_state·has_pluto_hook_features§cell_id$795c510e-6280-42d9-8820-290ca07be88ddepends_on_disabled_cells§runtime͌kpublished_object_keysdepends_on_skipped_cells§errored$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726queued¤logsrunning¦outputbodyPB
Move me!
1 mimetext/htmlrootassigneelast_run_timestampA&o6Hpersist_js_state·has_pluto_hook_features§cell_id$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726depends_on_disabled_cells§runtimeV5published_object_keysdepends_on_skipped_cells§errored$85098c31-04a1-421c-b3f6-25fec9360588queued¤logsrunning¦outputbody

What if we use a (leaky) rectified linear unit (ReLU) as activation?

mimetext/htmlrootassigneelast_run_timestampA&nmGذpersist_js_state·has_pluto_hook_features§cell_id$85098c31-04a1-421c-b3f6-25fec9360588depends_on_disabled_cells§runtime:published_object_keysdepends_on_skipped_cells§errored$8ab0f55d-a393-4a8a-a48c-9ced26033f57queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,7persist_js_state·has_pluto_hook_features§cell_id$8ab0f55d-a393-4a8a-a48c-9ced26033f57depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$e01114fa-a847-424a-931f-8f42e623109fqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,|Rpersist_js_state·has_pluto_hook_features§cell_id$e01114fa-a847-424a-931f-8f42e623109fdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$2ba83769-89b4-4ba0-8726-9210f394b4e7queued¤logsrunning¦outputbodyV

Operations on Tracked

mimetext/htmlrootassigneelast_run_timestampA&n\hpersist_js_state·has_pluto_hook_features§cell_id$2ba83769-89b4-4ba0-8726-9210f394b4e7depends_on_disabled_cells§runtimeK0published_object_keysdepends_on_skipped_cells§errored$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0queued¤logsrunning¦outputbody[+] → 34 mimetext/plainrootassigneelast_run_timestampA&o,7jwpersist_js_state·has_pluto_hook_features§cell_id$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0depends_on_disabled_cells§runtimempublished_object_keysdepends_on_skipped_cells§errored$a5671f2f-48ca-4896-9428-147dc671d2b9queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,

Ok, let's create our function grad which will accumulate all intermediate gradients into a dictionary:

mimetext/htmlrootassigneelast_run_timestampA&nlOpersist_js_state·has_pluto_hook_features§cell_id$d5565717-239b-41ab-b311-d59b383130eddepends_on_disabled_cells§runtime4published_object_keysdepends_on_skipped_cells§errored$81eb8a2d-a3a9-45af-a5a5-b96aefd48712queued¤logsrunning¦outputbody34mimetext/plainrootassigneelast_run_timestampA&o,91persist_js_state·has_pluto_hook_features§cell_id$81eb8a2d-a3a9-45af-a5a5-b96aefd48712depends_on_disabled_cells§runtime͒published_object_keysdepends_on_skipped_cells§errored$2188a663-5a85-4ce4-bc8d-20383481e59bqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,8persist_js_state·has_pluto_hook_features§cell_id$2188a663-5a85-4ce4-bc8d-20383481e59bdepends_on_disabled_cells§runtimeMõpublished_object_keysdepends_on_skipped_cells§errored$db4f871a-3ca1-4bd3-a2c5-da7d6f599281queued¤logsrunning¦outputbody15mimetext/plainrootassigneelast_run_timestampA&o+Jpersist_js_state·has_pluto_hook_features§cell_id$db4f871a-3ca1-4bd3-a2c5-da7d6f599281depends_on_disabled_cells§runtime:npublished_object_keysdepends_on_skipped_cells§errored$53ae4070-8818-4d21-8648-19df9319918aqueued¤logsrunning¦outputbodyق

Let's write our own neural network!

mimetext/htmlrootassigneelast_run_timestampA&nlpersist_js_state·has_pluto_hook_features§cell_id$53ae4070-8818-4d21-8648-19df9319918adepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$79f71f9d-b491-4a2c-85a4-29ae8da4f312queued¤logsrunning¦outputbody8 3y + 2 * (x - 1) 3 + 2 * (x - 1) + 2 * (x - 1) + 2 * ( - 1) + 2 + mimetext/htmlrootassigneelast_run_timestampA&o7Tpersist_js_state·has_pluto_hook_features§cell_id$79f71f9d-b491-4a2c-85a4-29ae8da4f312depends_on_disabled_cells§runtime8"@published_object_keys50e7f5148-7698-11f0-1006-399f8abb1fe9/43c4266ba4b738bb50e7f5148-7698-11f0-1006-399f8abb1fe9/ae663cf966d227c650e7f5148-7698-11f0-1006-399f8abb1fe9/ed2ed7f6d9d1664650e7f5148-7698-11f0-1006-399f8abb1fe9/b9f6587eade9ac6050e7f5148-7698-11f0-1006-399f8abb1fe9/fbc7f8abf7436a3850e7f5148-7698-11f0-1006-399f8abb1fe9/520b800031481075depends_on_skipped_cells§errored$e450329b-46ee-4f53-b6f0-891d6596938cqueued¤logsrunning¦outputbody[+] → 14 mimetext/plainrootassigneelast_run_timestampA&o,58persist_js_state·has_pluto_hook_features§cell_id$e450329b-46ee-4f53-b6f0-891d6596938cdepends_on_disabled_cells§runtimeGpublished_object_keysdepends_on_skipped_cells§errored$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7queued¤logsrunning¦outputbodyelements6text/plain3text/plaintypeTupleobjectidfa3317f9f54f06c9mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o,쒰persist_js_state·has_pluto_hook_features§cell_id$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7depends_on_disabled_cells§runtime]published_object_keysdepends_on_skipped_cells§errored$d1312080-272c-4bd5-bcfe-bf47f8320961queued¤logsrunning¦outputbody`

The Tracked Data Structure

mimetext/htmlrootassigneelast_run_timestampA&n[셰persist_js_state·has_pluto_hook_features§cell_id$d1312080-272c-4bd5-bcfe-bf47f8320961depends_on_disabled_cells§runtimeOĵpublished_object_keysdepends_on_skipped_cells§errored$885cd51d-895f-4996-a23b-780498b5b810queued¤logsrunning¦outputbody

All overloads will do the operation (e.g. sum x and y), but also remember the pullback map and input variables for the reverse pass.

mimetext/htmlrootassigneelast_run_timestampA&n\$persist_js_state·has_pluto_hook_features§cell_id$885cd51d-895f-4996-a23b-780498b5b810depends_on_disabled_cells§runtimeypublished_object_keysdepends_on_skipped_cells§errored$d5559dba-9fe4-11ec-3744-ebd1408e7dc4queued¤logslinemsgSPrecompiling Plots... 809.1 ms ✓ JLLWrappers 865.4 ms ✓ OpenSSL_jll 772.4 ms ✓ Graphite2_jll 2749.6 ms ✓ RecipesBase 602.2 ms ✓ EpollShim_jll 1215.2 ms ✓ Libmount_jll 764.6 ms ✓ LLVMOpenMP_jll 775.8 ms ✓ Bzip2_jll 925.3 ms ✓ Xorg_libXau_jll 813.6 ms ✓ libpng_jll 5424.3 ms ✓ ColorSchemes 789.0 ms ✓ libfdk_aac_jll 740.1 ms ✓ LERC_jll 982.4 ms ✓ LAME_jll 828.5 ms ✓ EarCut_jll 860.9 ms ✓ Ogg_jll 1057.1 ms ✓ JpegTurbo_jll 948.2 ms ✓ Xorg_libXdmcp_jll 992.3 ms ✓ x265_jll 860.9 ms ✓ x264_jll 894.9 ms ✓ libaom_jll 917.6 ms ✓ Expat_jll 975.7 ms ✓ Zstd_jll 750.8 ms ✓ LZO_jll 886.7 ms ✓ Xorg_xtrans_jll 958.8 ms ✓ Opus_jll 977.6 ms ✓ Libiconv_jll 994.7 ms ✓ Libffi_jll 1087.4 ms ✓ Libgpg_error_jll 1071.5 ms ✓ Wayland_protocols_jll 915.5 ms ✓ Libuuid_jll 1117.6 ms ✓ Xorg_libpthread_stubs_jll 12463.7 ms ✓ StaticArrays 1169.8 ms ✓ FriBidi_jll 947.4 ms ✓ Pixman_jll 852.0 ms ✓ FreeType2_jll 998.5 ms ✓ libvorbis_jll 1067.0 ms ✓ Dbus_jll 908.9 ms ✓ Libtiff_jll 977.0 ms ✓ XML2_jll 3729.4 ms ✓ OpenSSL 988.1 ms ✓ Libgcrypt_jll 1020.6 ms ✓ StaticArrays → StaticArraysStatisticsExt 1019.9 ms ✓ ConstructionBase → ConstructionBaseStaticArraysExt 1140.6 ms ✓ StructArrays → StructArraysStaticArraysExt 1261.8 ms ✓ Fontconfig_jll 1041.4 ms ✓ Gettext_jll 1252.4 ms ✓ Wayland_jll 1091.3 ms ✓ XSLT_jll 1386.6 ms ✓ Glib_jll 2997.3 ms ✓ Xorg_libxcb_jll 5671.3 ms ✓ HTTP 1244.5 ms ✓ Xorg_xcb_util_jll 1233.5 ms ✓ Xorg_libX11_jll 1251.2 ms ✓ Xorg_xcb_util_image_jll 1194.8 ms ✓ Xorg_xcb_util_keysyms_jll 1229.9 ms ✓ Xorg_xcb_util_wm_jll 1466.2 ms ✓ Xorg_xcb_util_renderutil_jll 1669.9 ms ✓ Xorg_libXrender_jll 1590.6 ms ✓ Xorg_libXext_jll 11352.4 ms ✓ GeometryBasics 1644.8 ms ✓ Xorg_libXfixes_jll 1616.2 ms ✓ Xorg_libxkbfile_jll 1033.6 ms ✓ Xorg_libXinerama_jll 1643.7 ms ✓ Xorg_libXrandr_jll 2095.2 ms ✓ Cairo_jll 2127.5 ms ✓ Libglvnd_jll 1218.3 ms ✓ Xorg_libXcursor_jll 1575.0 ms ✓ Xorg_libXi_jll 1662.6 ms ✓ Xorg_xkbcomp_jll 1505.5 ms ✓ HarfBuzz_jll 1150.6 ms ✓ Xorg_xkeyboard_config_jll 1667.0 ms ✓ libass_jll 1856.7 ms ✓ Pango_jll 1186.3 ms ✓ xkbcommon_jll 25009.2 ms ✓ PlotUtils 1271.6 ms ✓ libdecor_jll 2189.2 ms ✓ FFMPEG_jll 2589.7 ms ✓ Qt5Base_jll 1361.3 ms ✓ GLFW_jll 1233.0 ms ✓ FFMPEG 41357.2 ms ✓ Parsers 1498.0 ms ✓ GR_jll 5038.7 ms ✓ PlotThemes 1556.6 ms ✓ JSON 6275.9 ms ✓ RecipesPipeline 9048.6 ms ✓ Plots 87 dependencies successfully precompiled in 54 seconds. 95 already precompiled. text/plaincell_id$d5559dba-9fe4-11ec-3744-ebd1408e7dc4kwargsidPlutoRunner_d1acb81efileP/home/runner/.julia/packages/Pluto/u5fFh/src/runner/PlutoRunner/src/io/stdout.jlgroupstdoutlevelLogLevel(-555)running¦outputbodymimetext/plainrootassigneelast_run_timestampA&o*Źpersist_js_state·has_pluto_hook_features§cell_id$d5559dba-9fe4-11ec-3744-ebd1408e7dc4depends_on_disabled_cells§runtime ŵpublished_object_keysdepends_on_skipped_cells§errored$49333dc7-b809-4bb1-b285-20a12d8b9209queued¤logsrunning¦outputbody$ mimeimage/svg+xmlrootassigneelast_run_timestampA&o4(persist_js_state·has_pluto_hook_features§cell_id$49333dc7-b809-4bb1-b285-20a12d8b9209depends_on_disabled_cells§runtimeecpublished_object_keysdepends_on_skipped_cells§errored$f6ce8448-d9ce-4453-9e47-dc6443d50f55queued¤logsrunning¦outputbody
Move me!
1 mimetext/htmlrootassigneelast_run_timestampA&o6persist_js_state·has_pluto_hook_features§cell_id$ec723b90-1c8b-4a87-8a86-fe17d6e85f22depends_on_disabled_cells§runtimejWhpublished_object_keysdepends_on_skipped_cells§errored$c9d70049-4793-41db-9075-e9e73e926c1aqueued¤logsrunning¦outputbody[+] → 8 mimetext/plainrootassigneelast_run_timestampA&o+8persist_js_state·has_pluto_hook_features§cell_id$c9d70049-4793-41db-9075-e9e73e926c1adepends_on_disabled_cells§runtime+published_object_keysdepends_on_skipped_cells§errored$17f9e2c0-aa70-4d4e-ba42-51782df43785queued¤logsrunning¦outputbody~

Computational Tree (Forward Pass)

mimetext/htmlrootassigneelast_run_timestampA&nWݾpersist_js_state·has_pluto_hook_features§cell_id$17f9e2c0-aa70-4d4e-ba42-51782df43785depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$0d186a1b-eb89-4be8-a237-e5a9608f4a30queued¤logsrunning¦outputbody'
Move me!
1 mimetext/htmlrootassigneelast_run_timestampA&o6Ypersist_js_state·has_pluto_hook_features§cell_id$0d186a1b-eb89-4be8-a237-e5a9608f4a30depends_on_disabled_cells§runtimeppublished_object_keysdepends_on_skipped_cells§errored$d2d21d41-9a4e-428b-b696-6878901849baqueued¤logsrunning¦outputbodyo1×100 Matrix{Float64}: 0.0 0.0634239 0.126592 0.189251 0.251148 … -0.126592 -0.0634239 -2.44929e-16mimetext/plainrootassigneeŷlast_run_timestampA&o-persist_js_state·has_pluto_hook_features§cell_id$d2d21d41-9a4e-428b-b696-6878901849badepends_on_disabled_cells§runtime'published_object_keysdepends_on_skipped_cells§errored$0b094198-cf44-41d7-a8dc-fd8fd0716bb4queued¤logsrunning¦outputbodyS

Helper Functions

mimetext/htmlrootassigneelast_run_timestampA&nn}persist_js_state·has_pluto_hook_features§cell_id$0b094198-cf44-41d7-a8dc-fd8fd0716bb4depends_on_disabled_cells§runtimeUpublished_object_keysdepends_on_skipped_cells§errored$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o-"аpersist_js_state·has_pluto_hook_features§cell_id$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9depends_on_disabled_cells§runtime
Move me!
1 mimetext/htmlrootassigneelast_run_timestampA&o6 persist_js_state·has_pluto_hook_features§cell_id$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$0b5e6560-81fd-4182-bba5-aca702fb3048queued¤logsrunning¦outputbody2=u mimetext/plainrootassigneelast_run_timestampA&o+Qpersist_js_state·has_pluto_hook_features§cell_id$0b5e6560-81fd-4182-bba5-aca702fb3048depends_on_disabled_cells§runtimeQpublished_object_keysdepends_on_skipped_cells§errored$8a70e835-dae6-4727-8204-95c87d5c23daqueued¤logsrunning¦outputbody?

We need some more overloads

mimetext/htmlrootassigneelast_run_timestampA&nlܘpersist_js_state·has_pluto_hook_features§cell_id$8a70e835-dae6-4727-8204-95c87d5c23dadepends_on_disabled_cells§runtime vpublished_object_keysdepends_on_skipped_cells§errored$337bc837-174f-4ed6-bee1-c9e80dc03b52queued¤logsrunning¦outputbodyً1×100 adjoint(::Vector{Float64}) with eltype Float64: 0.0 0.0634665 0.126933 0.1904 0.253866 … 6.09279 6.15625 6.21972 6.28319mimetext/plainrootassigneeinputlast_run_timestampA&o,Ӈwpersist_js_state·has_pluto_hook_features§cell_id$337bc837-174f-4ed6-bee1-c9e80dc03b52depends_on_disabled_cells§runtime͍published_object_keysdepends_on_skipped_cells§errored$9a141034-17cb-4d85-a5a2-4724a38dd269queued¤logsrunning¦outputbody#@visual_debug (macro with 1 method)mimetext/plainrootassigneelast_run_timestampA&o6ѸApersist_js_state·has_pluto_hook_features§cell_id$9a141034-17cb-4d85-a5a2-4724a38dd269depends_on_disabled_cells§runtime Ipublished_object_keysdepends_on_skipped_cells§errored$be28f764-a415-46ed-a503-68e8d831f8a8queued¤logsrunning¦outputbodyelements3.0=x text/plain2.5 text/plaintypeTupleobjectid16aee82358cd366cmime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o+1+persist_js_state·has_pluto_hook_features§cell_id$be28f764-a415-46ed-a503-68e8d831f8a8depends_on_disabled_cells§runtimeJյpublished_object_keysdepends_on_skipped_cells§errored$5e02b2c0-10cf-4746-840d-7017648f89f8queued¤logsrunning¦outputbody

Besides the mathematics of reverse mode differentiation, there are many interesting computer science issues regarding the implementation of reverse diff. Probably the jury is still out which is best.

The implementation below is based on Julia's dispatch system. This means we create a type Tracked for keeping track of our input variables and everything we'll need to calculate later.

The other main approach (not illustrated here) is source to source translation of code. In Julia the zygote package translates at the high level, while the increasingly popular Enzyme package translates at the lower level.

mimetext/htmlrootassigneelast_run_timestampA&n[}persist_js_state·has_pluto_hook_features§cell_id$5e02b2c0-10cf-4746-840d-7017648f89f8depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$a6c2d3a2-326a-41dd-864d-aa3662466222queued¤logsrunning¦outputbodyelements3=x text/plain5=y text/plaintypeTupleobjectid106d234ff087a5f0mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o,persist_js_state·has_pluto_hook_features§cell_id$a6c2d3a2-326a-41dd-864d-aa3662466222depends_on_disabled_cells§runtime(published_object_keysdepends_on_skipped_cells§errored$2c3f3103-9ccb-446e-a289-87bbd0a1b97bqueued¤logsrunning¦outputbody5=y mimetext/plainrootassigneelast_run_timestampA&o,46persist_js_state·has_pluto_hook_features§cell_id$2c3f3103-9ccb-446e-a289-87bbd0a1b97bdepends_on_disabled_cells§runtime͎Bpublished_object_keysdepends_on_skipped_cells§errored$e52aa672-69a9-419b-a992-e7a3d1364fb6queued¤logsrunning¦outputbodyprefix*Text{Main.var"workspace#3".Tracked{Int64}}elements[+] → 34text/plain[*] → 30text/plain[*] → 6text/plain2text/plain3=xtext/plain5=ytext/plain[^2] → 4text/plain[-] → 2text/plain 3=xtext/plain 1text/plaintypeArrayprefix_shortobjectidcd2b653b4334943dmime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o,Upersist_js_state·has_pluto_hook_features§cell_id$e52aa672-69a9-419b-a992-e7a3d1364fb6depends_on_disabled_cells§runtime.published_object_keysdepends_on_skipped_cells§errored$7299bfea-e23e-4993-bebd-9d8762427294queued¤logsrunning¦outputbodyٌ

Pretty Viz: Forward Pass + Reverse Mode Diff

mimetext/htmlrootassigneelast_run_timestampA&nWpersist_js_state·has_pluto_hook_features§cell_id$7299bfea-e23e-4993-bebd-9d8762427294depends_on_disabled_cells§runtime}published_object_keysdepends_on_skipped_cells§errored$4da7a594-aec5-4c8d-ab20-6913ed6ff8acqueued¤logsrunning¦outputbodyM mimetext/htmlrootassigneelast_run_timestampA&o+Xpersist_js_state·has_pluto_hook_features§cell_id$4da7a594-aec5-4c8d-ab20-6913ed6ff8acdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$4bfc2f7d-a5b0-44c7-8bb6-f1b834c1cc51queued¤logsrunning¦outputbody

Tracked is a tree – We just need to tell AbstractTrees.jl how to get the children for each node and we get tree printing and iteration over all nodes for free.

mimetext/htmlrootassigneelast_run_timestampA&nkpersist_js_state·has_pluto_hook_features§cell_id$4bfc2f7d-a5b0-44c7-8bb6-f1b834c1cc51depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$0d2a3187-46f1-4705-bb3e-ecf74e34840dqueued¤logsrunning¦outputbody[+] → 34 mimetext/plainrootassigneezlast_run_timestampA&o,25ʰpersist_js_state·has_pluto_hook_features§cell_id$0d2a3187-46f1-4705-bb3e-ecf74e34840ddepends_on_disabled_cells§runtimeͯApublished_object_keysdepends_on_skipped_cells§errored$437285d4-ec53-4bb7-9966-fcfb5352e205queued¤logsrunning¦outputbody+show_tree (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA&o6opersist_js_state·has_pluto_hook_features§cell_id$437285d4-ec53-4bb7-9966-fcfb5352e205depends_on_disabled_cells§runtimeupublished_object_keysdepends_on_skipped_cells§errored$61513f88-c126-43ab-9515-85a2c8a472bfqueued¤logsrunning¦outputbodyprefixMain.var"workspace#3".Trackedelements[^2] → 9 text/plain5=y text/plaintypeArrayprefix_shortobjectidcda9316f1c16c7c4mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o,3Qpersist_js_state·has_pluto_hook_features§cell_id$61513f88-c126-43ab-9515-85a2c8a472bfdepends_on_disabled_cells§runtime5ָpublished_object_keysdepends_on_skipped_cells§errored$73d638bf-30c1-4694-b3a8-4b29c5e3fa65queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o+ypersist_js_state·has_pluto_hook_features§cell_id$73d638bf-30c1-4694-b3a8-4b29c5e3fa65depends_on_disabled_cells§runtime"6published_object_keysdepends_on_skipped_cells§errored$ccafd0b9-95aa-4e58-8afc-26cd3ee61cc9queued¤logsrunning¦outputbody^

Straight away we get the primal result of our calculation:

mimetext/htmlrootassigneelast_run_timestampA&nlpersist_js_state·has_pluto_hook_features§cell_id$ccafd0b9-95aa-4e58-8afc-26cd3ee61cc9depends_on_disabled_cells§runtime$4published_object_keysdepends_on_skipped_cells§errored$95ab6c83-0757-40b3-a116-6d872a074f69queued¤logsrunning¦outputbody[*] → 15 mimetext/plainrootassigneelast_run_timestampA&o+|(persist_js_state·has_pluto_hook_features§cell_id$95ab6c83-0757-40b3-a116-6d872a074f69depends_on_disabled_cells§runtime+˵published_object_keysdepends_on_skipped_cells§errored$86aa821b-a373-4814-953e-535f3a33c002queued¤logsrunning¦outputbody
mimetext/htmlrootassigneelast_run_timestampA&o6'Ppersist_js_state·has_pluto_hook_features§cell_id$86aa821b-a373-4814-953e-535f3a33c002depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$29360c2a-c47b-48a3-b225-248eb3ec8a42queued¤logsrunning¦outputbody:*mimetext/plainrootassigneelast_run_timestampA&o+:persist_js_state·has_pluto_hook_features§cell_id$29360c2a-c47b-48a3-b225-248eb3ec8a42depends_on_disabled_cells§runtime9Lpublished_object_keysdepends_on_skipped_cells§errored$814786be-a760-4023-a614-83e727681075queued¤logsrunning¦outputbodyvTracked{Float64} val: Float64 2.5 name: Symbol ##256 df: Nothing nothing deps: Array{Tracked}((0,)) Tracked[] mimetext/plainrootassigneelast_run_timestampA&o+ܰ!persist_js_state·has_pluto_hook_features§cell_id$814786be-a760-4023-a614-83e727681075depends_on_disabled_cells§runtime2Ipublished_object_keysdepends_on_skipped_cells§errored$ac097299-0a31-474c-ab26-a4fb24bb9046queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,+Opersist_js_state·has_pluto_hook_features§cell_id$ac097299-0a31-474c-ab26-a4fb24bb9046depends_on_disabled_cells§runtimeRpublished_object_keysdepends_on_skipped_cells§errored$d9304d8d-9a34-46f2-908d-42d5cc5f5c5fqueued¤logsrunning¦outputbody12=👽 mimetext/plainrootassignee👽last_run_timestampA&o,wİpersist_js_state·has_pluto_hook_features§cell_id$d9304d8d-9a34-46f2-908d-42d5cc5f5c5fdepends_on_disabled_cells§runtimeLpublished_object_keysdepends_on_skipped_cells§errored$788903f0-a372-4265-bbf3-5b507a705100queued¤logsrunning¦outputbody#NN (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA&o,persist_js_state·has_pluto_hook_features§cell_id$788903f0-a372-4265-bbf3-5b507a705100depends_on_disabled_cells§runtime"\ published_object_keysdepends_on_skipped_cells§errored$bb75e790-a628-4821-b7b1-11cd34a73d38queued¤logsrunning¦outputbodyprefixMain.var"workspace#3".Trackedelements3=x text/plain5=y text/plaintypeArrayprefix_shortobjectid5649d4636ab275admime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o,%Epersist_js_state·has_pluto_hook_features§cell_id$bb75e790-a628-4821-b7b1-11cd34a73d38depends_on_disabled_cells§runtimeApublished_object_keysdepends_on_skipped_cells§errored$094bf7ee-7c9b-458c-98dc-521768831654queued¤logsrunning¦outputbody+show_steps (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA&o6.j@persist_js_state·has_pluto_hook_features§cell_id$094bf7ee-7c9b-458c-98dc-521768831654depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44queued¤logsrunning¦outputbody@t (macro with 1 method)mimetext/plainrootassigneelast_run_timestampA&o,鴰persist_js_state·has_pluto_hook_features§cell_id$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44depends_on_disabled_cells§runtime}published_object_keysdepends_on_skipped_cells§errored$2141849b-675e-406c-8df4-34b2706507afqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,7persist_js_state·has_pluto_hook_features§cell_id$2141849b-675e-406c-8df4-34b2706507afdepends_on_disabled_cells§runtimeB6published_object_keysdepends_on_skipped_cells§errored$0911a08a-4290-455b-9b26-0bf2862296daqueued¤logsrunning¦outputbodyGd mimeimage/svg+xmlrootassigneelast_run_timestampA&o/Gpersist_js_state·has_pluto_hook_features§cell_id$0911a08a-4290-455b-9b26-0bf2862296dadepends_on_disabled_cells§runtimekpublished_object_keysdepends_on_skipped_cells§errored$20571e1a-9687-4d42-98b3-b3bc0b207b3bqueued¤logsrunning¦outputbodyX

Implementation in Code

mimetext/htmlrootassigneelast_run_timestampA&nZ%persist_js_state·has_pluto_hook_features§cell_id$20571e1a-9687-4d42-98b3-b3bc0b207b3bdepends_on_disabled_cells§runtimekpublished_object_keysdepends_on_skipped_cells§errored$01eacbd4-ef37-4524-aecc-2ef9a1044cf8queued¤logsrunning¦outputbodyٜ

To also get the gradient, we'll use PreOrderDFS to traverse the tree we just created from the top down.

mimetext/htmlrootassigneelast_run_timestampA&nl5vpersist_js_state·has_pluto_hook_features§cell_id$01eacbd4-ef37-4524-aecc-2ef9a1044cf8depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$99a3507b-ca03-429f-acde-e2d1ebb32054queued¤logsrunning¦outputbody%grad (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA&o,`persist_js_state·has_pluto_hook_features§cell_id$99a3507b-ca03-429f-acde-e2d1ebb32054depends_on_disabled_cells§runtimexpublished_object_keysdepends_on_skipped_cells§errored$7fcccb65-4a7b-4527-97be-d25f481f6eafqueued¤logsrunning¦outputbodyO

We can verify that it does the right thing:

mimetext/htmlrootassigneelast_run_timestampA&nli۰persist_js_state·has_pluto_hook_features§cell_id$7fcccb65-4a7b-4527-97be-d25f481f6eafdepends_on_disabled_cells§runtimexDpublished_object_keysdepends_on_skipped_cells§errored$6733267e-1251-49a7-86a0-ff45ba408205queued¤logsrunning¦outputbodyP

Tracking variables

mimetext/htmlrootassigneelast_run_timestampA&nWpersist_js_state·has_pluto_hook_features§cell_id$6733267e-1251-49a7-86a0-ff45ba408205depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$266bf2ed-d625-47be-8a87-21e12052b27fqueued¤logsrunning¦outputbody٘

Let's use three dense layers, all of the form

$$x \mapsto \tanh.(W \cdot x + b)$$

mimetext/htmlrootassigneelast_run_timestampA&nlpersist_js_state·has_pluto_hook_features§cell_id$266bf2ed-d625-47be-8a87-21e12052b27fdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$ea94bdf9-b0b1-42ad-ba44-aaba73ed1192queued¤logsrunning¦outputbody|

Let's look at our example from above, $z = x^2+y$. We'll introduce an intermediate variable $u = x^2$ so we can write $z$ as $u + y$.

We will first start with $1 \cdot dz$. The one here represents the fact that $\frac{\partial z}{\partial z} = 1$.

We then want to pull that back until we get something in terms of $dx$ and $dy$, so we then get the derivatives wrt $x$ and $y$

The pullback map for $+$ will first map $dz$ to $d(u + y) = du + dy$ and then we can use the pullback map for ^ to expand $du$ to $2x \cdot dx$.

Using that recursive method we end up with $dz = 2x \cdot dx + dy$, which is exactly what we'd get if we did this by hand. This tells us that $\frac{\partial z}{\partial x} = 2x$ and $\frac{\partial z}{\partial y} = y$.

mimetext/htmlrootassigneelast_run_timestampA&nk۰persist_js_state·has_pluto_hook_features§cell_id$ea94bdf9-b0b1-42ad-ba44-aaba73ed1192depends_on_disabled_cells§runtime Kϵpublished_object_keysdepends_on_skipped_cells§errored$98807f0d-f8a5-4fe0-9983-2b95290347d9queued¤logsrunning¦outputbodyه

Tada! 🎉

(It's even reasonably fast for neural nets)

mimetext/htmlrootassigneelast_run_timestampA&nm-persist_js_state·has_pluto_hook_features§cell_id$98807f0d-f8a5-4fe0-9983-2b95290347d9depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$8b486bd8-25a1-4895-99c8-0541c9e3c8b0queued¤logsrunning¦outputbody3
Move me!
1 mimetext/htmlrootassigneelast_run_timestampA&o6persist_js_state·has_pluto_hook_features§cell_id$8b486bd8-25a1-4895-99c8-0541c9e3c8b0depends_on_disabled_cells§runtime ȵpublished_object_keysdepends_on_skipped_cells§errored$bcff2aa8-2387-44c7-a28f-39cd505a7adfqueued¤logsrunning¦outputbodyq

We can also visualize what Julia does in the forward pass on the code itself:

mimetext/htmlrootassigneelast_run_timestampA&nljpersist_js_state·has_pluto_hook_features§cell_id$bcff2aa8-2387-44c7-a28f-39cd505a7adfdepends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$b47754ac-cb17-4ea3-a97d-20371fd341a5queued¤logsrunning¦outputbodyelements5text/plain3text/plaintypeTupleobjectidf8402269077b4228mime!application/vnd.pluto.tree+objectrootassigneelast_run_timestampA&o+*persist_js_state·has_pluto_hook_features§cell_id$b47754ac-cb17-4ea3-a97d-20371fd341a5depends_on_disabled_cells§runtimeBpublished_object_keysdepends_on_skipped_cells§errored$520f280d-c78e-433b-a0a2-8ef05b04f7ccqueued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o,-persist_js_state·has_pluto_hook_features§cell_id$520f280d-c78e-433b-a0a2-8ef05b04f7ccdepends_on_disabled_cells§runtime'dpublished_object_keysdepends_on_skipped_cells§errored$faf69c6f-b179-4ca2-aa22-641e9560e17bqueued¤logsrunning¦outputbody@ad_steps (macro with 1 method)mimetext/plainrootassigneelast_run_timestampA&o6:Gpersist_js_state·has_pluto_hook_features§cell_id$faf69c6f-b179-4ca2-aa22-641e9560e17bdepends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored$d82adc20-4c8c-4f2c-9839-d03ad7e7f581queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o/ðpersist_js_state·has_pluto_hook_features§cell_id$d82adc20-4c8c-4f2c-9839-d03ad7e7f581depends_on_disabled_cells§runtimeRxpublished_object_keysdepends_on_skipped_cells§errored$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96queued¤logsrunning¦outputbodyH[+] → 18 ├─ [*] → 15 │ ├─ 3=x │ └─ 5=y └─ 3=x mimetext/plainrootassigneewlast_run_timestampA&o,hypersist_js_state·has_pluto_hook_features§cell_id$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96depends_on_disabled_cells§runtime<1published_object_keysdepends_on_skipped_cells§errored$8919faec-753d-41d0-b0a8-c84c025b2919queued¤logsrunning¦outputbody3.0=x mimetext/plainrootassigneexfloatlast_run_timestampA&o+3persist_js_state·has_pluto_hook_features§cell_id$8919faec-753d-41d0-b0a8-c84c025b2919depends_on_disabled_cells§runtime@$published_object_keysdepends_on_skipped_cells§errored$0d006e4d-72e3-42ab-beb3-e5304a4d9ba9queued¤logsrunning¦outputbody3=x mimetext/plainrootassigneelast_run_timestampA&o+persist_js_state·has_pluto_hook_features§cell_id$0d006e4d-72e3-42ab-beb3-e5304a4d9ba9depends_on_disabled_cells§runtime$published_object_keysdepends_on_skipped_cells§errored$4813e57f-557a-4179-afd2-7925687b3d35queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&njpersist_js_state·has_pluto_hook_features§cell_id$4813e57f-557a-4179-afd2-7925687b3d35depends_on_disabled_cells§runtime#published_object_keysdepends_on_skipped_cells§errored$86fa378b-815d-4c3d-9121-1338ee54f30fqueued¤logslinemsgSaved animation to text/plaincell_id$86fa378b-815d-4c3d-9121-1338ee54f30fkwargsfn"/tmp/jl_VutevqHCBC.gif"text/plainidPlots_c6bd189bfile9/home/runner/.julia/packages/Plots/lW9ll/src/animation.jlgroupanimationlevelInforunning¦outputbody͜mimetext/htmlrootassigneelast_run_timestampA&o5˰persist_js_state·has_pluto_hook_features§cell_id$86fa378b-815d-4c3d-9121-1338ee54f30fdepends_on_disabled_cells§runtime߯Ppublished_object_keysdepends_on_skipped_cells§errored$6b1fb808-e993-4c2b-b81b-6710f8206de7queued¤logsrunning¦outputbody(to_json (generic function with 1 method)mimetext/plainrootassigneelast_run_timestampA&o6 ,Zpersist_js_state·has_pluto_hook_features§cell_id$6b1fb808-e993-4c2b-b81b-6710f8206de7depends_on_disabled_cells§runtimeɵpublished_object_keysdepends_on_skipped_cells§errored$8110f306-a7bb-43a2-bb36-6182c59b4b2equeued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o5dpersist_js_state·has_pluto_hook_features§cell_id$8110f306-a7bb-43a2-bb36-6182c59b4b2edepends_on_disabled_cells§runtime>published_object_keysdepends_on_skipped_cells§errored$99b6ab91-a022-449c-988c-0e5c5719c910queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o+persist_js_state·has_pluto_hook_features§cell_id$99b6ab91-a022-449c-988c-0e5c5719c910depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$db3389c0-cb00-404c-81a8-8c4ccdab22f4queued¤logsrunning¦outputbody
Move me!
1 mimetext/htmlrootassigneelast_run_timestampA&o6踰persist_js_state·has_pluto_hook_features§cell_id$db3389c0-cb00-404c-81a8-8c4ccdab22f4depends_on_disabled_cells§runtime!Kpublished_object_keysdepends_on_skipped_cells§errored$b0cc4665-eb45-48ea-9a33-5acf56d2a283queued¤logsrunning¦outputbodymimetext/plainrootassigneelast_run_timestampA&o+3persist_js_state·has_pluto_hook_features§cell_id$b0cc4665-eb45-48ea-9a33-5acf56d2a283depends_on_disabled_cells§runtimepublished_object_keysdepends_on_skipped_cells§errored$d4e9b202-242e-4420-986b-12d2ab57af93queued¤logsrunning¦outputbody&grad (generic function with 2 methods)mimetext/plainrootassigneelast_run_timestampA&o,`Mpersist_js_state·has_pluto_hook_features§cell_id$d4e9b202-242e-4420-986b-12d2ab57af93depends_on_disabled_cells§runtime published_object_keysdepends_on_skipped_cells§errored±cell_dependencies_$1b768306-0e0e-414d-a501-5c62a2db720fprecedence_heuristic cell_id$1b768306-0e0e-414d-a501-5c62a2db720fdownstream_cells_mapupstream_cells_map@md_strgetindex$aa99b123-7164-4ed7-833f-b99b19118a0dprecedence_heuristic cell_id$aa99b123-7164-4ed7-833f-b99b19118a0ddownstream_cells_mapupstream_cells_map^x$0b5e6560-81fd-4182-bba5-aca702fb3048+y$0b5e6560-81fd-4182-bba5-aca702fb3048$be693f92-de60-4b43-82b3-ee6cceea310aprecedence_heuristic cell_id$be693f92-de60-4b43-82b3-ee6cceea310adownstream_cells_mapupstream_cells_map@md_strgetindex$1f1b384a-6588-45a5-9dd3-6de3face8bfbprecedence_heuristic cell_id$1f1b384a-6588-45a5-9dd3-6de3face8bfbdownstream_cells_mapad_steps$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303upstream_cells_map"Dictpushfirst!!Tracked$99b6ab91-a022-449c-988c-0e5c5719c910zipEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581@htlExprnothingcopy HypertextLiteral.attribute_valuesprintHypertextLiteral.ResultisemptyevalchildrenIteratorsaccumulateHypertextLiteral.BypassPreOrderDFShaskeyHypertextLiteral.content=>AnyPostOrderDFSendgetisapush!===showHypertextLiteral$d5559dba-9fe4-11ec-3744-ebd1408e7dc4+Iterators.filter$e55dfad2-db50-459f-ab54-fa7637fc3638precedence_heuristic cell_id$e55dfad2-db50-459f-ab54-fa7637fc3638downstream_cells_mapupstream_cells_map@md_strgetindex$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bprecedence_heuristic cell_id$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bdownstream_cells_mapupstream_cells_mapDumpaa$be28f764-a415-46ed-a503-68e8d831f8a8$7dca2c78-6faf-450a-8096-1c5b9f7f1295precedence_heuristic cell_id$7dca2c78-6faf-450a-8096-1c5b9f7f1295downstream_cells_mapleaky_relu$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9$0911a08a-4290-455b-9b26-0bf2862296da$86fa378b-815d-4c3d-9121-1338ee54f30fupstream_cells_map<>isless*$13487e65-5e48-4a37-9bea-f262dd7b6d56precedence_heuristic cell_id$13487e65-5e48-4a37-9bea-f262dd7b6d56downstream_cells_mapBase.+upstream_cells_mapBaseTracked$99b6ab91-a022-449c-988c-0e5c5719c910$96286b65-1a22-4458-a399-46579248cce4precedence_heuristic cell_id$96286b65-1a22-4458-a399-46579248cce4downstream_cells_mapW3W2b3##anim#381W1b2params$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209_randpb1upstream_cells_map$randn>sqrt√islessPlots.frameNN$788903f0-a372-4265-bbf3-5b507a705100Tracked$99b6ab91-a022-449c-988c-0e5c5719c910adjointgrad$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93PlutoTest.ComputedPlutoTest.stacktrace-+PlutoTest.isaPlutoTest.push!#PlutoTest.PartialEvaluatedExceptionPlutoTest.catch_backtraceCore._expry$0b5e6560-81fd-4182-bba5-aca702fb3048$e450329b-46ee-4f53-b6f0-891d6596938cprecedence_heuristic cell_id$e450329b-46ee-4f53-b6f0-891d6596938cdownstream_cells_mapupstream_cells_map^x$0b5e6560-81fd-4182-bba5-aca702fb3048+y$0b5e6560-81fd-4182-bba5-aca702fb3048$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7precedence_heuristic cell_id$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7downstream_cells_mapupstream_cells_mapw$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96x$0b5e6560-81fd-4182-bba5-aca702fb3048grad$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93y$0b5e6560-81fd-4182-bba5-aca702fb3048$d1312080-272c-4bd5-bcfe-bf47f8320961precedence_heuristic cell_id$d1312080-272c-4bd5-bcfe-bf47f8320961downstream_cells_mapupstream_cells_map@md_strgetindex$885cd51d-895f-4996-a23b-780498b5b810precedence_heuristic cell_id$885cd51d-895f-4996-a23b-780498b5b810downstream_cells_mapupstream_cells_map@md_strgetindex$d5559dba-9fe4-11ec-3744-ebd1408e7dc4precedence_heuristiccell_id$d5559dba-9fe4-11ec-3744-ebd1408e7dc4downstream_cells_mapAbstractTrees$2188a663-5a85-4ce4-bc8d-20383481e59b$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$8110f306-a7bb-43a2-bb36-6182c59b4b2e$6b1fb808-e993-4c2b-b81b-6710f8206de7LinearAlgebra$e01114fa-a847-424a-931f-8f42e623109fPlutoUIPlots$96286b65-1a22-4458-a399-46579248cce4$86fa378b-815d-4c3d-9121-1338ee54f30fHypertextLiteral$1f1b384a-6588-45a5-9dd3-6de3face8bfb$8110f306-a7bb-43a2-bb36-6182c59b4b2e$1a154bb7-93a3-4973-8908-788db77ac294$437285d4-ec53-4bb7-9966-fcfb5352e205$094bf7ee-7c9b-458c-98dc-521768831654PlutoTest$79f71f9d-b491-4a2c-85a4-29ae8da4f312upstream_cells_map$49333dc7-b809-4bb1-b285-20a12d8b9209precedence_heuristic cell_id$49333dc7-b809-4bb1-b285-20a12d8b9209downstream_cells_mapupstream_cells_map@t$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44zero:Tracked$99b6ab91-a022-449c-988c-0e5c5719c910typeofnorminput$337bc837-174f-4ed6-bee1-c9e80dc03b52adjointendzipparams$96286b65-1a22-4458-a399-46579248cce4grad$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93ŷ$d2d21d41-9a4e-428b-b696-6878901849baNN$788903f0-a372-4265-bbf3-5b507a705100-deepcopyplot*quiver!conj$f6ce8448-d9ce-4453-9e47-dc6443d50f55precedence_heuristic cell_id$f6ce8448-d9ce-4453-9e47-dc6443d50f55downstream_cells_maps1$9a141034-17cb-4d85-a5a2-4724a38dd269upstream_cells_mapBaseBase.Docs.HTML@html_str$ec723b90-1c8b-4a87-8a86-fe17d6e85f22precedence_heuristic cell_id$ec723b90-1c8b-4a87-8a86-fe17d6e85f22downstream_cells_map#478#steps#477#exupstream_cells_mapTTREE$8110f306-a7bb-43a2-bb36-6182c59b4b2ead_steps$1f1b384a-6588-45a5-9dd3-6de3face8bfbshow_steps$094bf7ee-7c9b-458c-98dc-521768831654@ad_steps$faf69c6f-b179-4ca2-aa22-641e9560e17bEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$c9d70049-4793-41db-9075-e9e73e926c1aprecedence_heuristic cell_id$c9d70049-4793-41db-9075-e9e73e926c1adownstream_cells_mapupstream_cells_mapx$0b5e6560-81fd-4182-bba5-aca702fb3048+y$0b5e6560-81fd-4182-bba5-aca702fb3048$17f9e2c0-aa70-4d4e-ba42-51782df43785precedence_heuristic cell_id$17f9e2c0-aa70-4d4e-ba42-51782df43785downstream_cells_mapupstream_cells_map@md_strgetindex$0d186a1b-eb89-4be8-a237-e5a9608f4a30precedence_heuristic cell_id$0d186a1b-eb89-4be8-a237-e5a9608f4a30downstream_cells_map#468#ex#469#stepsupstream_cells_mapTTREE$8110f306-a7bb-43a2-bb36-6182c59b4b2ead_steps$1f1b384a-6588-45a5-9dd3-6de3face8bfbshow_steps$094bf7ee-7c9b-458c-98dc-521768831654@ad_steps$faf69c6f-b179-4ca2-aa22-641e9560e17bEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$d2d21d41-9a4e-428b-b696-6878901849baprecedence_heuristic cell_id$d2d21d41-9a4e-428b-b696-6878901849badownstream_cells_mapŷ$96286b65-1a22-4458-a399-46579248cce4$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209$86fa378b-815d-4c3d-9121-1338ee54f30fupstream_cells_mapinput$337bc837-174f-4ed6-bee1-c9e80dc03b52sin$0b094198-cf44-41d7-a8dc-fd8fd0716bb4precedence_heuristic cell_id$0b094198-cf44-41d7-a8dc-fd8fd0716bb4downstream_cells_mapupstream_cells_map@md_strgetindex$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9precedence_heuristic cell_id$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9downstream_cells_mapBase.broadcastedupstream_cells_mapTracked$99b6ab91-a022-449c-988c-0e5c5719c910typeofleaky_relu$7dca2c78-6faf-450a-8096-1c5b9f7f1295*isless$506d408e-dc2b-4e12-b917-286e3f4079a2precedence_heuristic cell_id$506d408e-dc2b-4e12-b917-286e3f4079a2downstream_cells_mapupstream_cells_mapw$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96grad$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303precedence_heuristic cell_id$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303downstream_cells_map#465#ex#466#stepsupstream_cells_mapTTREE$8110f306-a7bb-43a2-bb36-6182c59b4b2ead_steps$1f1b384a-6588-45a5-9dd3-6de3face8bfbshow_steps$094bf7ee-7c9b-458c-98dc-521768831654@ad_steps$faf69c6f-b179-4ca2-aa22-641e9560e17bEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$0b5e6560-81fd-4182-bba5-aca702fb3048precedence_heuristic cell_id$0b5e6560-81fd-4182-bba5-aca702fb3048downstream_cells_mapx$c9d70049-4793-41db-9075-e9e73e926c1a$aa99b123-7164-4ed7-833f-b99b19118a0d$0d2a3187-46f1-4705-bb3e-ecf74e34840d$61513f88-c126-43ab-9515-85a2c8a472bf$2c3f3103-9ccb-446e-a289-87bbd0a1b97b$795c510e-6280-42d9-8820-290ca07be88d$e450329b-46ee-4f53-b6f0-891d6596938c$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0$95ab6c83-0757-40b3-a116-6d872a074f69$db4f871a-3ca1-4bd3-a2c5-da7d6f599281$29360c2a-c47b-48a3-b225-248eb3ec8a42$b47754ac-cb17-4ea3-a97d-20371fd341a5$bb75e790-a628-4821-b7b1-11cd34a73d38$0d006e4d-72e3-42ab-beb3-e5304a4d9ba9$8919faec-753d-41d0-b0a8-c84c025b2919$be28f764-a415-46ed-a503-68e8d831f8a8$81eb8a2d-a3a9-45af-a5a5-b96aefd48712$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7$a6c2d3a2-326a-41dd-864d-aa3662466222$79f71f9d-b491-4a2c-85a4-29ae8da4f312uy$c9d70049-4793-41db-9075-e9e73e926c1a$aa99b123-7164-4ed7-833f-b99b19118a0d$0d2a3187-46f1-4705-bb3e-ecf74e34840d$61513f88-c126-43ab-9515-85a2c8a472bf$2c3f3103-9ccb-446e-a289-87bbd0a1b97b$795c510e-6280-42d9-8820-290ca07be88d$e450329b-46ee-4f53-b6f0-891d6596938c$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0$95ab6c83-0757-40b3-a116-6d872a074f69$db4f871a-3ca1-4bd3-a2c5-da7d6f599281$29360c2a-c47b-48a3-b225-248eb3ec8a42$b47754ac-cb17-4ea3-a97d-20371fd341a5$bb75e790-a628-4821-b7b1-11cd34a73d38$81eb8a2d-a3a9-45af-a5a5-b96aefd48712$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7$a6c2d3a2-326a-41dd-864d-aa3662466222$79f71f9d-b491-4a2c-85a4-29ae8da4f312upstream_cells_mapTracked$99b6ab91-a022-449c-988c-0e5c5719c910Int$8a70e835-dae6-4727-8204-95c87d5c23daprecedence_heuristic cell_id$8a70e835-dae6-4727-8204-95c87d5c23dadownstream_cells_mapupstream_cells_map@md_strgetindex$337bc837-174f-4ed6-bee1-c9e80dc03b52precedence_heuristic cell_id$337bc837-174f-4ed6-bee1-c9e80dc03b52downstream_cells_mapinput$d2d21d41-9a4e-428b-b696-6878901849ba$96286b65-1a22-4458-a399-46579248cce4$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209$86fa378b-815d-4c3d-9121-1338ee54f30fupstream_cells_maprangecollectπ*adjointconj$9a141034-17cb-4d85-a5a2-4724a38dd269precedence_heuristic cell_id$9a141034-17cb-4d85-a5a2-4724a38dd269downstream_cells_map@visual_debug$79f71f9d-b491-4a2c-85a4-29ae8da4f312upstream_cells_maps1$f6ce8448-d9ce-4453-9e47-dc6443d50f55esc$be28f764-a415-46ed-a503-68e8d831f8a8precedence_heuristic cell_id$be28f764-a415-46ed-a503-68e8d831f8a8downstream_cells_mapbb$814786be-a760-4023-a614-83e727681075aa$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bupstream_cells_mappromotex$0b5e6560-81fd-4182-bba5-aca702fb3048$5e02b2c0-10cf-4746-840d-7017648f89f8precedence_heuristic cell_id$5e02b2c0-10cf-4746-840d-7017648f89f8downstream_cells_mapupstream_cells_map@md_strgetindex$a6c2d3a2-326a-41dd-864d-aa3662466222precedence_heuristic cell_id$a6c2d3a2-326a-41dd-864d-aa3662466222downstream_cells_mapupstream_cells_mapx$0b5e6560-81fd-4182-bba5-aca702fb3048y$0b5e6560-81fd-4182-bba5-aca702fb3048$2c3f3103-9ccb-446e-a289-87bbd0a1b97bprecedence_heuristic cell_id$2c3f3103-9ccb-446e-a289-87bbd0a1b97bdownstream_cells_mapupstream_cells_map^x$0b5e6560-81fd-4182-bba5-aca702fb3048+y$0b5e6560-81fd-4182-bba5-aca702fb3048$e52aa672-69a9-419b-a992-e7a3d1364fb6precedence_heuristic cell_id$e52aa672-69a9-419b-a992-e7a3d1364fb6downstream_cells_mapupstream_cells_mapcollectPreOrderDFSz$0d2a3187-46f1-4705-bb3e-ecf74e34840dText$7299bfea-e23e-4993-bebd-9d8762427294precedence_heuristic cell_id$7299bfea-e23e-4993-bebd-9d8762427294downstream_cells_mapupstream_cells_map@md_strgetindex$4da7a594-aec5-4c8d-ab20-6913ed6ff8acprecedence_heuristic cell_id$4da7a594-aec5-4c8d-ab20-6913ed6ff8acdownstream_cells_mapupstream_cells_mapTableOfContents$4bfc2f7d-a5b0-44c7-8bb6-f1b834c1cc51precedence_heuristic cell_id$4bfc2f7d-a5b0-44c7-8bb6-f1b834c1cc51downstream_cells_mapupstream_cells_map@md_strgetindex$0d2a3187-46f1-4705-bb3e-ecf74e34840dprecedence_heuristic cell_id$0d2a3187-46f1-4705-bb3e-ecf74e34840ddownstream_cells_mapz$e52aa672-69a9-419b-a992-e7a3d1364fb6upstream_cells_map-^x$0b5e6560-81fd-4182-bba5-aca702fb3048+*y$0b5e6560-81fd-4182-bba5-aca702fb3048$437285d4-ec53-4bb7-9966-fcfb5352e205precedence_heuristic cell_id$437285d4-ec53-4bb7-9966-fcfb5352e205downstream_cells_mapshow_tree$86aa821b-a373-4814-953e-535f3a33c002upstream_cells_mapgensym HypertextLiteral.attribute_valueto_json$6b1fb808-e993-4c2b-b81b-6710f8206de7HypertextLiteral.ResultHypertextLiteral$d5559dba-9fe4-11ec-3744-ebd1408e7dc4HypertextLiteral.Bypasss2$1a154bb7-93a3-4973-8908-788db77ac294HypertextLiteral.contentHypertextLiteral.ScriptTag@htl$61513f88-c126-43ab-9515-85a2c8a472bfprecedence_heuristic cell_id$61513f88-c126-43ab-9515-85a2c8a472bfdownstream_cells_mapupstream_cells_map^x$0b5e6560-81fd-4182-bba5-aca702fb3048+y$0b5e6560-81fd-4182-bba5-aca702fb3048$73d638bf-30c1-4694-b3a8-4b29c5e3fa65precedence_heuristic cell_id$73d638bf-30c1-4694-b3a8-4b29c5e3fa65downstream_cells_mapBase.*upstream_cells_mapBaseTracked$99b6ab91-a022-449c-988c-0e5c5719c910adjointconj$ccafd0b9-95aa-4e58-8afc-26cd3ee61cc9precedence_heuristic cell_id$ccafd0b9-95aa-4e58-8afc-26cd3ee61cc9downstream_cells_mapupstream_cells_map@md_strgetindex$95ab6c83-0757-40b3-a116-6d872a074f69precedence_heuristic cell_id$95ab6c83-0757-40b3-a116-6d872a074f69downstream_cells_mapupstream_cells_mapx$0b5e6560-81fd-4182-bba5-aca702fb3048*y$0b5e6560-81fd-4182-bba5-aca702fb3048$86aa821b-a373-4814-953e-535f3a33c002precedence_heuristic cell_id$86aa821b-a373-4814-953e-535f3a33c002downstream_cells_mapupstream_cells_map@t$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44show_tree$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$437285d4-ec53-4bb7-9966-fcfb5352e205-Tracked$99b6ab91-a022-449c-988c-0e5c5719c910typeofparams$96286b65-1a22-4458-a399-46579248cce4norminput$337bc837-174f-4ed6-bee1-c9e80dc03b52ŷ$d2d21d41-9a4e-428b-b696-6878901849baNN$788903f0-a372-4265-bbf3-5b507a705100$29360c2a-c47b-48a3-b225-248eb3ec8a42precedence_heuristic cell_id$29360c2a-c47b-48a3-b225-248eb3ec8a42downstream_cells_mapupstream_cells_mapx$0b5e6560-81fd-4182-bba5-aca702fb3048*y$0b5e6560-81fd-4182-bba5-aca702fb3048$814786be-a760-4023-a614-83e727681075precedence_heuristic cell_id$814786be-a760-4023-a614-83e727681075downstream_cells_mapupstream_cells_mapbb$be28f764-a415-46ed-a503-68e8d831f8a8Dump$ac097299-0a31-474c-ab26-a4fb24bb9046precedence_heuristic cell_id$ac097299-0a31-474c-ab26-a4fb24bb9046downstream_cells_mapBase.^upstream_cells_mapBase-Tracked$99b6ab91-a022-449c-988c-0e5c5719c910Int*Symbol$d9304d8d-9a34-46f2-908d-42d5cc5f5c5fprecedence_heuristic cell_id$d9304d8d-9a34-46f2-908d-42d5cc5f5c5fdownstream_cells_map👽upstream_cells_mapTracked$99b6ab91-a022-449c-988c-0e5c5719c910Int$788903f0-a372-4265-bbf3-5b507a705100precedence_heuristic cell_id$788903f0-a372-4265-bbf3-5b507a705100downstream_cells_mapNN$96286b65-1a22-4458-a399-46579248cce4$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209upstream_cells_map@t$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44Tracked$99b6ab91-a022-449c-988c-0e5c5719c910typeof+*tanh$bb75e790-a628-4821-b7b1-11cd34a73d38precedence_heuristic cell_id$bb75e790-a628-4821-b7b1-11cd34a73d38downstream_cells_mapupstream_cells_mapx$0b5e6560-81fd-4182-bba5-aca702fb3048*y$0b5e6560-81fd-4182-bba5-aca702fb3048$094bf7ee-7c9b-458c-98dc-521768831654precedence_heuristic cell_id$094bf7ee-7c9b-458c-98dc-521768831654downstream_cells_mapshow_steps$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303upstream_cells_mapHypertextLiteral.Bypasss2$1a154bb7-93a3-4973-8908-788db77ac294HypertextLiteral.contentHypertextLiteral.ScriptTag@htlndigitslengthHypertextLiteral.attribute_pair HypertextLiteral.attribute_value-gensymto_json$6b1fb808-e993-4c2b-b81b-6710f8206de7HypertextLiteral.ResultHypertextLiteral$d5559dba-9fe4-11ec-3744-ebd1408e7dc4$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44precedence_heuristic cell_id$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44downstream_cells_map@t$788903f0-a372-4265-bbf3-5b507a705100$96286b65-1a22-4458-a399-46579248cce4$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209$86fa378b-815d-4c3d-9121-1338ee54f30fupstream_cells_mapQuoteNodeesc$2141849b-675e-406c-8df4-34b2706507afprecedence_heuristic cell_id$2141849b-675e-406c-8df4-34b2706507afdownstream_cells_mapBase./upstream_cells_mapBase-^Tracked$99b6ab91-a022-449c-988c-0e5c5719c910*$0911a08a-4290-455b-9b26-0bf2862296daprecedence_heuristic cell_id$0911a08a-4290-455b-9b26-0bf2862296dadownstream_cells_mapupstream_cells_mapplotleaky_relu$7dca2c78-6faf-450a-8096-1c5b9f7f1295$20571e1a-9687-4d42-98b3-b3bc0b207b3bprecedence_heuristic cell_id$20571e1a-9687-4d42-98b3-b3bc0b207b3bdownstream_cells_mapupstream_cells_map@md_strgetindex$01eacbd4-ef37-4524-aecc-2ef9a1044cf8precedence_heuristic cell_id$01eacbd4-ef37-4524-aecc-2ef9a1044cf8downstream_cells_mapupstream_cells_map@md_strgetindex$99a3507b-ca03-429f-acde-e2d1ebb32054precedence_heuristic cell_id$99a3507b-ca03-429f-acde-e2d1ebb32054downstream_cells_mapgrad$506d408e-dc2b-4e12-b917-286e3f4079a2$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7$96286b65-1a22-4458-a399-46579248cce4$49333dc7-b809-4bb1-b285-20a12d8b9209$86fa378b-815d-4c3d-9121-1338ee54f30fupstream_cells_mapDict===Tracked$99b6ab91-a022-449c-988c-0e5c5719c910+PreOrderDFSzip=>getAnynothing$7fcccb65-4a7b-4527-97be-d25f481f6eafprecedence_heuristic cell_id$7fcccb65-4a7b-4527-97be-d25f481f6eafdownstream_cells_mapupstream_cells_map@md_strgetindex$6733267e-1251-49a7-86a0-ff45ba408205precedence_heuristic cell_id$6733267e-1251-49a7-86a0-ff45ba408205downstream_cells_mapupstream_cells_map@md_strgetindex$266bf2ed-d625-47be-8a87-21e12052b27fprecedence_heuristic cell_id$266bf2ed-d625-47be-8a87-21e12052b27fdownstream_cells_mapupstream_cells_map@md_strgetindex$ea94bdf9-b0b1-42ad-ba44-aaba73ed1192precedence_heuristic cell_id$ea94bdf9-b0b1-42ad-ba44-aaba73ed1192downstream_cells_mapupstream_cells_map@md_strgetindex$98807f0d-f8a5-4fe0-9983-2b95290347d9precedence_heuristic cell_id$98807f0d-f8a5-4fe0-9983-2b95290347d9downstream_cells_mapupstream_cells_map@md_strgetindex$8b486bd8-25a1-4895-99c8-0541c9e3c8b0precedence_heuristic cell_id$8b486bd8-25a1-4895-99c8-0541c9e3c8b0downstream_cells_map#480#ex#481#stepsupstream_cells_mapTTREE$8110f306-a7bb-43a2-bb36-6182c59b4b2ead_steps$1f1b384a-6588-45a5-9dd3-6de3face8bfbshow_steps$094bf7ee-7c9b-458c-98dc-521768831654@ad_steps$faf69c6f-b179-4ca2-aa22-641e9560e17bEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$bcff2aa8-2387-44c7-a28f-39cd505a7adfprecedence_heuristic cell_id$bcff2aa8-2387-44c7-a28f-39cd505a7adfdownstream_cells_mapupstream_cells_map@md_strgetindex$b47754ac-cb17-4ea3-a97d-20371fd341a5precedence_heuristic cell_id$b47754ac-cb17-4ea3-a97d-20371fd341a5downstream_cells_mapupstream_cells_mapx$0b5e6560-81fd-4182-bba5-aca702fb3048*y$0b5e6560-81fd-4182-bba5-aca702fb3048$520f280d-c78e-433b-a0a2-8ef05b04f7ccprecedence_heuristic cell_id$520f280d-c78e-433b-a0a2-8ef05b04f7ccdownstream_cells_mapBase.broadcastedupstream_cells_mapBase-^Tracked$99b6ab91-a022-449c-988c-0e5c5719c910typeof*tanh$faf69c6f-b179-4ca2-aa22-641e9560e17bprecedence_heuristic cell_id$faf69c6f-b179-4ca2-aa22-641e9560e17bdownstream_cells_map@ad_steps$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303upstream_cells_mapQuoteNode$d82adc20-4c8c-4f2c-9839-d03ad7e7f581precedence_heuristic cell_id$d82adc20-4c8c-4f2c-9839-d03ad7e7f581downstream_cells_mapBase.hashshow_tree$86aa821b-a373-4814-953e-535f3a33c002Base.showAbstractTrees.childrenEX$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303$1f1b384a-6588-45a5-9dd3-6de3face8bfb$d82adc20-4c8c-4f2c-9839-d03ad7e7f581Base.==upstream_cells_mapMetaIO:>islessEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581AnyUIntBase.show_unquotedendsqrt√islessPlots.frameTracked$99b6ab91-a022-449c-988c-0e5c5719c910grad$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93adjointSymbolAnychildren$8110f306-a7bb-43a2-bb36-6182c59b4b2eprecedence_heuristic cell_id$8110f306-a7bb-43a2-bb36-6182c59b4b2edownstream_cells_mapBase.showAbstractTrees.childrenTTREE$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303$8110f306-a7bb-43a2-bb36-6182c59b4b2eupstream_cells_mapMIMEDictIOAnyget@htlBaseprintAbstractTrees$d5559dba-9fe4-11ec-3744-ebd1408e7dc4TTREE$8110f306-a7bb-43a2-bb36-6182c59b4b2eHypertextLiteral.ResultHypertextLiteral$d5559dba-9fe4-11ec-3744-ebd1408e7dc4$99b6ab91-a022-449c-988c-0e5c5719c910precedence_heuristic cell_id$99b6ab91-a022-449c-988c-0e5c5719c910downstream_cells_mapBase.promote_ruleBase.showBase.convertTracked$0b5e6560-81fd-4182-bba5-aca702fb3048$99b6ab91-a022-449c-988c-0e5c5719c910$8919faec-753d-41d0-b0a8-c84c025b2919$13487e65-5e48-4a37-9bea-f262dd7b6d56$b0cc4665-eb45-48ea-9a33-5acf56d2a283$73d638bf-30c1-4694-b3a8-4b29c5e3fa65$2141849b-675e-406c-8df4-34b2706507af$ac097299-0a31-474c-ab26-a4fb24bb9046$8ab0f55d-a393-4a8a-a48c-9ced26033f57$2188a663-5a85-4ce4-bc8d-20383481e59b$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93$d9304d8d-9a34-46f2-908d-42d5cc5f5c5f$520f280d-c78e-433b-a0a2-8ef05b04f7cc$a5671f2f-48ca-4896-9428-147dc671d2b9$e01114fa-a847-424a-931f-8f42e623109f$788903f0-a372-4265-bbf3-5b507a705100$96286b65-1a22-4458-a399-46579248cce4$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9$86fa378b-815d-4c3d-9121-1338ee54f30f$1f1b384a-6588-45a5-9dd3-6de3face8bfbBase.==upstream_cells_mapstringIOTracked$99b6ab91-a022-449c-988c-0e5c5719c910IOContextprint_treeAbstractArray@MIME_str=>Symbolpromote_typenothingVectorBasegensym===printTypeNumber$db3389c0-cb00-404c-81a8-8c4ccdab22f4precedence_heuristic cell_id$db3389c0-cb00-404c-81a8-8c4ccdab22f4downstream_cells_map#471#ex#472#stepsupstream_cells_mapTTREE$8110f306-a7bb-43a2-bb36-6182c59b4b2ead_steps$1f1b384a-6588-45a5-9dd3-6de3face8bfbshow_steps$094bf7ee-7c9b-458c-98dc-521768831654@ad_steps$faf69c6f-b179-4ca2-aa22-641e9560e17bEX$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$b0cc4665-eb45-48ea-9a33-5acf56d2a283precedence_heuristic cell_id$b0cc4665-eb45-48ea-9a33-5acf56d2a283downstream_cells_mapBase.-upstream_cells_mapBaseTracked$99b6ab91-a022-449c-988c-0e5c5719c910$d4e9b202-242e-4420-986b-12d2ab57af93precedence_heuristic cell_id$d4e9b202-242e-4420-986b-12d2ab57af93downstream_cells_mapgrad$506d408e-dc2b-4e12-b917-286e3f4079a2$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7$96286b65-1a22-4458-a399-46579248cce4$49333dc7-b809-4bb1-b285-20a12d8b9209$86fa378b-815d-4c3d-9121-1338ee54f30fupstream_cells_mapTracked$99b6ab91-a022-449c-988c-0e5c5719c910cell_execution_order_$d5559dba-9fe4-11ec-3744-ebd1408e7dc4$4da7a594-aec5-4c8d-ab20-6913ed6ff8ac$1b768306-0e0e-414d-a501-5c62a2db720f$6733267e-1251-49a7-86a0-ff45ba408205$17f9e2c0-aa70-4d4e-ba42-51782df43785$7299bfea-e23e-4993-bebd-9d8762427294$be693f92-de60-4b43-82b3-ee6cceea310a$20571e1a-9687-4d42-98b3-b3bc0b207b3b$5e02b2c0-10cf-4746-840d-7017648f89f8$d1312080-272c-4bd5-bcfe-bf47f8320961$99b6ab91-a022-449c-988c-0e5c5719c910$0b5e6560-81fd-4182-bba5-aca702fb3048$0d006e4d-72e3-42ab-beb3-e5304a4d9ba9$8919faec-753d-41d0-b0a8-c84c025b2919$be28f764-a415-46ed-a503-68e8d831f8a8$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958b$814786be-a760-4023-a614-83e727681075$2ba83769-89b4-4ba0-8726-9210f394b4e7$885cd51d-895f-4996-a23b-780498b5b810$4813e57f-557a-4179-afd2-7925687b3d35$ea94bdf9-b0b1-42ad-ba44-aaba73ed1192$13487e65-5e48-4a37-9bea-f262dd7b6d56$c9d70049-4793-41db-9075-e9e73e926c1a$b0cc4665-eb45-48ea-9a33-5acf56d2a283$73d638bf-30c1-4694-b3a8-4b29c5e3fa65$95ab6c83-0757-40b3-a116-6d872a074f69$db4f871a-3ca1-4bd3-a2c5-da7d6f599281$29360c2a-c47b-48a3-b225-248eb3ec8a42$b47754ac-cb17-4ea3-a97d-20371fd341a5$bb75e790-a628-4821-b7b1-11cd34a73d38$ac097299-0a31-474c-ab26-a4fb24bb9046$aa99b123-7164-4ed7-833f-b99b19118a0d$0d2a3187-46f1-4705-bb3e-ecf74e34840d$61513f88-c126-43ab-9515-85a2c8a472bf$2c3f3103-9ccb-446e-a289-87bbd0a1b97b$795c510e-6280-42d9-8820-290ca07be88d$e450329b-46ee-4f53-b6f0-891d6596938c$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0$2141849b-675e-406c-8df4-34b2706507af$8ab0f55d-a393-4a8a-a48c-9ced26033f57$4bfc2f7d-a5b0-44c7-8bb6-f1b834c1cc51$2188a663-5a85-4ce4-bc8d-20383481e59b$ccafd0b9-95aa-4e58-8afc-26cd3ee61cc9$81eb8a2d-a3a9-45af-a5a5-b96aefd48712$01eacbd4-ef37-4524-aecc-2ef9a1044cf8$e52aa672-69a9-419b-a992-e7a3d1364fb6$d5565717-239b-41ab-b311-d59b383130ed$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93$7fcccb65-4a7b-4527-97be-d25f481f6eaf$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96$506d408e-dc2b-4e12-b917-286e3f4079a2$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7$e55dfad2-db50-459f-ab54-fa7637fc3638$d9304d8d-9a34-46f2-908d-42d5cc5f5c5f$a6c2d3a2-326a-41dd-864d-aa3662466222$bcff2aa8-2387-44c7-a28f-39cd505a7adf$53ae4070-8818-4d21-8648-19df9319918a$8a70e835-dae6-4727-8204-95c87d5c23da$520f280d-c78e-433b-a0a2-8ef05b04f7cc$a5671f2f-48ca-4896-9428-147dc671d2b9$e01114fa-a847-424a-931f-8f42e623109f$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44$266bf2ed-d625-47be-8a87-21e12052b27f$788903f0-a372-4265-bbf3-5b507a705100$337bc837-174f-4ed6-bee1-c9e80dc03b52$8f1ec8d7-89ab-4bc5-9eef-2b0a69019854$d2d21d41-9a4e-428b-b696-6878901849ba$98807f0d-f8a5-4fe0-9983-2b95290347d9$85098c31-04a1-421c-b3f6-25fec9360588$7dca2c78-6faf-450a-8096-1c5b9f7f1295$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9$0911a08a-4290-455b-9b26-0bf2862296da$0b094198-cf44-41d7-a8dc-fd8fd0716bb4$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$96286b65-1a22-4458-a399-46579248cce4$49333dc7-b809-4bb1-b285-20a12d8b9209$86fa378b-815d-4c3d-9121-1338ee54f30f$8110f306-a7bb-43a2-bb36-6182c59b4b2e$1f1b384a-6588-45a5-9dd3-6de3face8bfb$1a154bb7-93a3-4973-8908-788db77ac294$6b1fb808-e993-4c2b-b81b-6710f8206de7$437285d4-ec53-4bb7-9966-fcfb5352e205$86aa821b-a373-4814-953e-535f3a33c002$094bf7ee-7c9b-458c-98dc-521768831654$faf69c6f-b179-4ca2-aa22-641e9560e17b$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303$f6ce8448-d9ce-4453-9e47-dc6443d50f55$9a141034-17cb-4d85-a5a2-4724a38dd269$79f71f9d-b491-4a2c-85a4-29ae8da4f312last_hot_reload_timeshortpathnotebook.jlprocess_statusreadypath?/home/runner/work/ReverseModePluto/ReverseModePluto/notebook.jlpluto_versionv0.20.13last_save_timeA&n}cell_order_$d5559dba-9fe4-11ec-3744-ebd1408e7dc4$4da7a594-aec5-4c8d-ab20-6913ed6ff8ac$1b768306-0e0e-414d-a501-5c62a2db720f$6733267e-1251-49a7-86a0-ff45ba408205$0b5e6560-81fd-4182-bba5-aca702fb3048$17f9e2c0-aa70-4d4e-ba42-51782df43785$c9d70049-4793-41db-9075-e9e73e926c1a$aa99b123-7164-4ed7-833f-b99b19118a0d$0d2a3187-46f1-4705-bb3e-ecf74e34840d$61513f88-c126-43ab-9515-85a2c8a472bf$2c3f3103-9ccb-446e-a289-87bbd0a1b97b$795c510e-6280-42d9-8820-290ca07be88d$e450329b-46ee-4f53-b6f0-891d6596938c$7299bfea-e23e-4993-bebd-9d8762427294$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0$be693f92-de60-4b43-82b3-ee6cceea310a$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726$ec723b90-1c8b-4a87-8a86-fe17d6e85f22$0d186a1b-eb89-4be8-a237-e5a9608f4a30$db3389c0-cb00-404c-81a8-8c4ccdab22f4$8b486bd8-25a1-4895-99c8-0541c9e3c8b0$20571e1a-9687-4d42-98b3-b3bc0b207b3b$5e02b2c0-10cf-4746-840d-7017648f89f8$95ab6c83-0757-40b3-a116-6d872a074f69$db4f871a-3ca1-4bd3-a2c5-da7d6f599281$29360c2a-c47b-48a3-b225-248eb3ec8a42$b47754ac-cb17-4ea3-a97d-20371fd341a5$bb75e790-a628-4821-b7b1-11cd34a73d38$d1312080-272c-4bd5-bcfe-bf47f8320961$99b6ab91-a022-449c-988c-0e5c5719c910$0d006e4d-72e3-42ab-beb3-e5304a4d9ba9$8919faec-753d-41d0-b0a8-c84c025b2919$be28f764-a415-46ed-a503-68e8d831f8a8$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958b$814786be-a760-4023-a614-83e727681075$2ba83769-89b4-4ba0-8726-9210f394b4e7$885cd51d-895f-4996-a23b-780498b5b810$4813e57f-557a-4179-afd2-7925687b3d35$ea94bdf9-b0b1-42ad-ba44-aaba73ed1192$13487e65-5e48-4a37-9bea-f262dd7b6d56$b0cc4665-eb45-48ea-9a33-5acf56d2a283$73d638bf-30c1-4694-b3a8-4b29c5e3fa65$2141849b-675e-406c-8df4-34b2706507af$ac097299-0a31-474c-ab26-a4fb24bb9046$8ab0f55d-a393-4a8a-a48c-9ced26033f57$4bfc2f7d-a5b0-44c7-8bb6-f1b834c1cc51$2188a663-5a85-4ce4-bc8d-20383481e59b$ccafd0b9-95aa-4e58-8afc-26cd3ee61cc9$81eb8a2d-a3a9-45af-a5a5-b96aefd48712$01eacbd4-ef37-4524-aecc-2ef9a1044cf8$e52aa672-69a9-419b-a992-e7a3d1364fb6$d5565717-239b-41ab-b311-d59b383130ed$99a3507b-ca03-429f-acde-e2d1ebb32054$d4e9b202-242e-4420-986b-12d2ab57af93$7fcccb65-4a7b-4527-97be-d25f481f6eaf$18b1c55d-a6b5-44f6-b0b3-50bdb0aa9d96$506d408e-dc2b-4e12-b917-286e3f4079a2$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7$e55dfad2-db50-459f-ab54-fa7637fc3638$d9304d8d-9a34-46f2-908d-42d5cc5f5c5f$a6c2d3a2-326a-41dd-864d-aa3662466222$bcff2aa8-2387-44c7-a28f-39cd505a7adf$2eb6af9c-9ac7-4b64-93fb-c5a2f037d303$79f71f9d-b491-4a2c-85a4-29ae8da4f312$53ae4070-8818-4d21-8648-19df9319918a$8a70e835-dae6-4727-8204-95c87d5c23da$520f280d-c78e-433b-a0a2-8ef05b04f7cc$a5671f2f-48ca-4896-9428-147dc671d2b9$e01114fa-a847-424a-931f-8f42e623109f$e6ee1921-57aa-4194-b2eb-8e32fa4a6c44$266bf2ed-d625-47be-8a87-21e12052b27f$788903f0-a372-4265-bbf3-5b507a705100$337bc837-174f-4ed6-bee1-c9e80dc03b52$8f1ec8d7-89ab-4bc5-9eef-2b0a69019854$d2d21d41-9a4e-428b-b696-6878901849ba$98807f0d-f8a5-4fe0-9983-2b95290347d9$96286b65-1a22-4458-a399-46579248cce4$86aa821b-a373-4814-953e-535f3a33c002$49333dc7-b809-4bb1-b285-20a12d8b9209$85098c31-04a1-421c-b3f6-25fec9360588$7dca2c78-6faf-450a-8096-1c5b9f7f1295$b2622dc4-2fd3-4d80-8efd-a6a985a6b2e9$0911a08a-4290-455b-9b26-0bf2862296da$86fa378b-815d-4c3d-9121-1338ee54f30f$0b094198-cf44-41d7-a8dc-fd8fd0716bb4$1f1b384a-6588-45a5-9dd3-6de3face8bfb$d82adc20-4c8c-4f2c-9839-d03ad7e7f581$8110f306-a7bb-43a2-bb36-6182c59b4b2e$1a154bb7-93a3-4973-8908-788db77ac294$6b1fb808-e993-4c2b-b81b-6710f8206de7$437285d4-ec53-4bb7-9966-fcfb5352e205$094bf7ee-7c9b-458c-98dc-521768831654$faf69c6f-b179-4ca2-aa22-641e9560e17b$f6ce8448-d9ce-4453-9e47-dc6443d50f55$9a141034-17cb-4d85-a5a2-4724a38dd269published_objects50e7f5148-7698-11f0-1006-399f8abb1fe9/43c4266ba4b738bb>[*] → 4 ├─ 2 └─ [-] → 2 ├─ 3=x └─ 1 50e7f5148-7698-11f0-1006-399f8abb1fe9/ae663cf966d227c6[-] → 2 ├─ 3=x └─ 1 50e7f5148-7698-11f0-1006-399f8abb1fe9/ed2ed7f6d9d166463=x 50e7f5148-7698-11f0-1006-399f8abb1fe9/b9f6587eade9ac605=y 50e7f5148-7698-11f0-1006-399f8abb1fe9/fbc7f8abf7436a38[*] → 15 ├─ 3 └─ 5=y 50e7f5148-7698-11f0-1006-399f8abb1fe9/520b800031481075ٌ[+] → 19 ├─ [*] → 15 │ ├─ 3 │ └─ 5=y └─ [*] → 4 ├─ 2 └─ [-] → 2 ├─ 3=x └─ 1 nbpkginstall_time_nsyZinstantiatedòinstalled_versionsAbstractTrees0.4.5LinearAlgebrastdlibPlutoUI0.7.60Plots1.31.7HypertextLiteral0.9.5PlutoTest0.2.2terminal_outputsnbpkg_sync Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ LinearAlgebra Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ PlutoUI Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ Plots Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ HypertextLiteral Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ AbstractTrees Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ PlutoTest Resolving... === ┌ Warning: Pkg operation failed. Fixing stdlib dependencies and trying again... └ @ GracefulPkg ~/.julia/packages/GracefulPkg/LmTcK/src/apply strategies.jl:92 Resolving... ===  No Changes to `/tmp/jl_hw5ONu/Project.toml`  No Changes to `/tmp/jl_hw5ONu/Manifest.toml` Instantiating... === Precompiling... ===  Activating project at `/tmp/jl_hw5ONu` Precompiling project... 148 dependencies successfully precompiled in 77 seconds. 43 already precompiled. 1 dependency had output during precompilation: ┌ Formatting │ ┌ Warning: DEPRECATION NOTICE │ │  │ │ Formatting.jl has been unmaintained for a while, with some serious │ │ correctness bugs compromising the original purpose of the package. As a result, │ │ it has been deprecated - consider using an alternative, such as │ │ `Format.jl` (https://github.com/JuliaString/Format.jl) or the `Printf` stdlib directly. │ │  │ │ If you are not using Formatting.jl as a direct dependency, please consider │ │ opening an issue on any packages you are using that do use it as a dependency. │ │ From Julia 1.9 onwards, you can query `]why Formatting` to figure out which │ │ package originally brings it in as a dependency. │ └ @ Formatting ~/.julia/packages/Formatting/3VxOt/src/Formatting.jl:12 └ enabled÷restart_recommended_msgrestart_required_msgbusy_packageswaiting_for_permission,waiting_for_permission_but_probably_disabled«cell_inputs_$1b768306-0e0e-414d-a501-5c62a2db720fcell_id$1b768306-0e0e-414d-a501-5c62a2db720fcodemd"## Computations as Trees"metadatashow_logsèdisabled®skip_as_script«code_folded$aa99b123-7164-4ed7-833f-b99b19118a0dcell_id$aa99b123-7164-4ed7-833f-b99b19118a0dcodex^2+ymetadatashow_logsèdisabled®skip_as_script«code_folded$be693f92-de60-4b43-82b3-ee6cceea310acell_id$be693f92-de60-4b43-82b3-ee6cceea310acodeٰmd""" $\frac{\partial}{\partial x} (2xy + (x-1)^2) = 2y + 2(x-1)$ $\frac{\partial}{\partial y} (2xy + (x-1)^2) = 2x$ If x = 3 and y = 5, then these evaluate to 14 and 6. """metadatashow_logsèdisabled®skip_as_script«code_folded$1f1b384a-6588-45a5-9dd3-6de3face8bfbcell_id$1f1b384a-6588-45a5-9dd3-6de3face8bfbcodefunction ad_steps(x::Expr; color_fwd="red", color_bwd="green", font_size=".8em") x = EX(x) repr(x) = sprint(show, x; context=:compact=>true) span_fwd = @htl "" span_bwd = @htl "" d1 = Dict( let e = eval(i.x) i => @htl " $(span_fwd)$(repr(e isa Tracked ? e.val : e))" end for i in PostOrderDFS(x) if isempty(children(i)) ) res = accumulate(Iterators.filter(x -> !isempty(children(x)), PostOrderDFS(x)); init=d1) do d,i d = copy(d) e = eval(i.x) d[i] = @htl " $(span_fwd)$(repr(e isa Tracked ? e.val : e))" d end pushfirst!(res, d1) f = eval(x.x) d = Dict{Any, Any}(f => 1) let d1 = copy(res[end]) d1[x] = @htl "$(d1[x]) $(span_bwd)1" push!(res, d1) end for (x, e) in zip(PreOrderDFS.((f, x))...) x.df === nothing && continue dy = x.df(d[x]) for (yᵢ, dyᵢ, e) in zip(x.deps, dy, children(e)) d1 = copy(res[end]) if haskey(d, yᵢ) d1[e] = @htl "$(get(d1, e, ""))$(span_bwd) + $(repr(dyᵢ))" else d1[e] = @htl "$(get(d1, e, "")) $(span_bwd)$(repr(dyᵢ))" end push!(res, d1) d[yᵢ] = get(d, yᵢ, 0) + dyᵢ end end res endmetadatashow_logsèdisabled®skip_as_script«code_folded$e55dfad2-db50-459f-ab54-fa7637fc3638cell_id$e55dfad2-db50-459f-ab54-fa7637fc3638codemd""" ## How can we visualize both the forward and the reverse pass? We can further visualize each steps we just took. First we do the forwards calculation, where we also build up our tree, then we go down the tree in the opposite direction to accumulate our gradient. """metadatashow_logsèdisabled®skip_as_script«code_folded$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bcell_id$a0c0f48d-e0b0-4f45-8e0e-d9ab5743958bcodeDump(aa)metadatashow_logsèdisabled®skip_as_script«code_folded$7dca2c78-6faf-450a-8096-1c5b9f7f1295cell_id$7dca2c78-6faf-450a-8096-1c5b9f7f1295codeleaky_relu(x) = x > 0 ? x : .1xmetadatashow_logsèdisabled®skip_as_script«code_folded$13487e65-5e48-4a37-9bea-f262dd7b6d56cell_id$13487e65-5e48-4a37-9bea-f262dd7b6d56codegfunction Base.:+(x::Tracked, y::Tracked) Tracked(x.val + y.val, :+, Δ -> (Δ, Δ), Tracked[x, y]) endmetadatashow_logsèdisabled®skip_as_script«code_folded$96286b65-1a22-4458-a399-46579248cce4cell_id$96286b65-1a22-4458-a399-46579248cce4codebegin _rand(s...) = randn(s...) / √prod(s) W1, W2, W3 = _rand(32, 1), _rand(32, 32), _rand(1, 32) b1, b2, b3 = _rand(32), _rand(32), _rand(1) params = [@t(W1), @t(W2), @t(W3), @t(b1), @t(b2), @t(b3)] p = @animate for i in 1:10000 loss = norm(NN(input, params) - @t(ŷ)) ∇ = grad(loss) for p in params p.val .-= 1e-3 .* ∇[p] end plot(input', [NN(input, params).val; ŷ]'; label=["prediction" "training data"]) end every 200 gif(p; fps=5) endmetadatashow_logsèdisabled®skip_as_script«code_folded$1a154bb7-93a3-4973-8908-788db77ac294cell_id$1a154bb7-93a3-4973-8908-788db77ac294code?s2 = @htl """ """metadatashow_logsèdisabled®skip_as_script«code_folded$8f1ec8d7-89ab-4bc5-9eef-2b0a69019854cell_id$8f1ec8d7-89ab-4bc5-9eef-2b0a69019854code0md""" We'll try to approximate a sine curve: """metadatashow_logsèdisabled®skip_as_script«code_folded$795c510e-6280-42d9-8820-290ca07be88dcell_id$795c510e-6280-42d9-8820-290ca07be88dcode((x^2+y).deps[2]).depsmetadatashow_logsèdisabled®skip_as_script«code_folded$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726cell_id$27b39d7d-fc08-4ccc-aea4-b64f8a4f5726codeU#@ad_steps 3y*x + 2(x-1)*x @ad_steps (2x*y + (x-1)^2) # type the steps for the macrometadatashow_logsèdisabled®skip_as_script«code_folded$85098c31-04a1-421c-b3f6-25fec9360588cell_id$85098c31-04a1-421c-b3f6-25fec9360588codeLmd"### What if we use a (leaky) rectified linear unit (ReLU) as activation?"metadatashow_logsèdisabled®skip_as_script«code_folded$8ab0f55d-a393-4a8a-a48c-9ced26033f57cell_id$8ab0f55d-a393-4a8a-a48c-9ced26033f57codecfunction Base.sin(x::Tracked) Tracked(sin(x.val), :sin, Δ -> (Δ * cos(x.val),), Tracked[x,]) endmetadatashow_logsèdisabled®skip_as_script«code_folded$e01114fa-a847-424a-931f-8f42e623109fcell_id$e01114fa-a847-424a-931f-8f42e623109fcodezfunction LinearAlgebra.norm(x::Tracked) res = norm(x.val) Tracked(res, :norm, Δ -> (Δ/res .* x.val,), Tracked[x,]) endmetadatashow_logsèdisabled®skip_as_script«code_folded$2ba83769-89b4-4ba0-8726-9210f394b4e7cell_id$2ba83769-89b4-4ba0-8726-9210f394b4e7codemd"### Operations on Tracked"metadatashow_logsèdisabled®skip_as_script«code_folded$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0cell_id$d48b9f23-b1b8-43f8-a05d-2234b69bf1f0code(2x*y + (x-1)^2)metadatashow_logsèdisabled®skip_as_script«code_folded$a5671f2f-48ca-4896-9428-147dc671d2b9cell_id$a5671f2f-48ca-4896-9428-147dc671d2b9code٢function Base.broadcasted(::typeof(+), x::Tracked{<:VecOrMat}, y::Tracked{<:Vector}) Tracked(x.val .+ y.val, :.+, Δ -> (Δ, sum(Δ; dims=2)), Tracked[x, y]) endmetadatashow_logsèdisabled®skip_as_script«code_folded$d5565717-239b-41ab-b311-d59b383130edcell_id$d5565717-239b-41ab-b311-d59b383130edcodermd""" Ok, let's create our function `grad` which will accumulate all intermediate gradients into a dictionary: """metadatashow_logsèdisabled®skip_as_script«code_folded$81eb8a2d-a3a9-45af-a5a5-b96aefd48712cell_id$81eb8a2d-a3a9-45af-a5a5-b96aefd48712code5(2x*y + (x-1)^2).val # The result of `2x*y + (x-1)^2`metadatashow_logsèdisabled®skip_as_script«code_folded$2188a663-5a85-4ce4-bc8d-20383481e59bcell_id$2188a663-5a85-4ce4-bc8d-20383481e59bcode+AbstractTrees.children(x::Tracked) = x.depsmetadatashow_logsèdisabled®skip_as_script«code_folded$db4f871a-3ca1-4bd3-a2c5-da7d6f599281cell_id$db4f871a-3ca1-4bd3-a2c5-da7d6f599281code(x*y).valmetadatashow_logsèdisabled®skip_as_script«code_folded$53ae4070-8818-4d21-8648-19df9319918acell_id$53ae4070-8818-4d21-8648-19df9319918acode0md""" ## Let's write our own neural network! """metadatashow_logsèdisabled®skip_as_script«code_folded$79f71f9d-b491-4a2c-85a4-29ae8da4f312cell_id$79f71f9d-b491-4a2c-85a4-29ae8da4f312code@visual_debug 3y + 2(x-1)metadatashow_logsèdisabled®skip_as_script«code_folded$e450329b-46ee-4f53-b6f0-891d6596938ccell_id$e450329b-46ee-4f53-b6f0-891d6596938ccodex^2+ymetadatashow_logsèdisabled®skip_as_script«code_folded$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7cell_id$a7c8cb6a-6e17-4d8f-8958-fe3527c5b8e7codegrad(w, x), grad(w, y)metadatashow_logsèdisabled®skip_as_script«code_folded$d1312080-272c-4bd5-bcfe-bf47f8320961cell_id$d1312080-272c-4bd5-bcfe-bf47f8320961code"md"### The Tracked Data Structure"metadatashow_logsèdisabled®skip_as_script«code_folded$885cd51d-895f-4996-a23b-780498b5b810cell_id$885cd51d-895f-4996-a23b-780498b5b810codeْmd""" All overloads will do the operation (e.g. sum `x` and `y`), but also remember the pullback map and input variables for the reverse pass. """metadatashow_logsèdisabled®skip_as_script«code_folded$d5559dba-9fe4-11ec-3744-ebd1408e7dc4cell_id$d5559dba-9fe4-11ec-3744-ebd1408e7dc4codeOusing AbstractTrees, PlutoUI, HypertextLiteral, PlutoTest, LinearAlgebra, Plotsmetadatashow_logsèdisabled®skip_as_script«code_folded$49333dc7-b809-4bb1-b285-20a12d8b9209cell_id$49333dc7-b809-4bb1-b285-20a12d8b9209codelet ŷ_t = @t(ŷ) loss = norm(NN(input, params) - ŷ_t) ∇ = grad(loss) params′ = deepcopy(params) for (p, p′) in zip(params, params′) p′.val .-= 1e-3 .* ∇[p] end plot(input', [NN(input, params).val; ŷ]'; label=["prediction" "training data"]) quiver!(input[1:4:end], NN(input, params).val[1:4:end]; quiver=(zero(input[1:4:end]), 3(NN(input, params′).val .- NN(input, params).val)[1:4:end])) endmetadatashow_logsèdisabled®skip_as_script«code_folded$f6ce8448-d9ce-4453-9e47-dc6443d50f55cell_id$f6ce8448-d9ce-4453-9e47-dc6443d50f55codes1 = html"""