marselester's bloghttps://marselester.com/2023-04-02T00:00:00+07:00DIY CPU profiler: position independent executable π₯§2023-04-02T00:00:00+07:002023-04-02T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2023-04-02:/diy-cpu-profiler-position-independent-executable.html<p>In the <a href="https://marselester.com/diy-cpu-profiler-the-simplest-case-of-symbolization.html">simplest case of symbolization</a>
we used <code>-fno-pie -no-pie</code> flags, so
gcc wouldn't produce a position independent executable (PIE).
It is produced by default for security measures such as
address space layout randomization (ASLR).
That means each time the program runs,
its segments are loaded into different regions of β¦</p><p>In the <a href="https://marselester.com/diy-cpu-profiler-the-simplest-case-of-symbolization.html">simplest case of symbolization</a>
we used <code>-fno-pie -no-pie</code> flags, so
gcc wouldn't produce a position independent executable (PIE).
It is produced by default for security measures such as
address space layout randomization (ASLR).
That means each time the program runs,
its segments are loaded into different regions of virtual memory,
but their relative positions stay the same.</p>
<p>Let's see what happens when those flags are omitted.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© gcc -Og -fcf-protection=none -o fib main.c</span>
<span class="go">οΉ© ./fib &</span>
<span class="go">οΉ© sudo go run ./cmd/profiler3 -pid=$(pidof fib)</span>
<span class="go">/proc/19978/maps</span>
<span class="go">start=0x555806f2c000 limit=0x555806f2d000 offset=0x1000 /vagrant/diy-parca-agent/fib</span>
<span class="go">start=0x7f4e21595000 limit=0x7f4e2172a000 offset=0x28000 /usr/lib/x86_64-linux-gnu/libc.so.6</span>
<span class="go">start=0x7f4e2179f000 limit=0x7f4e217c9000 offset=0x2000 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2</span>
<span class="go">start=0x7fffe4d9e000 limit=0x7fffe4da0000 offset=0x0 [vdso]</span>
<span class="go">start=0xffffffffff600000 limit=0xffffffffff601000 offset=0x0 [vsyscall]</span>
<span class="go">Waiting for stack traces for 10s...</span>
<span class="go">{PID:19978 UserStackID:288 KernelStackID:-14} seen 254 times</span>
<span class="go"> 288 [555806f2c139]</span>
<span class="go">...</span>
</code></pre></div>
<p>The first thing I noticed when looking at samples is much higher addresses,
e.g., we've just sampled <code>93836562055481</code> address vs <code>4198694</code> from
the <a href="https://marselester.com/diy-cpu-profiler-the-simplest-case-of-symbolization.html">previous post</a>.
The size of the mapping id 1 is still one page,
i.e., 4096 bytes = memory_limit - memory_start.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© zcat cpu.pprof | protoc --decode perftools.profiles.Profile ./pprof/proto/profile.proto</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="n">sample_type</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">type</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">unit</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">75</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">26</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">48</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">79</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">99</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">23</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">13</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">87</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">203</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">61</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">254</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">14</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">14</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">14</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055168</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562059264</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">4096</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">139973543677952</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">139973545336832</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">163840</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">139973545816064</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">139973545988096</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">8192</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">140737032871936</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">140737032880128</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">18446744073699065856</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">18446744073699069952</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055526</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055495</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055506</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055494</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055499</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055487</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055481</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055531</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055523</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055511</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055493</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055518</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">13</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055492</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">14</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055502</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055530</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">""</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"samples"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"count"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"/vagrant/diy-parca-agent/fib"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"/usr/lib/x86_64-linux-gnu/libc.so.6"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"[vdso]"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"[vsyscall]"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"cpu"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"nanoseconds"</span><span class="w"></span>
<span class="n">time_nanos</span><span class="p">:</span><span class="w"> </span><span class="mi">1680393594260736123</span><span class="w"></span>
<span class="n">duration_nanos</span><span class="p">:</span><span class="w"> </span><span class="mi">10000000000</span><span class="w"></span>
<span class="n">period_type</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">type</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"></span>
<span class="w"> </span><span class="n">unit</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">period</span><span class="p">:</span><span class="w"> </span><span class="mi">10000000</span><span class="w"></span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="p">...</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055168</span><span class="w"> </span><span class="c1"># 4198400 before</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562059264</span><span class="w"> </span><span class="c1"># 4202496 before</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">4096</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">...</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">93836562055481</span><span class="w"> </span><span class="c1"># 4198694 before</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">...</span><span class="w"></span>
</code></pre></div>
<p>The second thing is a mismatch between
where the segment should be loaded <code>VirtAddr=0x0000000000001000</code>
and where it was actually loaded <code>memory_start: 93836562055168</code>.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© readelf --segments --wide fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="go">Elf file type is DYN (Position-Independent Executable file)</span>
<span class="go">Entry point 0x1050</span>
<span class="go">There are 13 program headers, starting at offset 64</span>
<span class="go">Program Headers:</span>
<span class="go"> Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align</span>
<span class="go"> PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002d8 0x0002d8 R 0x8</span>
<span class="go"> INTERP 0x000318 0x0000000000000318 0x0000000000000318 0x00001c 0x00001c R 0x1</span>
<span class="go"> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]</span>
<span class="go"> LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000638 0x000638 R 0x1000</span>
<span class="go"> LOAD 0x001000 0x0000000000001000 0x0000000000001000 0x0001b1 0x0001b1 R E 0x1000</span>
<span class="go"> LOAD 0x002000 0x0000000000002000 0x0000000000002000 0x00010c 0x00010c R 0x1000</span>
<span class="go"> LOAD 0x002db8 0x0000000000003db8 0x0000000000003db8 0x000258 0x000260 RW 0x1000</span>
<span class="go"> DYNAMIC 0x002dc8 0x0000000000003dc8 0x0000000000003dc8 0x0001f0 0x0001f0 RW 0x8</span>
<span class="go"> NOTE 0x000338 0x0000000000000338 0x0000000000000338 0x000020 0x000020 R 0x8</span>
<span class="go"> NOTE 0x000358 0x0000000000000358 0x0000000000000358 0x000044 0x000044 R 0x4</span>
<span class="go"> GNU_PROPERTY 0x000338 0x0000000000000338 0x0000000000000338 0x000020 0x000020 R 0x8</span>
<span class="go"> GNU_EH_FRAME 0x002020 0x0000000000002020 0x0000000000002020 0x000034 0x000034 R 0x4</span>
<span class="go"> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10</span>
<span class="go"> GNU_RELRO 0x002db8 0x0000000000003db8 0x0000000000003db8 0x000248 0x000248 R 0x1</span>
<span class="go"> Section to Segment mapping:</span>
<span class="go"> Segment Sections...</span>
<span class="go"> 00</span>
<span class="go"> 01 .interp</span>
<span class="go"> 02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt</span>
<span class="go"> 03 .init .plt .plt.got .text .fini</span>
<span class="go"> 04 .rodata .eh_frame_hdr .eh_frame</span>
<span class="go"> 05 .init_array .fini_array .dynamic .got .data .bss</span>
<span class="go"> 06 .dynamic</span>
<span class="go"> 07 .note.gnu.property</span>
<span class="go"> 08 .note.gnu.build-id .note.ABI-tag</span>
<span class="go"> 09 .note.gnu.property</span>
<span class="go"> 10 .eh_frame_hdr</span>
<span class="go"> 11</span>
<span class="go"> 12 .init_array .fini_array .dynamic .got</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">Program Headers:</span>
<span class="go"> Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align</span>
<span class="go"> ...</span>
<span class="go"> LOAD 0x001000 0x0000000000001000 0x0000000000001000 0x0001b1 0x0001b1 R E 0x1000</span>
<span class="go"> ...</span>
<span class="go"> Section to Segment mapping:</span>
<span class="go"> Segment Sections...</span>
<span class="go"> ...</span>
<span class="go"> 03 .init .plt .plt.got .text .fini</span>
<span class="go"> ...</span>
</code></pre></div>
<p>The third thing is a difference in the segment's sections: the <code>.plt.got</code> section was added.</p>
<h2>How PIE affects symbolization</h2>
<p>Clearly none of the sampled addresses matches <code>fibNaive</code> function
when we look at the symbol table.
The function's address <code>0x1139</code> is very far from
the beginning of the loaded segment <code>0x555806f2c000</code>
(that's hex of <code>memory_start: 93836562055168</code>).</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© readelf --symbols fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="go">Symbol table '.dynsym' contains 7 entries:</span>
<span class="go"> Num: Value Size Type Bind Vis Ndx Name</span>
<span class="go"> 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND</span>
<span class="go"> 1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND _[...]@GLIBC_2.34 (2)</span>
<span class="go"> 2: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_deregisterT[...]</span>
<span class="go"> 3: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__</span>
<span class="go"> 4: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...]@GLIBC_2.3.4 (3)</span>
<span class="go"> 5: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_registerTMC[...]</span>
<span class="go"> 6: 0000000000000000 0 FUNC WEAK DEFAULT UND [...]@GLIBC_2.2.5 (4)</span>
<span class="go">Symbol table '.symtab' contains 37 entries:</span>
<span class="go"> Num: Value Size Type Bind Vis Ndx Name</span>
<span class="go"> 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND</span>
<span class="go"> 1: 0000000000000000 0 FILE LOCAL DEFAULT ABS Scrt1.o</span>
<span class="go"> 2: 000000000000037c 32 OBJECT LOCAL DEFAULT 4 __abi_tag</span>
<span class="go"> 3: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c</span>
<span class="go"> 4: 0000000000001080 0 FUNC LOCAL DEFAULT 15 deregister_tm_clones</span>
<span class="go"> 5: 00000000000010b0 0 FUNC LOCAL DEFAULT 15 register_tm_clones</span>
<span class="go"> 6: 00000000000010f0 0 FUNC LOCAL DEFAULT 15 __do_global_dtors_aux</span>
<span class="go"> 7: 0000000000004010 1 OBJECT LOCAL DEFAULT 25 completed.0</span>
<span class="go"> 8: 0000000000003dc0 0 OBJECT LOCAL DEFAULT 21 __do_global_dtor[...]</span>
<span class="go"> 9: 0000000000001130 0 FUNC LOCAL DEFAULT 15 frame_dummy</span>
<span class="go"> 10: 0000000000003db8 0 OBJECT LOCAL DEFAULT 20 __frame_dummy_in[...]</span>
<span class="go"> 11: 0000000000000000 0 FILE LOCAL DEFAULT ABS main.c</span>
<span class="go"> 12: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c</span>
<span class="go"> 13: 0000000000002108 0 OBJECT LOCAL DEFAULT 19 __FRAME_END__</span>
<span class="go"> 14: 0000000000000000 0 FILE LOCAL DEFAULT ABS</span>
<span class="go"> 15: 0000000000003dc8 0 OBJECT LOCAL DEFAULT 22 _DYNAMIC</span>
<span class="go"> 16: 0000000000002020 0 NOTYPE LOCAL DEFAULT 18 __GNU_EH_FRAME_HDR</span>
<span class="go"> 17: 0000000000003fb8 0 OBJECT LOCAL DEFAULT 23 _GLOBAL_OFFSET_TABLE_</span>
<span class="go"> 18: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_mai[...]</span>
<span class="go"> 19: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_deregisterT[...]</span>
<span class="go"> 20: 0000000000004000 0 NOTYPE WEAK DEFAULT 24 data_start</span>
<span class="go"> 21: 0000000000004010 0 NOTYPE GLOBAL DEFAULT 24 _edata</span>
<span class="go"> 22: 00000000000011a4 0 FUNC GLOBAL HIDDEN 16 _fini</span>
<span class="go"> 23: 0000000000004000 0 NOTYPE GLOBAL DEFAULT 24 __data_start</span>
<span class="go"> 24: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__</span>
<span class="go"> 25: 0000000000004008 0 OBJECT GLOBAL HIDDEN 24 __dso_handle</span>
<span class="go"> 26: 0000000000002000 4 OBJECT GLOBAL DEFAULT 17 _IO_stdin_used</span>
<span class="go"> 27: 0000000000004018 0 NOTYPE GLOBAL DEFAULT 25 _end</span>
<span class="go"> 28: 0000000000001050 38 FUNC GLOBAL DEFAULT 15 _start</span>
<span class="go"> 29: 0000000000004010 0 NOTYPE GLOBAL DEFAULT 25 __bss_start</span>
<span class="go"> 30: 000000000000116d 54 FUNC GLOBAL DEFAULT 15 main</span>
<span class="go"> 31: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __printf_chk@GLI[...]</span>
<span class="go"> 32: 0000000000004010 0 OBJECT GLOBAL HIDDEN 24 __TMC_END__</span>
<span class="go"> 33: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_registerTMC[...]</span>
<span class="go"> 34: 0000000000000000 0 FUNC WEAK DEFAULT UND __cxa_finalize@G[...]</span>
<span class="go"> 35: 0000000000001000 0 FUNC GLOBAL HIDDEN 12 _init</span>
<span class="go"> 36: 0000000000001139 52 FUNC GLOBAL DEFAULT 15 fibNaive</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">Symbol table '.symtab' contains 37 entries:</span>
<span class="go"> Num: Value Size Type Bind Vis Ndx Name</span>
<span class="go"> ...</span>
<span class="go"> 36: 0000000000001139 52 FUNC GLOBAL DEFAULT 15 fibNaive</span>
</code></pre></div>
<p>Let's try to figure out where sections are mapped in the virtual address space.
For that, we need to know the distance from the segment to each section,
i.e., the section's offset minus the segment's offset:</p>
<ul>
<li>0 bytes from <code>.init</code> section = 0x1000 - 0x1000</li>
<li>0x20 bytes from <code>.plt</code> section = 0x1020 - 0x1000</li>
<li>0x40 bytes from <code>.plt.got</code> section = 0x1040 - 0x1000</li>
<li>0x50 bytes from <code>.text</code> section = 0x1050 - 0x1000</li>
<li>0x1a4 bytes from <code>.fini</code> section = 0x11a4 - 0x1000</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© readelf --sections --wide fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="go">There are 30 section headers, starting at offset 0x36b8:</span>
<span class="go">Section Headers:</span>
<span class="go"> [Nr] Name Type Address Off Size ES Flg Lk Inf Al</span>
<span class="go"> [ 0] NULL 0000000000000000 000000 000000 00 0 0 0</span>
<span class="go"> [ 1] .interp PROGBITS 0000000000000318 000318 00001c 00 A 0 0 1</span>
<span class="go"> [ 2] .note.gnu.property NOTE 0000000000000338 000338 000020 00 A 0 0 8</span>
<span class="go"> [ 3] .note.gnu.build-id NOTE 0000000000000358 000358 000024 00 A 0 0 4</span>
<span class="go"> [ 4] .note.ABI-tag NOTE 000000000000037c 00037c 000020 00 A 0 0 4</span>
<span class="go"> [ 5] .gnu.hash GNU_HASH 00000000000003a0 0003a0 000024 00 A 6 0 8</span>
<span class="go"> [ 6] .dynsym DYNSYM 00000000000003c8 0003c8 0000a8 18 A 7 1 8</span>
<span class="go"> [ 7] .dynstr STRTAB 0000000000000470 000470 0000a1 00 A 0 0 1</span>
<span class="go"> [ 8] .gnu.version VERSYM 0000000000000512 000512 00000e 02 A 6 0 2</span>
<span class="go"> [ 9] .gnu.version_r VERNEED 0000000000000520 000520 000040 00 A 7 1 8</span>
<span class="go"> [10] .rela.dyn RELA 0000000000000560 000560 0000c0 18 A 6 0 8</span>
<span class="go"> [11] .rela.plt RELA 0000000000000620 000620 000018 18 AI 6 23 8</span>
<span class="go"> [12] .init PROGBITS 0000000000001000 001000 00001b 00 AX 0 0 4</span>
<span class="go"> [13] .plt PROGBITS 0000000000001020 001020 000020 10 AX 0 0 16</span>
<span class="go"> [14] .plt.got PROGBITS 0000000000001040 001040 000008 08 AX 0 0 8</span>
<span class="go"> [15] .text PROGBITS 0000000000001050 001050 000153 00 AX 0 0 16</span>
<span class="go"> [16] .fini PROGBITS 00000000000011a4 0011a4 00000d 00 AX 0 0 4</span>
<span class="go"> [17] .rodata PROGBITS 0000000000002000 002000 00001f 00 A 0 0 4</span>
<span class="go"> [18] .eh_frame_hdr PROGBITS 0000000000002020 002020 000034 00 A 0 0 4</span>
<span class="go"> [19] .eh_frame PROGBITS 0000000000002058 002058 0000b4 00 A 0 0 8</span>
<span class="go"> [20] .init_array INIT_ARRAY 0000000000003db8 002db8 000008 08 WA 0 0 8</span>
<span class="go"> [21] .fini_array FINI_ARRAY 0000000000003dc0 002dc0 000008 08 WA 0 0 8</span>
<span class="go"> [22] .dynamic DYNAMIC 0000000000003dc8 002dc8 0001f0 10 WA 7 0 8</span>
<span class="go"> [23] .got PROGBITS 0000000000003fb8 002fb8 000048 08 WA 0 0 8</span>
<span class="go"> [24] .data PROGBITS 0000000000004000 003000 000010 00 WA 0 0 8</span>
<span class="go"> [25] .bss NOBITS 0000000000004010 003010 000008 00 WA 0 0 1</span>
<span class="go"> [26] .comment PROGBITS 0000000000000000 003010 00002b 01 MS 0 0 1</span>
<span class="go"> [27] .symtab SYMTAB 0000000000000000 003040 000378 18 28 18 8</span>
<span class="go"> [28] .strtab STRTAB 0000000000000000 0033b8 0001eb 00 0 0 1</span>
<span class="go"> [29] .shstrtab STRTAB 0000000000000000 0035a3 000111 00 0 0 1</span>
<span class="go">Key to Flags:</span>
<span class="go"> W (write), A (alloc), X (execute), M (merge), S (strings), I (info),</span>
<span class="go"> L (link order), O (extra OS processing required), G (group), T (TLS),</span>
<span class="go"> C (compressed), x (unknown), o (OS specific), E (exclude),</span>
<span class="go"> D (mbind), l (large), p (processor specific)</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">Section Headers:</span>
<span class="go"> [Nr] Name Type Address Off Size ES Flg Lk Inf Al</span>
<span class="go"> ...</span>
<span class="go"> [12] .init PROGBITS 0000000000001000 001000 00001b 00 AX 0 0 4</span>
<span class="go"> [13] .plt PROGBITS 0000000000001020 001020 000020 10 AX 0 0 16</span>
<span class="go"> [14] .plt.got PROGBITS 0000000000001040 001040 000008 08 AX 0 0 8</span>
<span class="go"> [15] .text PROGBITS 0000000000001050 001050 000153 00 AX 0 0 16</span>
<span class="go"> [16] .fini PROGBITS 00000000000011a4 0011a4 00000d 00 AX 0 0 4</span>
<span class="go"> ...</span>
</code></pre></div>
<p>Then we can add those distances to <code>memory_start: 0x555806f2c000</code>
to get an idea how the process's virtual address space looked like
when the executable file was mapped.</p>
<div class="highlight"><pre><span></span><code><span class="go">Executable object file Virtual address space</span>
<span class="go"> __________ __________</span>
<span class="go">| | 0x3e37 | | N</span>
<span class="go">| | | |</span>
<span class="go">| ... | 0x2000 | ... | 0x555806f2d000</span>
<span class="go">|----------| -----------> vm_end |----------|</span>
<span class="go">| .fini | 0x11a4 | .fini | 0x555806f2c1a4</span>
<span class="go">| .text | 0x1050 | .text | 0x555806f2c050</span>
<span class="go">| .plt.got | 0x1040 | .plt.got | 0x555806f2c040</span>
<span class="go">| .plt | 0x1020 | .plt | 0x555806f2c020</span>
<span class="go">| .init | 0x1000 | .init | 0x555806f2c000</span>
<span class="go">|----------| ---------> vm_start |----------|</span>
<span class="go">| ... | | ... |</span>
<span class="go">|__________| 0 |__________| 0</span>
</code></pre></div>
<p>The sampled addresses should be in the <code>.text</code> section
which starts at <code>0x1050</code> offset within the file.
Since the segment itself starts at <code>0x1000</code> offset,
the distance between them is 0x50 bytes = 0x1050 - 0x1000.
Therefore, in virtual address space the <code>.text</code> section starts at
<code>0x555806f2c050</code> address = memory_start + distance = 0x555806f2c000 + 0x50.</p>
<p>Our sampled address <code>0x555806f2c139</code> (that's <code>address: 93836562055481</code> in hexadecimal form)
can be adjusted to the ELF file's address range as follows:</p>
<ol>
<li>Find the distance between the sampled memory address and
beginning of the loaded segment:
313 bytes = 0x555806f2c139 - 0x555806f2c000.</li>
<li>Add the segment's offset to the distance
to find the address used in the <code>.symtab</code> section:
0x1139 = 0x1000 + 313.</li>
</ol>
<p>The <code>0x1139</code> address belongs to <code>fibNaive</code> function as we've seen
in the symbol table.
That's nice! π</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Distance between the sampled memory address and</span><span class="w"></span>
<span class="c1">// beginning of the loaded segment (vm_start memory address).</span><span class="w"></span>
<span class="nx">d</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">sampledAddr</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nx">mappingMemoryStart</span><span class="w"></span>
<span class="c1">// Sampled address adjusted to .symtab address range.</span><span class="w"></span>
<span class="nx">adjustedAddr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">mappingFileOffset</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">d</span><span class="w"></span>
</code></pre></div>
<p>Here are all the addresses from <code>cpu.pprof</code> file sorted in ascending order,
their distances from <code>memory_start: 0x555806f2c000</code>,
and addresses adjusted to the file's range.</p>
<table>
<thead>
<tr>
<th>sampled address</th>
<th>distance</th>
<th>adjusted address</th>
</tr>
</thead>
<tbody>
<tr>
<td>0x555806f2c139</td>
<td>313 bytes</td>
<td>0x1139</td>
</tr>
<tr>
<td>0x555806f2c13f</td>
<td>319 bytes</td>
<td>0x113f</td>
</tr>
<tr>
<td>0x555806f2c144</td>
<td>324 bytes</td>
<td>0x1144</td>
</tr>
<tr>
<td>0x555806f2c145</td>
<td>325 bytes</td>
<td>0x1145</td>
</tr>
<tr>
<td>0x555806f2c146</td>
<td>326 bytes</td>
<td>0x1146</td>
</tr>
<tr>
<td>0x555806f2c147</td>
<td>327 bytes</td>
<td>0x1147</td>
</tr>
<tr>
<td>0x555806f2c14b</td>
<td>331 bytes</td>
<td>0x114b</td>
</tr>
<tr>
<td>0x555806f2c14e</td>
<td>334 bytes</td>
<td>0x114e</td>
</tr>
<tr>
<td>0x555806f2c152</td>
<td>338 bytes</td>
<td>0x1152</td>
</tr>
<tr>
<td>0x555806f2c157</td>
<td>343 bytes</td>
<td>0x1157</td>
</tr>
<tr>
<td>0x555806f2c15e</td>
<td>350 bytes</td>
<td>0x115e</td>
</tr>
<tr>
<td>0x555806f2c163</td>
<td>355 bytes</td>
<td>0x1163</td>
</tr>
<tr>
<td>0x555806f2c166</td>
<td>358 bytes</td>
<td>0x1166</td>
</tr>
<tr>
<td>0x555806f2c16a</td>
<td>362 bytes</td>
<td>0x116a</td>
</tr>
<tr>
<td>0x555806f2c16b</td>
<td>363 bytes</td>
<td>0x116b</td>
</tr>
</tbody>
</table>
<p>Disassembling <code>fib</code> program confirms that adjusted memory addresses
correspond to instructions of <code>fibNaive</code> function.</p>
<div class="highlight"><pre><span></span><code><span class="x">οΉ© objdump --disassemble=fibNaive fib</span>
<span class="x">...</span>
<span class="mh">0000000000001139</span><span class="w"> </span><span class="p"><</span><span class="nf">fibNaive</span><span class="p">>:</span>
<span class="x"> 1139: 48 83 ff 02 cmp $0x2,%rdi</span>
<span class="x"> 113d: 7f 06 jg 1145 <fibNaive+0xc></span>
<span class="x"> 113f: b8 01 00 00 00 mov $0x1,%eax</span>
<span class="x"> 1144: c3 ret</span>
<span class="x"> 1145: 55 push %rbp</span>
<span class="x"> 1146: 53 push %rbx</span>
<span class="x"> 1147: 48 83 ec 08 sub $0x8,%rsp</span>
<span class="x"> 114b: 48 89 fb mov %rdi,%rbx</span>
<span class="x"> 114e: 48 8d 7f fe lea -0x2(%rdi),%rdi</span>
<span class="x"> 1152: e8 e2 ff ff ff call 1139 <fibNaive></span>
<span class="x"> 1157: 48 89 c5 mov %rax,%rbp</span>
<span class="x"> 115a: 48 8d 7b ff lea -0x1(%rbx),%rdi</span>
<span class="x"> 115e: e8 d6 ff ff ff call 1139 <fibNaive></span>
<span class="x"> 1163: 48 01 e8 add %rbp,%rax</span>
<span class="x"> 1166: 48 83 c4 08 add $0x8,%rsp</span>
<span class="x"> 116a: 5b pop %rbx</span>
<span class="x"> 116b: 5d pop %rbp</span>
<span class="x"> 116c: c3 ret</span>
<span class="x">...</span>
</code></pre></div>
<p>Based on these observations, I made a symbolization tool
<a href="https://github.com/marselester/diy-parca-agent/blob/main/cmd/addr2func/main.go">addr2func</a>
and proposed a change in Parca, see <a href="https://github.com/parca-dev/parca/pull/2910">#2910</a>.</p>DIY CPU profiler: the simplest case of symbolization2023-03-23T00:00:00+07:002023-03-23T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2023-03-23:/diy-cpu-profiler-the-simplest-case-of-symbolization.html<p>In the <a href="https://marselester.com/diy-cpu-profiler-from-bpf-maps-to-pprof.html">BPF maps to pprof</a> post
we managed to collect CPU samples and store them in pprof format.
Here I would like to explore the simplest case of symbolization
and put shared libraries aside.</p>
<p>Symbolization is resolving sampled memory addresses to function names (symbols).
In our case, simply searching β¦</p><p>In the <a href="https://marselester.com/diy-cpu-profiler-from-bpf-maps-to-pprof.html">BPF maps to pprof</a> post
we managed to collect CPU samples and store them in pprof format.
Here I would like to explore the simplest case of symbolization
and put shared libraries aside.</p>
<p>Symbolization is resolving sampled memory addresses to function names (symbols).
In our case, simply searching for an address in <code>.symtab</code> ELF section.</p>
<h2>What's in the profile?</h2>
<p>Let's profile a C program that ensures we'll get lots of CPU samples.
It computes Fibonacci numbers naively (without caching),
so <code>fibNaive</code> function should dominate the profile.</p>
<div class="highlight"><pre><span></span><code><span class="kt">long</span><span class="w"> </span><span class="nf">fibNaive</span><span class="p">(</span><span class="kt">long</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">fibNaive</span><span class="p">(</span><span class="n">n</span><span class="mi">-2</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">fibNaive</span><span class="p">(</span><span class="n">n</span><span class="mi">-1</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="c1">// fibNaive calculates a Fibonacci number n,</span>
<span class="c1">// e.g., when n=7, the result is 13: 1 1 2 3 5 8 13.</span>
<span class="c1">//</span>
<span class="c1">// The running time complexity is exponential 2**n,</span>
<span class="c1">// the space complexity is n because</span>
<span class="c1">// when we descend the binary tree recursively,</span>
<span class="c1">// we use number of stacks equal to its height which is n</span>
<span class="c1">// (the stack frames n=7->5->3->1).</span>
<span class="kt">long</span><span class="w"> </span><span class="nf">fibNaive</span><span class="p">(</span><span class="kt">long</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">fibNaive</span><span class="p">(</span><span class="n">n</span><span class="mi">-2</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">fibNaive</span><span class="p">(</span><span class="n">n</span><span class="mi">-1</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Calculating 50th Fibonacci number will keep CPU busy</span>
<span class="w"> </span><span class="c1">// and give us time to run a CPU profiler.</span>
<span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">50</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fibNaive</span><span class="p">(</span><span class="n">n</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Fibonacci number %li: %li</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="p">,</span><span class="w"> </span><span class="n">res</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© # For reference, I used gcc 11.3.0 and Ubuntu 22.04.</span>
<span class="go">οΉ© gcc -Og -fno-pie -no-pie -fcf-protection=none -o fib main.c</span>
<span class="go">οΉ© ./fib &</span>
<span class="go">οΉ© sudo go run ./cmd/profiler3 -pid=$(pidof fib)</span>
<span class="go">/proc/14548/maps</span>
<span class="go">start=0x401000 limit=0x402000 offset=0x1000 /vagrant/diy-parca-agent/fib</span>
<span class="go">start=0x7f3cd28ac000 limit=0x7f3cd2a41000 offset=0x28000 /usr/lib/x86_64-linux-gnu/libc.so.6</span>
<span class="go">start=0x7f3cd2ab6000 limit=0x7f3cd2ae0000 offset=0x2000 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2</span>
<span class="go">start=0x7ffc69d62000 limit=0x7ffc69d64000 offset=0x0 [vdso]</span>
<span class="go">start=0xffffffffff600000 limit=0xffffffffff601000 offset=0x0 [vsyscall]</span>
<span class="go">Waiting for stack traces for 10s...</span>
<span class="go">{PID:14548 UserStackID:535 KernelStackID:-14} seen 138 times</span>
<span class="go"> 535 [401126]</span>
<span class="go">...</span>
</code></pre></div>
<p>After ten seconds the profiler will terminate leaving us with <code>cpu.pprof</code> file.
Let's inspect it with
<a href="https://github.com/DataDog/go-profiler-notes/blob/main/pprof.md#using-protoc">protoc</a>.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© git clone https://github.com/google/pprof.git</span>
<span class="go">οΉ© sudo snap install protobuf --classic</span>
<span class="go">οΉ© zcat cpu.pprof | \</span>
<span class="go"> protoc --decode perftools.profiles.Profile ./pprof/proto/profile.proto</span>
<span class="go">...</span>
<span class="go">sample {</span>
<span class="go"> location_id: 10</span>
<span class="go"> value: 138</span>
<span class="go">}</span>
<span class="go">...</span>
<span class="go">mapping {</span>
<span class="go"> id: 1</span>
<span class="go"> memory_start: 4198400</span>
<span class="go"> memory_limit: 4202496</span>
<span class="go"> file_offset: 4096</span>
<span class="go"> filename: 3</span>
<span class="go">}</span>
<span class="go">...</span>
<span class="go">location {</span>
<span class="go"> id: 10</span>
<span class="go"> mapping_id: 1</span>
<span class="go"> address: 4198694</span>
<span class="go">}</span>
<span class="go">...</span>
<span class="go">string_table: ""</span>
<span class="go">string_table: "samples"</span>
<span class="go">string_table: "count"</span>
<span class="go">string_table: "/vagrant/diy-parca-agent/fib"</span>
<span class="go">string_table: "/usr/lib/x86_64-linux-gnu/libc.so.6"</span>
<span class="go">string_table: "/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2"</span>
<span class="go">string_table: "[vdso]"</span>
<span class="go">string_table: "[vsyscall]"</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="n">sample_type</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">type</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">unit</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">18</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">27</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">286</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">111</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">79</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">29</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">138</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">36</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">34</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">13</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">14</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">22</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">37</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">105</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">sample</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">location_id</span><span class="p">:</span><span class="w"> </span><span class="mi">16</span><span class="w"></span>
<span class="w"> </span><span class="n">value</span><span class="p">:</span><span class="w"> </span><span class="mi">54</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">4198400</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">4202496</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">4096</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">139899207073792</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">139899208732672</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">163840</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">139899209211904</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">139899209383936</span><span class="w"></span>
<span class="w"> </span><span class="n">file_offset</span><span class="p">:</span><span class="w"> </span><span class="mi">8192</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">140722084126720</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">140722084134912</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">mapping</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_start</span><span class="p">:</span><span class="w"> </span><span class="mi">18446744073699065856</span><span class="w"></span>
<span class="w"> </span><span class="n">memory_limit</span><span class="p">:</span><span class="w"> </span><span class="mi">18446744073699069952</span><span class="w"></span>
<span class="w"> </span><span class="n">filename</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198715</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198705</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198700</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198724</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198706</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198708</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">7</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198727</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198736</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198745</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198694</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">11</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198712</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">12</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198707</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">13</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198739</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">14</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198743</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198744</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">location</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">id</span><span class="p">:</span><span class="w"> </span><span class="mi">16</span><span class="w"></span>
<span class="w"> </span><span class="n">mapping_id</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="p">:</span><span class="w"> </span><span class="mi">4198719</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">""</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"samples"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"count"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"/vagrant/diy-parca-agent/fib"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"/usr/lib/x86_64-linux-gnu/libc.so.6"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"[vdso]"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"[vsyscall]"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"cpu"</span><span class="w"></span>
<span class="n">string_table</span><span class="p">:</span><span class="w"> </span><span class="s">"nanoseconds"</span><span class="w"></span>
<span class="n">time_nanos</span><span class="p">:</span><span class="w"> </span><span class="mi">1679419143530427477</span><span class="w"></span>
<span class="n">duration_nanos</span><span class="p">:</span><span class="w"> </span><span class="mi">10000000000</span><span class="w"></span>
<span class="n">period_type</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">type</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w"></span>
<span class="w"> </span><span class="n">unit</span><span class="p">:</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">period</span><span class="p">:</span><span class="w"> </span><span class="mi">10000000</span><span class="w"></span>
</code></pre></div>
</details>
<p>The example above shows a pprof sample that tells us
that some function (its memory <code>address: 4198694</code>) has been seen 138 times in stack traces.
That function is located in the mapping id 1 according to location id 10,
somewhere between <code>memory_start: 4198400</code> and <code>memory_limit: 4202496</code> addresses.
The mapping points to <code>file_offset: 4096</code> within some file whose path is defined by <code>filename: 3</code>.
I assume it's the third value in the pprof string table counting from zero:
<code>string_table: "/vagrant/diy-parca-agent/fib"</code>.</p>
<p>What does it all mean?
If we recall from <a href="https://marselester.com/linux-process.html">Linux process</a> post,
Linux organizes the virtual memory of a process as a collection of areas (segments)
where an area is a chunk of related virtual pages.
An area can be associated (mapped) with a contiguous chunk of a file,
in this case it is <code>fib</code> executable object file starting at file offset 4096.
Let's look at the file's segments and find the one mentioned in the sample.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© readelf --segments --wide fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="go">Elf file type is EXEC (Executable file)</span>
<span class="go">Entry point 0x401040</span>
<span class="go">There are 13 program headers, starting at offset 64</span>
<span class="go">Program Headers:</span>
<span class="go"> Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align</span>
<span class="go"> PHDR 0x000040 0x0000000000400040 0x0000000000400040 0x0002d8 0x0002d8 R 0x8</span>
<span class="go"> INTERP 0x000318 0x0000000000400318 0x0000000000400318 0x00001c 0x00001c R 0x1</span>
<span class="go"> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]</span>
<span class="go"> LOAD 0x000000 0x0000000000400000 0x0000000000400000 0x0004f0 0x0004f0 R 0x1000</span>
<span class="go"> LOAD 0x001000 0x0000000000401000 0x0000000000401000 0x00019d 0x00019d R E 0x1000</span>
<span class="go"> LOAD 0x002000 0x0000000000402000 0x0000000000402000 0x000104 0x000104 R 0x1000</span>
<span class="go"> LOAD 0x002e10 0x0000000000403e10 0x0000000000403e10 0x000220 0x000228 RW 0x1000</span>
<span class="go"> DYNAMIC 0x002e20 0x0000000000403e20 0x0000000000403e20 0x0001d0 0x0001d0 RW 0x8</span>
<span class="go"> NOTE 0x000338 0x0000000000400338 0x0000000000400338 0x000020 0x000020 R 0x8</span>
<span class="go"> NOTE 0x000358 0x0000000000400358 0x0000000000400358 0x000044 0x000044 R 0x4</span>
<span class="go"> GNU_PROPERTY 0x000338 0x0000000000400338 0x0000000000400338 0x000020 0x000020 R 0x8</span>
<span class="go"> GNU_EH_FRAME 0x002020 0x0000000000402020 0x0000000000402020 0x000034 0x000034 R 0x4</span>
<span class="go"> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10</span>
<span class="go"> GNU_RELRO 0x002e10 0x0000000000403e10 0x0000000000403e10 0x0001f0 0x0001f0 R 0x1</span>
<span class="go"> Section to Segment mapping:</span>
<span class="go"> Segment Sections...</span>
<span class="go"> 00</span>
<span class="go"> 01 .interp</span>
<span class="go"> 02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt</span>
<span class="go"> 03 .init .plt .text .fini</span>
<span class="go"> 04 .rodata .eh_frame_hdr .eh_frame</span>
<span class="go"> 05 .init_array .fini_array .dynamic .got .got.plt .data .bss</span>
<span class="go"> 06 .dynamic</span>
<span class="go"> 07 .note.gnu.property</span>
<span class="go"> 08 .note.gnu.build-id .note.ABI-tag</span>
<span class="go"> 09 .note.gnu.property</span>
<span class="go"> 10 .eh_frame_hdr</span>
<span class="go"> 11</span>
<span class="go"> 12 .init_array .fini_array .dynamic .got</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">Program Headers:</span>
<span class="go"> Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align</span>
<span class="go"> ...</span>
<span class="go"> LOAD 0x001000 0x0000000000401000 0x0000000000401000 0x00019d 0x00019d R E 0x1000</span>
<span class="go"> ...</span>
<span class="go"> Section to Segment mapping:</span>
<span class="go"> Segment Sections...</span>
<span class="go"> ...</span>
<span class="go"> 03 .init .plt .text .fini</span>
<span class="go"> ...</span>
</code></pre></div>
<p>As we can see the ELF program header describes
<code>Flg=R E</code> readable and executable segment
loaded from <code>Offset=0x001000</code> (that's hex of <code>file_offset: 4096</code>) within the file
into memory starting at <code>0x0000000000401000</code> virtual address (that's hex of <code>memory_start: 4198400</code>).
According to <code>FileSiz=0x00019d</code> only 413 bytes are loaded from the file.
Given the alignment requirement <code>Align=0x1000</code>,
the memory area ends at <code>0x0000000000402000</code> virtual address (that's hex of <code>memory_limit: 4202496</code>).
So the area consists of a single page, i.e., 4096 bytes = memory_limit - memory_start.</p>
<p>Here is how a process's virtual address space looks
when the executable file is mapped.</p>
<div class="highlight"><pre><span></span><code><span class="go">Executable object file Virtual address space</span>
<span class="go"> _______ _______</span>
<span class="go">| | 0x3def | | N</span>
<span class="go">| | | |</span>
<span class="go">| ... | 0x2000 | ... | 0x402000</span>
<span class="go">|-------| -----------> vm_end |-------|</span>
<span class="go">| .fini | 0x1190 | .fini | 0x401190</span>
<span class="go">| .text | 0x1040 | .text | 0x401040</span>
<span class="go">| .plt | 0x1020 | .plt | 0x401020</span>
<span class="go">| .init | 0x1000 | .init | 0x401000</span>
<span class="go">|-------| ---------> vm_start |-------|</span>
<span class="go">| ... | | ... |</span>
<span class="go">|_______| 0 |_______| 0</span>
</code></pre></div>
<p>We can also see which ELF sections are placed within the segment: <code>.init .plt .text .fini</code>.
Our function in question should be somewhere in <code>.text</code> section,
because it's where machine code of the compiled program lives.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© readelf --sections --wide fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="go">There are 30 section headers, starting at offset 0x3670:</span>
<span class="go">Section Headers:</span>
<span class="go"> [Nr] Name Type Address Off Size ES Flg Lk Inf Al</span>
<span class="go"> [ 0] NULL 0000000000000000 000000 000000 00 0 0 0</span>
<span class="go"> [ 1] .interp PROGBITS 0000000000400318 000318 00001c 00 A 0 0 1</span>
<span class="go"> [ 2] .note.gnu.property NOTE 0000000000400338 000338 000020 00 A 0 0 8</span>
<span class="go"> [ 3] .note.gnu.build-id NOTE 0000000000400358 000358 000024 00 A 0 0 4</span>
<span class="go"> [ 4] .note.ABI-tag NOTE 000000000040037c 00037c 000020 00 A 0 0 4</span>
<span class="go"> [ 5] .gnu.hash GNU_HASH 00000000004003a0 0003a0 00001c 00 A 6 0 8</span>
<span class="go"> [ 6] .dynsym DYNSYM 00000000004003c0 0003c0 000060 18 A 7 1 8</span>
<span class="go"> [ 7] .dynstr STRTAB 0000000000400420 000420 000050 00 A 0 0 1</span>
<span class="go"> [ 8] .gnu.version VERSYM 0000000000400470 000470 000008 02 A 6 0 2</span>
<span class="go"> [ 9] .gnu.version_r VERNEED 0000000000400478 000478 000030 00 A 7 1 8</span>
<span class="go"> [10] .rela.dyn RELA 00000000004004a8 0004a8 000030 18 A 6 0 8</span>
<span class="go"> [11] .rela.plt RELA 00000000004004d8 0004d8 000018 18 AI 6 23 8</span>
<span class="go"> [12] .init PROGBITS 0000000000401000 001000 00001b 00 AX 0 0 4</span>
<span class="go"> [13] .plt PROGBITS 0000000000401020 001020 000020 10 AX 0 0 16</span>
<span class="go"> [14] .text PROGBITS 0000000000401040 001040 00014e 00 AX 0 0 16</span>
<span class="go"> [15] .fini PROGBITS 0000000000401190 001190 00000d 00 AX 0 0 4</span>
<span class="go"> [16] .rodata PROGBITS 0000000000402000 002000 00001f 00 A 0 0 4</span>
<span class="go"> [17] .eh_frame_hdr PROGBITS 0000000000402020 002020 000034 00 A 0 0 4</span>
<span class="go"> [18] .eh_frame PROGBITS 0000000000402058 002058 0000ac 00 A 0 0 8</span>
<span class="go"> [19] .init_array INIT_ARRAY 0000000000403e10 002e10 000008 08 WA 0 0 8</span>
<span class="go"> [20] .fini_array FINI_ARRAY 0000000000403e18 002e18 000008 08 WA 0 0 8</span>
<span class="go"> [21] .dynamic DYNAMIC 0000000000403e20 002e20 0001d0 10 WA 7 0 8</span>
<span class="go"> [22] .got PROGBITS 0000000000403ff0 002ff0 000010 08 WA 0 0 8</span>
<span class="go"> [23] .got.plt PROGBITS 0000000000404000 003000 000020 08 WA 0 0 8</span>
<span class="go"> [24] .data PROGBITS 0000000000404020 003020 000010 00 WA 0 0 8</span>
<span class="go"> [25] .bss NOBITS 0000000000404030 003030 000008 00 WA 0 0 1</span>
<span class="go"> [26] .comment PROGBITS 0000000000000000 003030 00002b 01 MS 0 0 1</span>
<span class="go"> [27] .symtab SYMTAB 0000000000000000 003060 000348 18 28 18 8</span>
<span class="go"> [28] .strtab STRTAB 0000000000000000 0033a8 0001b1 00 0 0 1</span>
<span class="go"> [29] .shstrtab STRTAB 0000000000000000 003559 000116 00 0 0 1</span>
<span class="go">Key to Flags:</span>
<span class="go"> W (write), A (alloc), X (execute), M (merge), S (strings), I (info),</span>
<span class="go"> L (link order), O (extra OS processing required), G (group), T (TLS),</span>
<span class="go"> C (compressed), x (unknown), o (OS specific), E (exclude),</span>
<span class="go"> D (mbind), l (large), p (processor specific)</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">Section Headers:</span>
<span class="go"> [Nr] Name Type Address Off Size ES Flg Lk Inf Al</span>
<span class="go"> ...</span>
<span class="go"> [12] .init PROGBITS 0000000000401000 001000 00001b 00 AX 0 0 4</span>
<span class="go"> [13] .plt PROGBITS 0000000000401020 001020 000020 10 AX 0 0 16</span>
<span class="go"> [14] .text PROGBITS 0000000000401040 001040 00014e 00 AX 0 0 16</span>
<span class="go"> [15] .fini PROGBITS 0000000000401190 001190 00000d 00 AX 0 0 4</span>
<span class="go"> ...</span>
</code></pre></div>
<h2>Symbol table</h2>
<p>Information about functions and global variables (aka symbols)
defined and referenced in our program lives in a symbol table in <code>.symtab</code> section.</p>
<blockquote>
<p>An object file's symbol table holds information needed to locate and relocate a program's symbolic definitions and references. https://www.man7.org/linux/man-pages/man5/elf.5.html</p>
</blockquote>
<p>This implies that we should be able to find
which symbol corresponds to memory <code>address: 4198694</code> or <code>0x401126</code> in hexadecimal form.
Let's inspect the symbol table.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© readelf --symbols fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="go">Symbol table '.dynsym' contains 4 entries:</span>
<span class="go"> Num: Value Size Type Bind Vis Ndx Name</span>
<span class="go"> 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND</span>
<span class="go"> 1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND _[...]@GLIBC_2.34 (2)</span>
<span class="go"> 2: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__</span>
<span class="go"> 3: 0000000000000000 0 FUNC GLOBAL DEFAULT UND [...]@GLIBC_2.3.4 (3)</span>
<span class="go">Symbol table '.symtab' contains 35 entries:</span>
<span class="go"> Num: Value Size Type Bind Vis Ndx Name</span>
<span class="go"> 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND</span>
<span class="go"> 1: 0000000000000000 0 FILE LOCAL DEFAULT ABS crt1.o</span>
<span class="go"> 2: 000000000040037c 32 OBJECT LOCAL DEFAULT 4 __abi_tag</span>
<span class="go"> 3: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c</span>
<span class="go"> 4: 0000000000401080 0 FUNC LOCAL DEFAULT 14 deregister_tm_clones</span>
<span class="go"> 5: 00000000004010b0 0 FUNC LOCAL DEFAULT 14 register_tm_clones</span>
<span class="go"> 6: 00000000004010f0 0 FUNC LOCAL DEFAULT 14 __do_global_dtors_aux</span>
<span class="go"> 7: 0000000000404030 1 OBJECT LOCAL DEFAULT 25 completed.0</span>
<span class="go"> 8: 0000000000403e18 0 OBJECT LOCAL DEFAULT 20 __do_global_dtor[...]</span>
<span class="go"> 9: 0000000000401120 0 FUNC LOCAL DEFAULT 14 frame_dummy</span>
<span class="go"> 10: 0000000000403e10 0 OBJECT LOCAL DEFAULT 19 __frame_dummy_in[...]</span>
<span class="go"> 11: 0000000000000000 0 FILE LOCAL DEFAULT ABS main.c</span>
<span class="go"> 12: 0000000000000000 0 FILE LOCAL DEFAULT ABS crtstuff.c</span>
<span class="go"> 13: 0000000000402100 0 OBJECT LOCAL DEFAULT 18 __FRAME_END__</span>
<span class="go"> 14: 0000000000000000 0 FILE LOCAL DEFAULT ABS</span>
<span class="go"> 15: 0000000000403e20 0 OBJECT LOCAL DEFAULT 21 _DYNAMIC</span>
<span class="go"> 16: 0000000000402020 0 NOTYPE LOCAL DEFAULT 17 __GNU_EH_FRAME_HDR</span>
<span class="go"> 17: 0000000000404000 0 OBJECT LOCAL DEFAULT 23 _GLOBAL_OFFSET_TABLE_</span>
<span class="go"> 18: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_mai[...]</span>
<span class="go"> 19: 0000000000404020 0 NOTYPE WEAK DEFAULT 24 data_start</span>
<span class="go"> 20: 0000000000404030 0 NOTYPE GLOBAL DEFAULT 24 _edata</span>
<span class="go"> 21: 0000000000401190 0 FUNC GLOBAL HIDDEN 15 _fini</span>
<span class="go"> 22: 0000000000404020 0 NOTYPE GLOBAL DEFAULT 24 __data_start</span>
<span class="go"> 23: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__</span>
<span class="go"> 24: 0000000000404028 0 OBJECT GLOBAL HIDDEN 24 __dso_handle</span>
<span class="go"> 25: 0000000000402000 4 OBJECT GLOBAL DEFAULT 16 _IO_stdin_used</span>
<span class="go"> 26: 0000000000404038 0 NOTYPE GLOBAL DEFAULT 25 _end</span>
<span class="go"> 27: 0000000000401070 5 FUNC GLOBAL HIDDEN 14 _dl_relocate_sta[...]</span>
<span class="go"> 28: 0000000000401040 38 FUNC GLOBAL DEFAULT 14 _start</span>
<span class="go"> 29: 0000000000404030 0 NOTYPE GLOBAL DEFAULT 25 __bss_start</span>
<span class="go"> 30: 000000000040115a 52 FUNC GLOBAL DEFAULT 14 main</span>
<span class="go"> 31: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __printf_chk@GLI[...]</span>
<span class="go"> 32: 0000000000404030 0 OBJECT GLOBAL HIDDEN 24 __TMC_END__</span>
<span class="go"> 33: 0000000000401000 0 FUNC GLOBAL HIDDEN 12 _init</span>
<span class="go"> 34: 0000000000401126 52 FUNC GLOBAL DEFAULT 14 fibNaive</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">Symbol table '.symtab' contains 35 entries:</span>
<span class="go"> Num: Value Size Type Bind Vis Ndx Name</span>
<span class="go"> ...</span>
<span class="go"> 34: 0000000000401126 52 FUNC GLOBAL DEFAULT 14 fibNaive</span>
</code></pre></div>
<p>So here you go, <strong>fibNaive</strong> C function we were looking for:</p>
<ul>
<li><code>Value=0000000000401126</code> The symbol's absolute run-time address
since it's an executable object file.
For relocatable object files, it would be an offset from
the beginning of the section defined by <code>Ndx=14</code>.</li>
<li><code>Size=52</code> The size of the function's code is 52 bytes.</li>
<li><code>Type=FUNC</code> The symbol is associated with a function.</li>
<li><code>Bind=GLOBAL</code> This indicates that the symbol is global,
i.e., it can be referenced by other modules.</li>
<li><code>Vis=DEFAULT</code> Default symbol visibility rules apply:
strong (functions and initialized global vars)
and weak symbols (uninitialized global vars) are available to other modules.</li>
<li><code>Ndx=14</code> A symbol is "defined" in relation to <code>.text</code> section
according to section header table index, see <code>readelf --sections fib</code>.</li>
<li><code>Name=fibNaive</code> The symbol name <code>fibNaive</code> was taken from
string table in section <code>.strtab</code>, see <code>readelf --string-dump .strtab fib</code>.</li>
</ul>
<p>π</p>
<p>Great, let's check the remaining memory addresses we've got in the pprof file
(I've sorted them for our convenience).
Unfortunately, I couldn't find corresponding symbols in <code>.symtab</code> section.</p>
<div class="highlight"><pre><span></span><code><span class="go">4198694 = 0x401126 fibNaive</span>
<span class="go">4198700 = 0x40112c ?</span>
<span class="go">4198705 = 0x401131 ?</span>
<span class="go">4198706 = 0x401132 ?</span>
<span class="go">4198707 = 0x401133 ?</span>
<span class="go">4198708 = 0x401134 ?</span>
<span class="go">4198712 = 0x401138 ?</span>
<span class="go">4198715 = 0x40113b ?</span>
<span class="go">4198719 = 0x40113f ?</span>
<span class="go">4198724 = 0x401144 ?</span>
<span class="go">4198727 = 0x401147 ?</span>
<span class="go">4198736 = 0x401150 ?</span>
<span class="go">4198739 = 0x401153 ?</span>
<span class="go">4198743 = 0x401157 ?</span>
<span class="go">4198744 = 0x401158 ?</span>
<span class="go">4198745 = 0x401159 ?</span>
</code></pre></div>
<p>What's interesting is that these addresses are within <code>Size=52</code> bytes
from <code>fibNaive</code> memory address, e.g.,
the distance to the highest address is 51 bytes = 4198745 - 4198694.</p>
<p>If we sort the symbols from <code>.symtab</code> section by <code>Value</code>,
we'll see that <code>fibNaive</code> is between <code>frame_dummy</code> and <code>main</code> symbols.
The rest of unresolved memory addresses from the samples
fit in between <code>fibNaive</code> and <code>main</code> symbols.</p>
<details>
<div class="highlight"><pre><span></span><code><span class="go">0000000 = 0x000000 crt1.o</span>
<span class="go">0000000 = 0x000000 crtstuff.c</span>
<span class="go">0000000 = 0x000000 main.c</span>
<span class="go">0000000 = 0x000000 crtstuff.c</span>
<span class="go">0000000 = 0x000000</span>
<span class="go">0000000 = 0x000000 __libc_start_main@GLIBC_2.34</span>
<span class="go">0000000 = 0x000000 __gmon_start__</span>
<span class="go">0000000 = 0x000000 __printf_chk@GLIBC_2.3.4</span>
<span class="go">4195196 = 0x40037c __abi_tag</span>
<span class="go">4198400 = 0x401000 _init</span>
<span class="go">4198464 = 0x401040 _start</span>
<span class="go">4198512 = 0x401070 _dl_relocate_static_pie</span>
<span class="go">4198528 = 0x401080 deregister_tm_clones</span>
<span class="go">4198576 = 0x4010b0 register_tm_clones</span>
<span class="go">4198640 = 0x4010f0 __do_global_dtors_aux</span>
<span class="go">4198688 = 0x401120 frame_dummy</span>
<span class="go">4198694 = 0x401126 fibNaive</span>
<span class="go">4198746 = 0x40115a main</span>
<span class="go">4198800 = 0x401190 _fini</span>
<span class="go">4202496 = 0x402000 _IO_stdin_used</span>
<span class="go">4202528 = 0x402020 __GNU_EH_FRAME_HDR</span>
<span class="go">4202752 = 0x402100 __FRAME_END__</span>
<span class="go">4210192 = 0x403e10 __frame_dummy_init_array_entry</span>
<span class="go">4210200 = 0x403e18 __do_global_dtors_aux_fini_array_entry</span>
<span class="go">4210208 = 0x403e20 _DYNAMIC</span>
<span class="go">4210688 = 0x404000 _GLOBAL_OFFSET_TABLE_</span>
<span class="go">4210720 = 0x404020 data_start</span>
<span class="go">4210720 = 0x404020 __data_start</span>
<span class="go">4210728 = 0x404028 __dso_handle</span>
<span class="go">4210736 = 0x404030 completed.0</span>
<span class="go">4210736 = 0x404030 _edata</span>
<span class="go">4210736 = 0x404030 __bss_start</span>
<span class="go">4210736 = 0x404030 __TMC_END__</span>
<span class="go">4210744 = 0x404038 _end</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="go">0x401120 frame_dummy</span>
<span class="go">0x401126 fibNaive</span>
<span class="go">0x40112c ?</span>
<span class="go">0x401131 ?</span>
<span class="go">0x401132 ?</span>
<span class="go">0x401133 ?</span>
<span class="go">0x401134 ?</span>
<span class="go">0x401138 ?</span>
<span class="go">0x40113b ?</span>
<span class="go">0x40113f ?</span>
<span class="go">0x401144 ?</span>
<span class="go">0x401147 ?</span>
<span class="go">0x401150 ?</span>
<span class="go">0x401153 ?</span>
<span class="go">0x401157 ?</span>
<span class="go">0x401158 ?</span>
<span class="go">0x401159 ?</span>
<span class="go">0x40115a main</span>
</code></pre></div>
<p>Disassembling the program confirms that memory addresses in question
correspond to instructions of <code>fibNaive</code> function.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© objdump --disassemble fib</span>
</code></pre></div>
<details>
<div class="highlight"><pre><span></span><code><span class="x">...</span>
<span class="mh">0000000000401120</span><span class="w"> </span><span class="p"><</span><span class="nf">frame_dummy</span><span class="p">>:</span>
<span class="x"> 401120: f3 0f 1e fa endbr64</span>
<span class="x"> 401124: eb 8a jmp 4010b0 <register_tm_clones></span>
<span class="mh">0000000000401126</span><span class="w"> </span><span class="p"><</span><span class="nf">fibNaive</span><span class="p">>:</span>
<span class="x"> 401126: 48 83 ff 02 cmp $0x2,%rdi</span>
<span class="x"> 40112a: 7f 06 jg 401132 <fibNaive+0xc></span>
<span class="x"> 40112c: b8 01 00 00 00 mov $0x1,%eax</span>
<span class="x"> 401131: c3 ret</span>
<span class="x"> 401132: 55 push %rbp</span>
<span class="x"> 401133: 53 push %rbx</span>
<span class="x"> 401134: 48 83 ec 08 sub $0x8,%rsp</span>
<span class="x"> 401138: 48 89 fb mov %rdi,%rbx</span>
<span class="x"> 40113b: 48 8d 7f fe lea -0x2(%rdi),%rdi</span>
<span class="x"> 40113f: e8 e2 ff ff ff call 401126 <fibNaive></span>
<span class="x"> 401144: 48 89 c5 mov %rax,%rbp</span>
<span class="x"> 401147: 48 8d 7b ff lea -0x1(%rbx),%rdi</span>
<span class="x"> 40114b: e8 d6 ff ff ff call 401126 <fibNaive></span>
<span class="x"> 401150: 48 01 e8 add %rbp,%rax</span>
<span class="x"> 401153: 48 83 c4 08 add $0x8,%rsp</span>
<span class="x"> 401157: 5b pop %rbx</span>
<span class="x"> 401158: 5d pop %rbp</span>
<span class="x"> 401159: c3 ret</span>
<span class="mh">000000000040115a</span><span class="w"> </span><span class="p"><</span><span class="nf">main</span><span class="p">>:</span>
<span class="x"> 40115a: 48 83 ec 08 sub $0x8,%rsp</span>
<span class="x"> 40115e: bf 32 00 00 00 mov $0x32,%edi</span>
<span class="x"> 401163: e8 be ff ff ff call 401126 <fibNaive></span>
<span class="x"> 401168: 48 89 c1 mov %rax,%rcx</span>
<span class="x"> 40116b: ba 32 00 00 00 mov $0x32,%edx</span>
<span class="x"> 401170: be 04 20 40 00 mov $0x402004,%esi</span>
<span class="x"> 401175: bf 01 00 00 00 mov $0x1,%edi</span>
<span class="x"> 40117a: b8 00 00 00 00 mov $0x0,%eax</span>
<span class="x"> 40117f: e8 ac fe ff ff call 401030 <__printf_chk@plt></span>
<span class="x"> 401184: b8 00 00 00 00 mov $0x0,%eax</span>
<span class="x"> 401189: 48 83 c4 08 add $0x8,%rsp</span>
<span class="x"> 40118d: c3 ret</span>
<span class="x">...</span>
</code></pre></div>
</details>
<div class="highlight"><pre><span></span><code><span class="mh">0000000000401126</span><span class="w"> </span><span class="p"><</span><span class="nf">fibNaive</span><span class="p">>:</span>
<span class="x"> 401126: 48 83 ff 02 cmp $0x2,%rdi</span>
<span class="x"> 40112a: 7f 06 jg 401132 <fibNaive+0xc></span>
<span class="x"> 40112c: b8 01 00 00 00 mov $0x1,%eax</span>
<span class="x"> 401131: c3 ret</span>
<span class="x"> 401132: 55 push %rbp</span>
<span class="x"> 401133: 53 push %rbx</span>
<span class="x"> 401134: 48 83 ec 08 sub $0x8,%rsp</span>
<span class="x"> 401138: 48 89 fb mov %rdi,%rbx</span>
<span class="x"> 40113b: 48 8d 7f fe lea -0x2(%rdi),%rdi</span>
<span class="x"> 40113f: e8 e2 ff ff ff call 401126 <fibNaive></span>
<span class="x"> 401144: 48 89 c5 mov %rax,%rbp</span>
<span class="x"> 401147: 48 8d 7b ff lea -0x1(%rbx),%rdi</span>
<span class="x"> 40114b: e8 d6 ff ff ff call 401126 <fibNaive></span>
<span class="x"> 401150: 48 01 e8 add %rbp,%rax</span>
<span class="x"> 401153: 48 83 c4 08 add $0x8,%rsp</span>
<span class="x"> 401157: 5b pop %rbx</span>
<span class="x"> 401158: 5d pop %rbp</span>
<span class="x"> 401159: c3 ret</span>
</code></pre></div>
<p>It means that <code>bpf_get_stackid</code> we used in
<a href="https://github.com/marselester/diy-parca-agent/blob/main/cmd/profiler/bpf/parca-agent.bpf.c">profiler</a>
collected values stored in <code>rip</code> register,
i.e., memory addresses of machine instructions.</p>
<p>If we compiled the program with frame pointers,
<code>bpf_get_stackid</code> would have additionally unwound a stack which
could look like <code>[401126, ..., 40115a]</code> (fibNaive, ..., main)
instead of <code>[401126]</code> when frame pointers are disabled (only <code>rip</code> value is stored),
see <a href="https://github.com/parca-dev/parca-agent/pull/1489#issuecomment-1488182745">Javier's explanation</a>.</p>
<h2>PC to symbol address</h2>
<p>Since we have sorted symbol addresses from <code>.symtab</code>,
we can binary search them to find
whose instructions were executing when a sample was taken.</p>
<div class="highlight"><pre><span></span><code><span class="go">0x401120 frame_dummy</span>
<span class="go">0x401126 fibNaive</span>
<span class="go">0x40112c ?</span>
<span class="go">0x40115a main</span>
</code></pre></div>
<p>When the program counter (PC) contains instruction address <code>0x401126</code>,
then <code>fibNaive</code> function symbol is found in the symbol table.
When PC points to <code>0x40112c</code> instruction,
there is no match in the symbol table,
so the <code>main</code> symbol's predecessor is resolved
because binary search stops at <code>0x40115a</code>.</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">PC2SymbolAddress</span><span class="p">(</span><span class="nx">symbols</span><span class="w"> </span><span class="p">[]</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="nx">pc</span><span class="w"> </span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">lft</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">mid</span><span class="w"> </span><span class="kt">int</span><span class="w"></span>
<span class="w"> </span><span class="nx">rgt</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">symbols</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">lft</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="nx">rgt</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">mid</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">lft</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="nx">rgt</span><span class="o">-</span><span class="nx">lft</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="nx">symAddr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">symbols</span><span class="p">[</span><span class="nx">mid</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="nx">symAddr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">pc</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">symAddr</span><span class="w"></span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="nx">symAddr</span><span class="w"> </span><span class="p"><</span><span class="w"> </span><span class="nx">pc</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nx">lft</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">mid</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="nx">symAddr</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">pc</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nx">rgt</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">mid</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Since pc wasn't found in the array,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// now mid points to symbol address > pc, i.e., 0x40115a.</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 0x401120 frame_dummy</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 0x401126 fibNaive</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 0x40112c ?</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 0x40115a main</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Therefore the desired symbol's address is 0x401126.</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">mid</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">symbols</span><span class="p">[</span><span class="nx">mid</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">-</span><span class="mi">1</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>In reality symbolization is much more complicated.
Most likely you would have to resolve symbols in shared libraries,
and also deal with things like:</p>
<ul>
<li><a href="https://www.polarsignals.com/blog/posts/2022/11/29/profiling-without-frame-pointers/">DWARF-based Stack Walking Using eBPF</a>.
Alas, frame pointers are disabled by default in gcc,
but you can enable them with the <code>-fno-omit-frame-pointer</code> flag.</li>
<li>Looking for symbols formatted in DWARF
which could be part of an executable file or distributed separately,
see <a href="https://www.polarsignals.com/blog/posts/2022/01/13/fantastic-symbols-and-where-to-find-them/">Fantastic Symbols and Where to Find Them</a>.
The <code>-g</code> gcc flag produces debugging information in DWARF format.</li>
</ul>
<p>When profiling JIT-compiled code you would have to
look for symbols in <code>/tmp/perf-$PID.map</code> file, see
<a href="https://www.polarsignals.com/blog/posts/2022/01/27/fantastic-symbols-and-where-to-find-them-part-2/">Fantastic Symbols and Where to Find Them - Part 2</a>.</p>Linux process2023-01-31T00:00:00+07:002023-01-31T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2023-01-31:/linux-process.html<p>Being curious about BPF, I studied source code of several programs from the BCC libbpf-tools.
BPF performance tools book aided me to navigate BPF C code.
For example, it explained that a BPF program has to use helpers because it can't access arbitrary memory (outside of BPF) and can't call β¦</p><p>Being curious about BPF, I studied source code of several programs from the BCC libbpf-tools.
BPF performance tools book aided me to navigate BPF C code.
For example, it explained that a BPF program has to use helpers because it can't access arbitrary memory (outside of BPF) and can't call arbitrary Linux kernel functions.</p>
<p>BPF has opened a door into the kernel for me, but I quickly realized that I don't know much about it.
Take, for example, <code>bpf_get_current_pid_tgid()</code> and <code>bpf_get_current_task()</code> helpers.
What are <code>tgid</code>, <code>pid</code>, and <code>task</code> in the function names?
It turns out that <code>tgid</code> is what user space calls a process ID,
and <code>pid</code> is what user space calls a thread ID.</p>
<div class="highlight"><pre><span></span><code><span class="n">u64</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_get_current_pid_tgid</span><span class="p">();</span><span class="w"></span>
<span class="kt">pid_t</span><span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">pid_t</span><span class="p">)</span><span class="n">id</span><span class="p">;</span><span class="w"></span>
<span class="kt">pid_t</span><span class="w"> </span><span class="n">tgid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="mi">32</span><span class="p">;</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="w"> </span><span class="o">*</span><span class="n">t</span><span class="p">;</span><span class="w"></span>
<span class="n">t</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="o">*</span><span class="p">)</span><span class="n">bpf_get_current_task</span><span class="p">();</span><span class="w"></span>
</code></pre></div>
<p>I found the terminology a bit confusing, so I spent some time trying to clarify it for myself.</p>
<h2>Task</h2>
<p>A process is an instance of a program in execution
that provides a program an illusion that it is the only one currently running
with exclusive use of CPU and memory.
A thread (thread of execution) is a unit of execution (sequence of machine instructions)
that can be managed by an OS scheduler.
Typically threads share the process's resources, e.g., memory and file descriptors.</p>
<p>The process and the thread are abstractions whose implementation depends on the OS.
Linux implements those abstractions with "light weight processes" which can share resources with each other,
thus in a way blending the process and the thread concepts.
For example, <code>ps</code> shows parca-agent Go program running as 8 light weight processes (also referred to as threads in the man page).
As we can see, they all have unique light weight process ID (LWP column), and the same process ID 93015 (PID column).
Together they form the thread group with ID 93015 which acts as a whole,
e.g., we can terminate it by sending a TERM signal <code>kill 93015</code>.
Note, the light weight process whose PID and LWP columns contain 93015 indicates that it was created first (the main thread).</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© ps -aL</span>
<span class="go"> PID LWP TTY TIME CMD</span>
<span class="go"> 93013 93013 pts/0 00:00:00 sudo</span>
<span class="go"> 93015 93015 pts/3 00:00:33 parca-agent</span>
<span class="go"> 93015 93016 pts/3 00:00:24 parca-agent</span>
<span class="go"> 93015 93017 pts/3 00:00:30 parca-agent</span>
<span class="go"> 93015 93018 pts/3 00:00:00 parca-agent</span>
<span class="go"> 93015 93019 pts/3 00:00:00 parca-agent</span>
<span class="go"> 93015 93020 pts/3 00:00:00 parca-agent</span>
<span class="go"> 93015 93021 pts/3 00:00:30 parca-agent</span>
<span class="go"> 93015 93022 pts/3 00:00:32 parca-agent</span>
<span class="go"> 94130 94130 pts/1 00:00:00 ps</span>
</code></pre></div>
<p>Linux kernel maintains a <a href="https://elixir.bootlin.com/linux/v6.1.8/source/include/linux/sched.h#L737">task_struct</a> for each light weight process.
Therefore there should be 8 such structures with <code>tgid = 93015</code>
(thread group ID or PID column)
and <code>pid</code> (LWP column) in a range from 93015 to 93022.
The maximum PID value is 4194304,
see <a href="https://elixir.bootlin.com/linux/v6.1.8/source/include/linux/threads.h#L34">PID_MAX_LIMIT</a>
and <code>/proc/sys/kernel/pid_max</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Possible states: running, interruptible (sleeping),</span>
<span class="w"> </span><span class="c1">// uninterruptible, stopped, traced.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">__state</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Pointer to kernel stack (each task has one),</span>
<span class="w"> </span><span class="c1">// see https://www.kernel.org/doc/html/latest/x86/kernel-stacks.html.</span>
<span class="w"> </span><span class="c1">// General-purpose registers (e.g., rax, rbx)</span>
<span class="w"> </span><span class="c1">// used by a task in user mode are saved on</span>
<span class="w"> </span><span class="c1">// the kernel stack before performing process switching.</span>
<span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="o">*</span><span class="n">stack</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the previous and to the next task_struct</span>
<span class="w"> </span><span class="c1">// thus representing a list of all tasks in the system.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">list_head</span><span class="w"> </span><span class="n">tasks</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to a structure that describes</span>
<span class="w"> </span><span class="c1">// the current state of virtual memory.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">mm_struct</span><span class="w"> </span><span class="o">*</span><span class="n">mm</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Possible exit states: zombie, dead.</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">exit_state</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">exit_code</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ID of this light weight process.</span>
<span class="w"> </span><span class="kt">pid_t</span><span class="w"> </span><span class="n">pid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ID of the thread group (PID column in ps).</span>
<span class="w"> </span><span class="kt">pid_t</span><span class="w"> </span><span class="n">tgid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the task that created this task</span>
<span class="w"> </span><span class="c1">// or to the init task if the parent no longer exists.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="w"> </span><span class="n">__rcu</span><span class="w"> </span><span class="o">*</span><span class="n">real_parent</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// A list of all children created by this task.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">list_head</span><span class="w"> </span><span class="n">children</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the next and previous sibling tasks.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">list_head</span><span class="w"> </span><span class="n">sibling</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the process group leader (PGID),</span>
<span class="w"> </span><span class="c1">// e.g., of "sleep 10 | sleep 20".</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="w"> </span><span class="o">*</span><span class="n">group_leader</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Executable name, excluding path.</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">comm</span><span class="p">[</span><span class="n">TASK_COMM_LEN</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Open file information (contains pointers to file descriptors).</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">files_struct</span><span class="w"> </span><span class="o">*</span><span class="n">files</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// CPU-specific state of this task is stored here</span>
<span class="w"> </span><span class="c1">// when the task is being switched out, i.e.,</span>
<span class="w"> </span><span class="c1">// most of the CPU registers (es, ds, fs, gs, FPU registers),</span>
<span class="w"> </span><span class="c1">// except the general-purpose registers.</span>
<span class="w"> </span><span class="c1">// The sp0 and sp fields are the user and</span>
<span class="w"> </span><span class="c1">// kernel stack pointers respectively.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">thread_struct</span><span class="w"> </span><span class="kr">thread</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>All <code>task_struct</code> that currently present in the system are linked using a double linked list.
When we run <code>kill 93015</code> to terminate 8 light weight processes by <code>tgid</code>,
the kernel looks up the thread group's leader by ID in
a <a href="https://elixir.bootlin.com/linux/v3.19.8/source/kernel/pid.c#L44">hash table</a>
and then walks the list of the group.
That applies at least to the kernel v3, and the v6 seems to be using a radix tree,
see <a href="https://elixir.bootlin.com/linux/v6.1.8/source/kernel/pid.c">pid.c</a>,
<a href="https://elixir.bootlin.com/linux/v6.1.8/source/lib/idr.c">idr.c</a>,
<a href="https://www.kernel.org/doc/html/latest/core-api/idr.html">ID allocation</a>.</p>
<p>With namespaces each PID may have several values, with each one being valid in one namespace.
Linux kernel has a <a href="https://elixir.bootlin.com/linux/v6.1.8/source/include/linux/pid.h">pid</a> structure
that refers to individual tasks, process groups, and sessions.
Check out <a href="https://stackoverflow.com/questions/26779416/what-is-the-relation-between-task-struct-and-pid-namespace">#26779416</a> stackoverflow answer for more details.</p>
<details>
<div class="highlight"><pre><span></span><code><span class="cm">/*</span>
<span class="cm"> * struct upid is used to get the id of the struct pid, as it is</span>
<span class="cm"> * seen in particular namespace. Later the struct pid is found with</span>
<span class="cm"> * find_pid_ns() using the int nr and struct pid_namespace *ns.</span>
<span class="cm"> */</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">upid</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The pid value.</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">nr</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The namespace this value is visible in.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">pid_namespace</span><span class="w"> </span><span class="o">*</span><span class="n">ns</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="cm">/*</span>
<span class="cm"> * What is struct pid?</span>
<span class="cm"> *</span>
<span class="cm"> * A struct pid is the kernel's internal notion of a process identifier.</span>
<span class="cm"> * It refers to individual tasks, process groups, and sessions. While</span>
<span class="cm"> * there are processes attached to it the struct pid lives in a hash</span>
<span class="cm"> * table, so it and then the processes that it refers to can be found</span>
<span class="cm"> * quickly from the numeric pid value. The attached processes may be</span>
<span class="cm"> * quickly accessed by following pointers from struct pid.</span>
<span class="cm"> *</span>
<span class="cm"> * Storing pid_t values in the kernel and referring to them later has a</span>
<span class="cm"> * problem. The process originally with that pid may have exited and the</span>
<span class="cm"> * pid allocator wrapped, and another process could have come along</span>
<span class="cm"> * and been assigned that pid.</span>
<span class="cm"> *</span>
<span class="cm"> * Referring to user space processes by holding a reference to struct</span>
<span class="cm"> * task_struct has a problem. When the user space process exits</span>
<span class="cm"> * the now useless task_struct is still kept. A task_struct plus a</span>
<span class="cm"> * stack consumes around 10K of low kernel memory. More precisely</span>
<span class="cm"> * this is THREAD_SIZE + sizeof(struct task_struct). By comparison</span>
<span class="cm"> * a struct pid is about 64 bytes.</span>
<span class="cm"> *</span>
<span class="cm"> * Holding a reference to struct pid solves both of these problems.</span>
<span class="cm"> * It is small so holding a reference does not consume a lot of</span>
<span class="cm"> * resources, and since a new struct pid is allocated when the numeric pid</span>
<span class="cm"> * value is reused (when pids wrap around) we don't mistakenly refer to new</span>
<span class="cm"> * processes.</span>
<span class="cm"> */</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">pid</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Reference counter.</span>
<span class="w"> </span><span class="n">refcount_t</span><span class="w"> </span><span class="n">count</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The number of upids.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">level</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Lists of tasks that use this pid.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">hlist_head</span><span class="w"> </span><span class="n">tasks</span><span class="p">[</span><span class="n">PIDTYPE_MAX</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">hlist_head</span><span class="w"> </span><span class="n">inodes</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">upid</span><span class="w"> </span><span class="n">numbers</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</code></pre></div>
</details>
<h2>Address space</h2>
<p>A process provides a program an illusion that
it has exclusive use of the whole memory address space in the system
using virtual memory abstraction.</p>
<p>Physical memory is organized as an array of bytes and
it's partitioned into fixed-size blocks
(usually 4KB depending on CPU architecture) called page frames.
A virtual page can be:</p>
<ul>
<li>cached in DRAM β backed by a page frame, i.e.,
4KB of data is already loaded to physical memory from disk</li>
<li>uncached in DRAM β doesn't occupy physical memory yet,
but is already associated with a 4KB part of a file on disk.
Once some virtual address within this virtual page is accessed by the CPU
causing a page fault exception (DRAM cache miss),
the 4KB of data will be loaded from disk to memory.</li>
<li>unallocated β NULL, doesn't point to physical memory or to disk</li>
</ul>
<p>This mapping is stored in a data structure called page table.
A memory management unit (MMU) relies on page tables
to translate virtual addresses to physical addresses.
Here is how a process's virtual address space could approximately look.</p>
<div class="highlight"><pre><span></span><code><span class="go"> _________________________</span>
<span class="go">| page tables, |</span>
<span class="go">| task and mm structs |</span>
<span class="go">|-------------------------|</span>
<span class="go">| kernel stack | β¬οΈ</span>
<span class="go">|-------------------------| rsp (stack pointer in kernel mode)</span>
<span class="go">| thread info struct |</span>
<span class="go">|-------------------------|</span>
<span class="go">| physical memory |</span>
<span class="go">|-------------------------|</span>
<span class="go">| kernel code & data |</span>
<span class="go">|_________________________| top part is reserved for the kernel</span>
<span class="go">| arg, env |</span>
<span class="go">|-------------------------|</span>
<span class="go">| stack for main thread | β¬οΈ</span>
<span class="go">|-------------------------| rsp (stack pointer in user mode)</span>
<span class="go">| ... |</span>
<span class="go">|-------------------------|</span>
<span class="go">| stack for thread 3 | β¬οΈ</span>
<span class="go">|-------------------------|</span>
<span class="go">| stack for thread 2 | β¬οΈ</span>
<span class="go">|-------------------------|</span>
<span class="go">| stack for thread 1 | β¬οΈ</span>
<span class="go">|-------------------------|</span>
<span class="go">| shared libraries |</span>
<span class="go">|-------------------------|</span>
<span class="go">| ... |</span>
<span class="go">|-------------------------| brk (top of the heap)</span>
<span class="go">| heap | β¬οΈ</span>
<span class="go">|-------------------------|</span>
<span class="go">| uninitialized data .bss |</span>
<span class="go">|-------------------------| vm_end</span>
<span class="go">| initialized data .data |</span>
<span class="go">|-------------------------| vm_start</span>
<span class="go">| code .text | thread 3 executing</span>
<span class="go">| | main thread executing</span>
<span class="go">| | thread 1 executing</span>
<span class="go">| | thread 2 executing</span>
<span class="go">| |</span>
<span class="go">|-------------------------| 0x400000</span>
<span class="go">| ... |</span>
<span class="go">|_________________________| 0</span>
</code></pre></div>
<p>The bottom part describes user space addresses of a process.
The top part is a kernel virtual memory:</p>
<ul>
<li>process-specific structures such as page tables, task and mm structs,
kernel stack, thread info structure.</li>
<li>physical memory part which is identical for each process.
Linux maps a set of virtual pages equal in size to DRAM
to the corresponding set of physical pages
to access any specific location in physical memory,
e.g., to access page tables.</li>
<li>kernel code and data which is identical for each process</li>
</ul>
<p>Linux organizes the virtual memory as a collection of areas (segments)
where an area is a chunk of related pages, e.g., code, data,
heap, shared libraries, user stack segments.
A process can create arbitrary number of areas using <code>mmap</code> function.</p>
<p>The <code>vm_area_struct</code> structures keep track of virtual memory areas of a task.
They can be reached via <code>task->mm->mm_mt</code> tree.
The fields <code>vma.vm_start</code> and <code>vma.vm_end</code> point to the beginning and the end of an area.</p>
<details>
<div class="highlight"><pre><span></span><code><span class="c1">// See https://elixir.bootlin.com/linux/v6.1.8/source/include/linux/mm_types.h#L512.</span>
<span class="k">struct</span><span class="w"> </span><span class="nc">mm_struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// A tree to look up vm_area_struct by user address.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">maple_tree</span><span class="w"> </span><span class="n">mm_mt</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Memory mapping.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">mmap_base</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to a page global directory,</span>
<span class="w"> </span><span class="c1">// i.e., to the first entry of the level 1 page table.</span>
<span class="w"> </span><span class="c1">// The physical address of PGD in use is stored in the cr3 control register.</span>
<span class="w"> </span><span class="n">pgd_t</span><span class="w"> </span><span class="o">*</span><span class="n">pgd</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Addresses of text and data segments.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">start_code</span><span class="p">,</span><span class="w"> </span><span class="n">end_code</span><span class="p">,</span><span class="w"> </span><span class="n">start_data</span><span class="p">,</span><span class="w"> </span><span class="n">end_data</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Addresses of heap and stack segments.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">start_brk</span><span class="p">,</span><span class="w"> </span><span class="n">brk</span><span class="p">,</span><span class="w"> </span><span class="n">start_stack</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">arg_start</span><span class="p">,</span><span class="w"> </span><span class="n">arg_end</span><span class="p">,</span><span class="w"> </span><span class="n">env_start</span><span class="p">,</span><span class="w"> </span><span class="n">env_end</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cm">/*</span>
<span class="cm"> * This struct describes a virtual memory area. There is one of these</span>
<span class="cm"> * per VM-area/task. A VM area is any part of the process virtual memory</span>
<span class="cm"> * space that has a special rule for the page-fault handlers (ie a shared</span>
<span class="cm"> * library, the executable area).</span>
<span class="cm"> */</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">vm_area_struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the beginning of the area within vm_mm.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">vm_start</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the end of the area within vm_mm.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">vm_end</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Points to the address space this area belongs to.</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">mm_struct</span><span class="w"> </span><span class="o">*</span><span class="n">vm_mm</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Access permissions of this VMA, e.g., read/write.</span>
<span class="w"> </span><span class="n">pgprot_t</span><span class="w"> </span><span class="n">vm_page_prot</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Describes whether the pages in the area are shared with other tasks</span>
<span class="w"> </span><span class="c1">// or private to this task.</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">vm_flags</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The offset (within vm_file) in PAGE_SIZE units (number of pages).</span>
<span class="w"> </span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">long</span><span class="w"> </span><span class="n">vm_pgoff</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The file backing this mapping.</span>
<span class="w"> </span><span class="c1">// It can be NULL, e.g., anonymous mapping (stack, heap, bss).</span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">file</span><span class="w"> </span><span class="o">*</span><span class="n">vm_file</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
</details>
<p>Linux can associate a virtual memory area with a contiguous section of a file,
e.g., executable object file.
That section (e.g., code segment) is divided into page-size chunks.
When the CPU accesses a virtual address that is within some page's region,
that virtual page (e.g., some content of an executable file)
is loaded to the physical memory from disk.
You can check out an example of a memory mapping in the context of
<a href="https://marselester.com/diy-cpu-profiler-from-bpf-maps-to-pprof.html">CPU profiler</a>.</p>
<p>An area can also be mapped (associated) to
an anonymous file (it doesn't exist on disk).
When the CPU accesses a virtual page within that area (e.g., heap or user stack),
the kernel finds a free page in physical memory, zeroes it,
and marks it as resident in the page table.
If no free pages exist (run out of physical memory),
then some dirty page gets swapped out to disk.</p>
<p>A file can be mapped as the shared object into areas of different processes.
The memory mappings are unique for each process, but the underlying physical pages are the same.
So if one process writes to its area, the changes are visible to other processes,
and the file is updated on disk as well.
Shared libraries are mapped as shared objects by many processes thus saving physical memory pages.</p>
<p>If a file is mapped as the private object,
the writes to that area are not visible to other processes and
the changes are not written to disk.
For example, two processes mapped a private object into their areas of virtual memories.
The same physical memory pages will be used as long as both processes only read data from those areas.
Once a process writes to its area, a new copy of pages is created in physical memory.
This copy-on-write allows to save memory, e.g., multiple processes share .text segment which is never modified
(recall 8 parca-agent light weight processes).</p>
<hr>
<p>To wrap up, I must say that I might have misunderstood some of the parts
and posted inaccurate information, sorry about that.
The Linux kernel is a complicated project π§,
and I wouldn't be able to navigate it without books such as:</p>
<ul>
<li>Computer Systems, Randal E. Bryant, David R. O'Hallaron</li>
<li>Understanding the Linux Kernel, Daniel P. Bovet, Marco Cesati</li>
<li>The Linux Programming Interface: A Linux and UNIX System Programming Handbook, Michael Kerrisk</li>
</ul>Bandwitch π§ββοΈ of CPU and storages2023-01-08T00:00:00+07:002023-01-08T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2023-01-08:/bandwitch-of-cpu-and-storages.html<p>Let us begin with a definition of a CPU clock rate β°.
It refers to the frequency at which the clock generator (an oscillator crystal)
of a processor can generate pulses, which are used to synchronize the operations of its components.
For example, 1GHz CPU implies that its clock runs at β¦</p><p>Let us begin with a definition of a CPU clock rate β°.
It refers to the frequency at which the clock generator (an oscillator crystal)
of a processor can generate pulses, which are used to synchronize the operations of its components.
For example, 1GHz CPU implies that its clock runs at 10^9 cycles per second,
therefore the time required for each clock cycle is 1 nanosecond.
Faster the clock -- more instructions a processor can execute per second.</p>
<p>Performance of a program depends on how fast the data gets accessed,
so the speed of a storage is paramount.
The CPU has the fastest storage device called the register file
which consists of 16 word-sized (64 bits) registers.
The data stored in a register requires 0 cycles to access.</p>
<p>The program counter (PC) <code>rip</code> register contains the memory address of a machine instruction.
CPU reads the instruction from memory pointed at by the PC,
executes the instruction, and updates the PC to point to the next instruction.
In general, CPU performs the following operations as per instruction:</p>
<ul>
<li>load -- copy a byte/word from main memory into register</li>
<li>store -- copy a byte/word from register into main memory</li>
<li>operate -- copy the contents of two registers to the arithmetic/logic unit (ALU),
perform an arithmetic operation, and store the result in a register</li>
<li>jump -- copy a word from the instruction into the PC</li>
</ul>
<p>The memory addresses used by a program are virtual addresses,
so the main memory appears to be a byte array where array index is a memory address.
It can range from 0 to 2^63-1, but currently is
<a href="https://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details">limited to 2^48 or 256TB</a>.
The lower-addressed half (user space) is occupied by a process,
and the rest is left to kernel virtual memory (it is identical for each process).
Here is how a 4-byte integer might be stored in memory (little endian byte ordering).</p>
<div class="highlight"><pre><span></span><code><span class="c1">// The variable's address &x is 0x100.</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">x</span><span class="w"> </span><span class="kt">int32</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0x00001e61</span><span class="w"></span>
</code></pre></div>
<table>
<thead>
<tr>
<th>memory address</th>
<th>byte</th>
</tr>
</thead>
<tbody>
<tr>
<td>0x100</td>
<td>61</td>
</tr>
<tr>
<td>0x101</td>
<td>1e</td>
</tr>
<tr>
<td>0x102</td>
<td>00</td>
</tr>
<tr>
<td>0x103</td>
<td>00</td>
</tr>
</tbody>
</table>
<p>Arrays and structures are stored as a contiguous sequence of bytes.
The smallest memory address of the bytes used is a variable's address.</p>
<p>Memory access is often the slowest of CPU operations, as it may take ~200 cycles to access it,
during which instruction execution has stalled.
Memory caches blocks of virtual memory (4-KB pages) and parts of files (buffer cache).
Disk controller caches disk sectors.
Its access time is ~100,000 cycles, and disk's access time is ~10,000,000 cycles.
Disks read and write data in sector-size blocks (512 bytes) which takes ~10ms.
CPU L1-L3 caches cache 64-byte blocks (cache lines) of the main memory
to significantly reduce the number of cycles needed for memory access.
L1 cache can be accessed in ~4 clock cycles, L2 in ~10 cycles, and L3's access time is ~50 cycles.</p>
<p>Buses transfer word-sized information between CPU, main memory, disk, etc.
It takes ~256ns for L1 cache to read 512 bytes (access time is ~4ns for a 64-bit word).</p>
<div class="highlight"><pre><span></span><code><span class="go">Register file -- L1-3 caches -- Bus interface</span>
<span class="go"> |</span>
<span class="go"> | System bus</span>
<span class="go"> |</span>
<span class="go">Disk -- Disk controller --------- I/O bridge</span>
<span class="go"> I/O bus |</span>
<span class="go"> | Memory bus</span>
<span class="go"> |</span>
<span class="go"> Main memory</span>
</code></pre></div>
<p>Each CPU core has its own L1 and L2 caches.
L1 is split into i-cache to read recently fetched instructions, and d-cache to read/write data.
All cores share L3 cache and the interface to main memory.</p>
<div class="highlight"><pre><span></span><code><span class="go">Registers Registers</span>
<span class="go"> | |</span>
<span class="go">L1 d-cache L1 i-cache ... L1 d-cache L1 i-cache</span>
<span class="go"> \/ \/</span>
<span class="go"> L2 cache L2 cache</span>
<span class="go"> \ /</span>
<span class="go"> \-- L3 cache --/</span>
</code></pre></div>
<p>An instruction requires ~20 clock cycles to execute,
but a processor manages to execute more of them per clock cycle using instruction pipelining,
i.e., the actions are partitioned into ~15 steps which are performed in parallel
working on different instructions, e.g.,</p>
<ul>
<li>fetching the instruction from memory</li>
<li>determining the instruction type</li>
<li>reading from memory</li>
<li>arithmetic operation</li>
<li>writing to memory</li>
<li>updating the program counter</li>
</ul>
<p>CPU can perform out-of-order execution of the pipeline,
where later instructions can be completed while earlier instructions are stalled,
improving instruction throughput.
The instruction control unit (ICU) fetches instructions from L1 i-cache ahead of time.
It generates a sequence of primitive operations from instructions
and sends them to the execution unit (EU).
EU executes the operations using multiple functional units
(e.g., store to L1 d-cache) some of which of the same type
and tells ICU whether branches were correctly predicted.
Mispredicting a conditional jump can incur ~15-30 clock cycles of wasted effort, hurting performance.</p>
<p>Memory management unit (MMU) uses translation look-aside buffer (TLB) which is a cache
that provides a fast translation (0 cycles access time) from virtual to physical addresses.
On TLB miss, MMU searches the address mapping in the page table which is maintained by the kernel.
When it fails, a page fault exception is signalled to get a referenced memory address from the disk.
The OS's exception handler sets up a transfer from disk to memory (several thousands cycles).
Once this completes (millions of cycles),
the OS will return to the original program to retry the instruction that caused a page fault.
Direct memory access (DMA) technique allows data to travel directly from disk to memory avoiding CPU.</p>
<h2>General advice</h2>
<p>Intel Vtune defines CPI (cycles per instruction retired) as a metric indicating how much time
each executed instruction took, in units of cycles.
The CPU issues up to four instructions per cycle, suggesting a theoretical best CPI of 0.25.
A CPI < 1 is typical for instruction bound code,
while a CPI > 1 may show up for a stall cycle bound application, also likely memory bound.
If you're curious, check out a <a href="https://github.com/klauspost/compress/discussions/717">Vtune perf report</a> example.</p>
<p>Hyper-threading allows the CPU core to run more than one thread,
effectively scheduling between them when one instruction stalls on memory I/O.
Workloads that are stall cycle-heavy (low IPC instructions per cycle)
could have better performance than those that are instruction-heavy
because stall cycles reduce core contention.</p>
<p>IPC < 1 (or CPI > 1) indicates that the CPU is often stalled,
typically for memory access, so a general advice is to:</p>
<ul>
<li>keep intermediate calculation values in registers (i.e., local variables) instead of the memory
and store results in memory when the final value was computed</li>
<li>use a value as often as possible once it has been read from memory (temporal locality),
i.e., it will be read from L1 cache</li>
<li>access data (arrays, structs) sequentially in the order they are stored in memory (spatial locality),
so multiple values are read from the same L1 cache line</li>
<li>reduce the data dependencies in a loop between different parts of a computation.
Otherwise the CPU would have to wait for dependent load/store operations that form a critical path.
Similar to databases, partitioning helps, e.g., introduce more variables to accumulate computed values.</li>
</ul>
<p>Beware of false sharing -- a usage pattern occurring when two CPU cores read from and write to
unrelated variables that happen to share the same L1 cache line,
e.g., two pointers declared in the same struct next to each other.
When write happens in one core,
the cache controller invalidates that cache line in other cores,
so they have to reload non-stale data to do their reads.
The solution is to add 64-byte padding between those variables so they end up in different cache lines.
Check out an interesting case study:
<a href="https://surfingcomplexity.blog/2022/11/25/cache-invalidation-really-is-one-of-the-hardest-things-in-computer-science/">Cache invalidation really is one of the hardest problems in computer science</a>
and <a href="https://netflixtechblog.com/seeing-through-hardware-counters-a-journey-to-threefold-performance-increase-2721924a2822">Seeing through hardware counters: a journey to threefold performance increase</a>.
That post also mentions true sharing -- a usage pattern occurring
when multiple cores read from and write to the same variable.
The CPU-enforced memory ordering causes slowdown
(<a href="https://go-talks.appspot.com/github.com/marselester/scalability/scalability.slide#3">coherency-limited scalability</a> due to inconsistent copies of data).</p>
<p>If you're curious about systems performance, I recommend Brendan Gregg's books,
Chris Kanich's <a href="https://www.youtube.com/@ChrisKanich">YouTube channel</a>,
Computer Systems book by Bryant and O'Hallaron.</p>DIY CPU profiler: from BPF maps to pprof2022-10-20T00:00:00+07:002022-10-20T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2022-10-20:/diy-cpu-profiler-from-bpf-maps-to-pprof.html<p><a href="https://marselester.com/continuous-profiling-in-go.html">In previous post</a>
I had a DIY BPF profiler printing stack trace IDs of a given process,
though they were not very helpful.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/profiler/ -pid 15958</span>
<span class="go">Waiting for stack traces...</span>
<span class="go">{PID:15958 UserStackID:132 KernelStackID:114} seen 1 times</span>
</code></pre></div>
<p>Let's try to show more useful information β¦</p><p><a href="https://marselester.com/continuous-profiling-in-go.html">In previous post</a>
I had a DIY BPF profiler printing stack trace IDs of a given process,
though they were not very helpful.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/profiler/ -pid 15958</span>
<span class="go">Waiting for stack traces...</span>
<span class="go">{PID:15958 UserStackID:132 KernelStackID:114} seen 1 times</span>
</code></pre></div>
<p>Let's try to show more useful information this time,
i.e., sampled memory addresses.
You can find the results of my experiments in
<a href="https://github.com/marselester/diy-parca-agent">github.com/marselester/diy-parca-agent</a>.</p>
<h2>Reading addresses from BPF maps</h2>
<p>I continued to explore
<a href="https://github.com/parca-dev/parca-agent/blob/1e935f4d5f7b4484f6cf3d4ee26340f3718ff37d/pkg/profiler/profiler.go#L336">github.com/parca-dev/parca-agent/pkg/profiler/profiler.go</a>
and found the place where stack trace samples are read from the BPF maps β <code>profileLoop()</code> method.
Here is its simplified version.</p>
<div class="highlight"><pre><span></span><code><span class="kd">const</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Always needs to be sync with MAX_STACK_DEPTH in parca-agent.bpf.c.</span><span class="w"></span>
<span class="w"> </span><span class="nx">stackDepth</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">127</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Twice the stack depth because we have a User and a potential Kernel stack.</span><span class="w"></span>
<span class="w"> </span><span class="nx">doubleStackDepth</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">254</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="o">*</span><span class="nx">CgroupProfiler</span><span class="p">)</span><span class="w"> </span><span class="nx">profileLoop</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">captureTime</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">it</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">p</span><span class="p">.</span><span class="nx">bpfMaps</span><span class="p">.</span><span class="nx">counts</span><span class="p">.</span><span class="nx">Iterator</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">it</span><span class="p">.</span><span class="nx">Next</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Note, pid, userStackID, and kernelStackID are read from it.Key().</span><span class="w"></span>
<span class="w"> </span><span class="nx">stack</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[</span><span class="nx">doubleStackDepth</span><span class="p">]</span><span class="kt">uint64</span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Collect user-space stack trace samples.</span><span class="w"></span>
<span class="w"> </span><span class="nx">stackBytes</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">p</span><span class="p">.</span><span class="nx">bpfMaps</span><span class="p">.</span><span class="nx">stackTraces</span><span class="p">.</span><span class="nx">GetValue</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">unsafe</span><span class="p">.</span><span class="nx">Pointer</span><span class="p">(</span><span class="o">&</span><span class="nx">userStackID</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">Read</span><span class="p">(</span><span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBuffer</span><span class="p">(</span><span class="nx">stackBytes</span><span class="p">),</span><span class="w"> </span><span class="nx">byteOrder</span><span class="p">,</span><span class="w"> </span><span class="nx">stack</span><span class="p">[:</span><span class="nx">stackDepth</span><span class="p">])</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">stack</span><span class="p">[:</span><span class="nx">stackDepth</span><span class="p">]</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">m</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">mapping</span><span class="p">.</span><span class="nx">PIDAddrMapping</span><span class="p">(</span><span class="nx">pid</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Collect kernel-space stack trace samples.</span><span class="w"></span>
<span class="w"> </span><span class="nx">stackBytes</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">p</span><span class="p">.</span><span class="nx">bpfMaps</span><span class="p">.</span><span class="nx">stackTraces</span><span class="p">.</span><span class="nx">GetValue</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">unsafe</span><span class="p">.</span><span class="nx">Pointer</span><span class="p">(</span><span class="o">&</span><span class="nx">kernelStackID</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">Read</span><span class="p">(</span><span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBuffer</span><span class="p">(</span><span class="nx">stackBytes</span><span class="p">),</span><span class="w"> </span><span class="nx">byteOrder</span><span class="p">,</span><span class="w"> </span><span class="nx">stack</span><span class="p">[</span><span class="nx">stackDepth</span><span class="p">:])</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">stack</span><span class="p">[</span><span class="nx">stackDepth</span><span class="p">:]</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>From looking at it I can tell that PID, user-space stack ID, and kernel-space stack ID are read
from the <code>counts</code> BPF map on each iteration.</p>
<table>
<thead>
<tr>
<th>{ pid; userStackID; kernelStackID }</th>
<th>seen</th>
</tr>
</thead>
<tbody>
<tr>
<td>{ 10342; 1253; 0234 }</td>
<td>45</td>
</tr>
</tbody>
</table>
<p>A stack ID allows to fetch a stack trace
(an array of memory addresses that represent the code executed during profiling)
from the <code>stack_traces</code> BPF map.</p>
<table>
<thead>
<tr>
<th>stack ID</th>
<th>memory addresses</th>
</tr>
</thead>
<tbody>
<tr>
<td>1253</td>
<td>[ 0xdeadbeef; 0x123abcde; ... ]</td>
</tr>
<tr>
<td>0234</td>
<td>[ 0x597be95a; 0xae5ee03; ... ]</td>
</tr>
</tbody>
</table>
<p>A user-space stack trace indexed by <code>userStackID=1253</code> is placed
in the beginning of the <code>stack [254]uint64</code> array.
The <code>kernelStackID=0234</code> kernel-space stack trace is placed in the middle of the array.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// User-space stack trace is limited by stackDepth=127.</span><span class="w"></span>
<span class="nx">stack</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0xdeadbeef</span><span class="w"></span>
<span class="nx">stack</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0x123abcde</span><span class="w"></span>
<span class="nx">stack</span><span class="p">[</span><span class="mi">126</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">...</span><span class="w"></span>
<span class="c1">// Kernel-space stack trace is limited by doubleStackDepth=254.</span><span class="w"></span>
<span class="nx">stack</span><span class="p">[</span><span class="mi">127</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0x597be95a</span><span class="w"></span>
<span class="nx">stack</span><span class="p">[</span><span class="mi">128</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mh">0xae5ee03</span><span class="w"></span>
<span class="nx">stack</span><span class="p">[</span><span class="mi">253</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="o">...</span><span class="w"></span>
</code></pre></div>
<h2>Memory mapping</h2>
<p>Alright, but what does
<code>m, err := mapping.PIDAddrMapping(pid, addr)</code> line do?
Basically the function call does the following
(see <a href="https://github.com/parca-dev/parca-agent/blob/1e935f4d5f7b4484f6cf3d4ee26340f3718ff37d/pkg/maps/maps.go#L85">maps.go</a>):</p>
<ol>
<li>opens a memory map file of a given process <code>/proc/PID/maps</code></li>
<li>parses a memory map using <code>github.com/google/pprof/profile</code> package</li>
<li>looks up a mapping for a given memory address previously found in a stack trace (e.g., <code>0xdeadbeef</code>),
i.e., it should be located between <code>m.Start</code> and <code>m.Limit</code> inclusively.
For example, in <code>[0x55dd2d5eb000; 0x55dd2d65f000]</code> interval.</li>
</ol>
<div class="highlight"><pre><span></span><code><span class="nx">f</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Open</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"/proc/%d/maps"</span><span class="p">,</span><span class="w"> </span><span class="nx">pid</span><span class="p">),</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="nx">mm</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">profile</span><span class="p">.</span><span class="nx">ParseProcMaps</span><span class="p">(</span><span class="nx">f</span><span class="p">)</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">m</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">mm</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">Start</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">Limit</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">m</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Note, <code>mm</code> is a slice of <code>*profile.Mapping{}</code> structs
that could look as it's shown below.</p>
<div class="highlight"><pre><span></span><code><span class="nx">mm</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">[]</span><span class="o">*</span><span class="nx">profile</span><span class="p">.</span><span class="nx">Mapping</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Start</span><span class="p">:</span><span class="w"> </span><span class="mi">94408437313536</span><span class="p">,</span><span class="w"> </span><span class="c1">// 0x55dd2d5eb000</span><span class="w"></span>
<span class="w"> </span><span class="nx">Limit</span><span class="p">:</span><span class="w"> </span><span class="mi">94408437788672</span><span class="p">,</span><span class="w"> </span><span class="c1">// 0x55dd2d65f000</span><span class="w"></span>
<span class="w"> </span><span class="nx">Offset</span><span class="p">:</span><span class="w"> </span><span class="mi">45056</span><span class="p">,</span><span class="w"> </span><span class="c1">// 0xb000</span><span class="w"></span>
<span class="w"> </span><span class="nx">File</span><span class="p">:</span><span class="w"> </span><span class="s">"/usr/sbin/sshd"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">BuildID</span><span class="p">:</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasFunctions</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasFilenames</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasLineNumbers</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasInlineFrames</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">KernelRelocationSymbol</span><span class="p">:</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Start</span><span class="p">:</span><span class="w"> </span><span class="mi">140349706260480</span><span class="p">,</span><span class="w"> </span><span class="c1">// 0x7fa5b662d000</span><span class="w"></span>
<span class="w"> </span><span class="nx">Limit</span><span class="p">:</span><span class="w"> </span><span class="mi">140349706448896</span><span class="p">,</span><span class="w"> </span><span class="c1">// 0x7fa5b665b000</span><span class="w"></span>
<span class="w"> </span><span class="nx">Offset</span><span class="p">:</span><span class="w"> </span><span class="mi">24576</span><span class="p">,</span><span class="w"> </span><span class="c1">// 0x6000</span><span class="w"></span>
<span class="w"> </span><span class="nx">File</span><span class="p">:</span><span class="w"> </span><span class="s">"/usr/lib/x86_64-linux-gnu/libnss_systemd.so.2"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">BuildID</span><span class="p">:</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasFunctions</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasFilenames</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasLineNumbers</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HasInlineFrames</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">KernelRelocationSymbol</span><span class="p">:</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ...</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Here is the corresponding memory map file.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo cat /proc/4068/maps</span>
<span class="go">55dd2d5e0000-55dd2d5eb000 r--p 00000000 08:01 2869 /usr/sbin/sshd</span>
<span class="go">55dd2d5eb000-55dd2d65f000 r-xp 0000b000 08:01 2869 /usr/sbin/sshd</span>
<span class="go">55dd2d65f000-55dd2d69d000 r--p 0007f000 08:01 2869 /usr/sbin/sshd</span>
<span class="go">55dd2d69e000-55dd2d6a1000 r--p 000bd000 08:01 2869 /usr/sbin/sshd</span>
<span class="go">55dd2d6a1000-55dd2d6a2000 rw-p 000c0000 08:01 2869 /usr/sbin/sshd</span>
<span class="go">55dd2d6a2000-55dd2d6a6000 rw-p 00000000 00:00 0</span>
<span class="go">55dd2e9d6000-55dd2ea68000 rw-p 00000000 00:00 0 [heap]</span>
<span class="go">55dd2ea68000-55dd2ea8a000 rw-p 00000000 00:00 0 [heap]</span>
<span class="go">7fa5b6627000-7fa5b662d000 r--p 00000000 08:01 4761 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2</span>
<span class="go">7fa5b662d000-7fa5b665b000 r-xp 00006000 08:01 4761 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2</span>
<span class="go">7fa5b665b000-7fa5b666a000 r--p 00034000 08:01 4761 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2</span>
<span class="go">7fa5b666a000-7fa5b666e000 r--p 00042000 08:01 4761 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2</span>
<span class="go">7fa5b666e000-7fa5b666f000 rw-p 00046000 08:01 4761 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2</span>
<span class="go">7fa5b6675000-7fa5b6676000 r--p 00000000 08:01 3974 /usr/lib/x86_64-linux-gnu/security/pam_env.so</span>
<span class="go">...</span>
<span class="go">7fa5b72f4000-7fa5b72f6000 rw-p 00034000 08:01 3657 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2</span>
<span class="go">7fff5522d000-7fff55271000 rw-p 00000000 00:00 0 [stack]</span>
<span class="go">7fff55285000-7fff55289000 r--p 00000000 00:00 0 [vvar]</span>
<span class="go">7fff55289000-7fff5528b000 r-xp 00000000 00:00 0 [vdso]</span>
<span class="go">ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]</span>
</code></pre></div>
<p>π€</p>
<p>Let's do a similar thing in a new profiler called profiler2.
The v2 DIY profiler is able to print a memory map file of a given process
and show the traced memory addresses of user and kernel space.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/profiler2 -pid=4068</span>
<span class="go">/proc/4068/maps</span>
<span class="go">start=0x55dd2d5eb000 limit=0x55dd2d65f000 offset=0xb000 /usr/sbin/sshd</span>
<span class="go">...</span>
<span class="go">Waiting for stack traces...</span>
</code></pre></div>
<h2>pprof</h2>
<p>It would be great to finally see symbols (function names) instead of memory addresses
and pprof is going to help us with that.
pprof is a tool for visualization and analysis of profiling data.
It reads a collection of profiling samples in
<a href="https://github.com/google/pprof/blob/main/proto/profile.proto">profile.proto</a> format
and generates reports to visualize the collected profiles.</p>
<p>Let's get back to <code>profileLoop()</code> method and see what else it's doing:</p>
<ol>
<li>it prepares <code>prof := profile.Profile{}</code> struct which is an in-memory representation of profile.proto</li>
<li>using addresses from BPF maps it fills <code>Profile.Location</code> β the set of program locations;
each location describes function and line table debug information</li>
<li>it fills <code>Profile.Sample</code> β the set of samples recorded in a profile</li>
<li>it fills <code>Profile.Mapping</code> β mapping from address ranges to the object file</li>
<li>it uploads object files to Parca server</li>
<li>it uploads pprof profile</li>
</ol>
<p>The first step is to create a CPU profile which will be saved somewhere by <code>prof.Write()</code>
after a profile was filled with samples.</p>
<div class="highlight"><pre><span></span><code><span class="nx">prof</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">profile</span><span class="p">.</span><span class="nx">Profile</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// SampleType is a description of the samples associated with each Sample.Value.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// By convention the number of events should use unit "count" and</span><span class="w"></span>
<span class="w"> </span><span class="c1">// it should be at index 0.</span><span class="w"></span>
<span class="w"> </span><span class="nx">SampleType</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="o">*</span><span class="nx">profile</span><span class="p">.</span><span class="nx">ValueType</span><span class="p">{{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Type</span><span class="p">:</span><span class="w"> </span><span class="s">"samples"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Unit</span><span class="p">:</span><span class="w"> </span><span class="s">"count"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}},</span><span class="w"></span>
<span class="w"> </span><span class="c1">// TimeNanos is a time of collection (UTC) represented as nanoseconds past the epoch.</span><span class="w"></span>
<span class="w"> </span><span class="nx">TimeNanos</span><span class="p">:</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">().</span><span class="nx">UnixNano</span><span class="p">(),</span><span class="w"></span>
<span class="w"> </span><span class="c1">// DurationNanos is a profiling duration in nanoseconds, Parca uses 10s by default.</span><span class="w"></span>
<span class="w"> </span><span class="nx">DurationNanos</span><span class="p">:</span><span class="w"> </span><span class="nb">int64</span><span class="p">(</span><span class="mi">10</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PeriodType is the kind of events between sampled ocurrences,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// i.e., ["cpu", "nanoseconds"].</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ValueType describes the semantics and measurement units of a value.</span><span class="w"></span>
<span class="w"> </span><span class="nx">PeriodType</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">profile</span><span class="p">.</span><span class="nx">ValueType</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Type</span><span class="p">:</span><span class="w"> </span><span class="s">"cpu"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Unit</span><span class="p">:</span><span class="w"> </span><span class="s">"nanoseconds"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Period is the number of events between sampled occurrences.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We sample at 100Hz (100 times per second),</span><span class="w"></span>
<span class="w"> </span><span class="c1">// which is every 0.01s or 10 million nanoseconds.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Period</span><span class="p">:</span><span class="w"> </span><span class="mi">10000000</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>A simplified version of the steps 2 to 4 show
how user-space stack traces (samples) are added to a pprof profile.</p>
<p>For every non-zero unique address found in stack traces
we create a <code>profile.Location{}</code> to describe a function.
A corresponding memory mapping is added to that newly created location
("Memory mapping" section above shows how to find a mapping for an address).
All the locations are appended to <code>prof.Location</code> slice.</p>
<div class="highlight"><pre><span></span><code><span class="nx">prof</span><span class="p">.</span><span class="nx">Location</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">prof</span><span class="p">.</span><span class="nx">Location</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">profile</span><span class="p">.</span><span class="nx">Location</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ID is a unique nonzero id for the location.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">locIndex</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Address is the instruction address for this location.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// It should be within [Mapping.memory_start...Mapping.memory_limit]</span><span class="w"></span>
<span class="w"> </span><span class="c1">// for the corresponding mapping.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Address</span><span class="p">:</span><span class="w"> </span><span class="nx">addr</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Mapping is the corresponding profile.Mapping for this location.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// It can be nil if the mapping is unknown.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Mapping</span><span class="p">:</span><span class="w"> </span><span class="nx">m</span><span class="p">,</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
</code></pre></div>
<p>Locations related to a given stack trace are added to a sample.
Each <code>profile.Sample</code> records how many times a particular stack trace was seen.</p>
<div class="highlight"><pre><span></span><code><span class="nx">prof</span><span class="p">.</span><span class="nx">Sample</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">prof</span><span class="p">.</span><span class="nx">Sample</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">profile</span><span class="p">.</span><span class="nx">Sample</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="kt">int64</span><span class="p">{</span><span class="nb">int64</span><span class="p">(</span><span class="nx">seen</span><span class="p">)},</span><span class="w"></span>
<span class="w"> </span><span class="nx">Location</span><span class="p">:</span><span class="w"> </span><span class="nx">sampleLocations</span><span class="p">,</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
</code></pre></div>
<p>At the end, all the mappings get assigned IDs starting with one
because ID zero is reserved.
The pprof tool will show the following error if ID zero is used:
<code>malformed profile: found mapping with reserved ID=0</code>.
See the whole code snippet in the details below.</p>
<details>
<div class="highlight"><pre><span></span><code><span class="kd">type</span><span class="w"> </span><span class="nx">locationIndex</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">PID</span><span class="w"> </span><span class="kt">int</span><span class="w"></span>
<span class="w"> </span><span class="nx">Addr</span><span class="w"> </span><span class="kt">uint64</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// locationIndices maps {pid; addr} found in the sample</span><span class="w"></span>
<span class="c1">// to an index in the Profile.Location slice</span><span class="w"></span>
<span class="c1">// to look up a respective Location.</span><span class="w"></span>
<span class="nx">locationIndices</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="nx">locationIndex</span><span class="p">]</span><span class="kt">int</span><span class="p">)</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="nx">it</span><span class="p">.</span><span class="nx">Next</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Collect user-space stack trace samples.</span><span class="w"></span>
<span class="w"> </span><span class="nx">sampleLocations</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="p">[]</span><span class="o">*</span><span class="nx">profile</span><span class="p">.</span><span class="nx">Location</span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">userStack</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">continue</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Look up a location for the given PID and address</span><span class="w"></span>
<span class="w"> </span><span class="c1">// and append it to the current sample's locations.</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// In case a location wasn't found, create one.</span><span class="w"></span>
<span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">locationIndex</span><span class="p">{</span><span class="nx">pid</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">locIndex</span><span class="p">,</span><span class="w"> </span><span class="nx">found</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">locationIndices</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">found</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">m</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">mappingForAddr</span><span class="p">(</span><span class="nx">mm</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">m</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to get process mapping for addr: %x"</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">locIndex</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">prof</span><span class="p">.</span><span class="nx">Location</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Each Location describes function and line table debug information.</span><span class="w"></span>
<span class="w"> </span><span class="nx">prof</span><span class="p">.</span><span class="nx">Location</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">prof</span><span class="p">.</span><span class="nx">Location</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">profile</span><span class="p">.</span><span class="nx">Location</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ID is a unique nonzero id for the location.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">locIndex</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Address is the instruction address for this location.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// It should be within [Mapping.memory_start...Mapping.memory_limit]</span><span class="w"></span>
<span class="w"> </span><span class="c1">// for the corresponding mapping.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Address</span><span class="p">:</span><span class="w"> </span><span class="nx">addr</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Mapping is the corresponding profile.Mapping for this location.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// It can be nil if the mapping is unknown.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Mapping</span><span class="p">:</span><span class="w"> </span><span class="nx">m</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="nx">locationIndices</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">locIndex</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">sampleLocations</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">sampleLocations</span><span class="p">,</span><span class="w"> </span><span class="nx">prof</span><span class="p">.</span><span class="nx">Location</span><span class="p">[</span><span class="nx">locIndex</span><span class="p">])</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Each Sample records how many times a particular stack trace was seen</span><span class="w"></span>
<span class="w"> </span><span class="c1">// along with the corresponding locations.</span><span class="w"></span>
<span class="w"> </span><span class="nx">prof</span><span class="p">.</span><span class="nx">Sample</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">prof</span><span class="p">.</span><span class="nx">Sample</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">profile</span><span class="p">.</span><span class="nx">Sample</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="kt">int64</span><span class="p">{</span><span class="nb">int64</span><span class="p">(</span><span class="nx">seen</span><span class="p">)},</span><span class="w"></span>
<span class="w"> </span><span class="nx">Location</span><span class="p">:</span><span class="w"> </span><span class="nx">sampleLocations</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// Mappings in pprof must have IDs and need to start with 1.</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="nx">i</span><span class="p">,</span><span class="w"> </span><span class="nx">m</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">mm</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">ID</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">uint64</span><span class="p">(</span><span class="nx">i</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="nx">prof</span><span class="p">.</span><span class="nx">Mapping</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">mm</span><span class="w"></span>
</code></pre></div>
</details>
<p>π§ͺ</p>
<p>Putting those steps together gives us profiler3.
In v3 DIY profiler we write a CPU profile to <code>cpu.pprof</code> file
instead of uploading it to a Parca server.
Let's run profiler3 and make sure the profiled program does something instead of idling,
so there will be enough CPU samples.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/profiler3 -pid=4068</span>
<span class="go">/proc/4068/maps</span>
<span class="go">start=0x55dd2d5eb000 limit=0x55dd2d65f000 offset=0xb000 /usr/sbin/sshd</span>
<span class="go">...</span>
<span class="go">Waiting for stack traces for 10s...</span>
</code></pre></div>
<p>After ten seconds the profiler will terminate leaving you with <code>cpu.pprof</code> file.
You can inspect it with <code>go tool pprof cpu.pprof</code> command.</p>
<p>Exploring Parca Agent's internals and reconstructing it in the simplest form
helped me demystify some of aspects of BPF CPU profilers.</p>Continuous profiling in Go2022-04-22T00:00:00+07:002022-04-22T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2022-04-22:/continuous-profiling-in-go.html<p>In this post I explore possibilities of continuous profiling of Go programs
using Parca and also peek under the hood of its BPF agent.
You can find the results of my experiments in
<a href="https://github.com/marselester/diy-parca-agent">github.com/marselester/diy-parca-agent</a>.</p>
<h2>Ad hoc profiling</h2>
<p>I was looking for a way to profile Go programs β¦</p><p>In this post I explore possibilities of continuous profiling of Go programs
using Parca and also peek under the hood of its BPF agent.
You can find the results of my experiments in
<a href="https://github.com/marselester/diy-parca-agent">github.com/marselester/diy-parca-agent</a>.</p>
<h2>Ad hoc profiling</h2>
<p>I was looking for a way to profile Go programs running in Kubernetes
to see where a program spends its CPU time (which functions consume most CPU)
or where memory allocations happen.
Allocations in a frequently called function could cause GC pressure
which in turn could cause latency spikes in HTTP requests.</p>
<p>Latency of the Bookstore service is taken seriously,
so its API server already had <code>import _ "net/http/pprof"</code> to enable profiling with <code>go tool pprof</code> program.
In order to get access to the HTTP server running in the pod
a port forwarder could be used.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© kubectl get pods -A | grep bookstore</span>
<span class="go">default bookstore-574897d6-e18e</span>
<span class="go">οΉ© kubectl port-forward -n default bookstore-574897d6-e18e 6060 &</span>
<span class="go">Forwarding from 127.0.0.1:6060 -> 6060</span>
<span class="go">Forwarding from [::1]:6060 -> 6060</span>
<span class="go">οΉ© fg 1</span>
</code></pre></div>
<p>The following URLs under <code>/debug/pprof/</code> provide profiling data.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΌ 30-second CPU profile.</span>
<span class="go">οΉ© curl http://0.0.0.0:6060/debug/pprof/profile --output ./cpu.pprof</span>
<span class="go">οΌ Heap allocations profile since the process started</span>
<span class="go">οΌ including garbage-collected bytes (useful when optimising code).</span>
<span class="go">οΉ© curl http://0.0.0.0:6060/debug/pprof/allocs --output ./heap-all.pprof</span>
<span class="go">οΌ Heap allocations of live objects (useful when looking for memory leaks).</span>
<span class="go">οΉ© curl http://0.0.0.0:6060/debug/pprof/heap --output ./heap-live.pprof</span>
</code></pre></div>
<p>In order to inspect the heap profile <code>heap-all.pprof</code>
I would need to compile the API <code>server</code> program and fix the search path for source files:</p>
<ul>
<li><code>source_path</code> is an absolute path to your source code (the Bookstore's git repository),
e.g., on Mac it could be <code>/Users/bob/code/bookstore/</code>.</li>
<li><code>trim_path</code> is a path the profile expects the code to be at, e.g.,
<code>/opt/bookstore/</code> because the binary was compiled in Docker by CI/CD</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© GOOS=linux go build ./cmd/server/</span>
<span class="go">οΉ© go tool pprof -source_path="$(pwd)" -trim_path=/opt/bookstore/ ./server ./heap-all.pprof</span>
<span class="go">οΉͺ web</span>
<span class="go">οΉͺ top10 -cum</span>
</code></pre></div>
<p>The <code>web</code> command opens an allocation graph of function calls.
The <code>top10 -cum</code> command shows top ten functions by allocations in that function
or a function it called (cumulative) down the stack:</p>
<ul>
<li><code>flat</code> is a memory allocated by that function and is held by it</li>
<li><code>cum</code> is a memory allocated by that function or a function it called down the stack.
When <code>flat</code> and <code>cum</code> numbers match,
this might indicate the allocated memory is retained.</li>
</ul>
<h2>Continuous profiling</h2>
<p>I could be lucky and discover some obvious problem,
though most likely by the time the profiles are collected it is already too late
because a Kubernetes pod in question is already gone.</p>
<p>A continuous profiler could have collected the data
when a problem was occuring on production,
so I was curious to see what open source tooling is available.
Here is what I found so far:</p>
<ul>
<li><a href="https://www.parca.dev/">Parca</a></li>
<li><a href="https://pyroscope.io/">Pyroscope</a></li>
<li><a href="https://github.com/profefe/profefe">profefe</a></li>
</ul>
<p>All of them are capable, moreover Parca and Pyroscope have BPF agents.
I experimented with Parca more because I stumbled on it earlier.
So here is how Parca Server configuration looked like
when I set it up to collect profiles every minute from the Bookstore service.</p>
<div class="highlight"><pre><span></span><code><span class="nt">debug_info</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">bucket</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="s">"FILESYSTEM"</span><span class="w"></span>
<span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">directory</span><span class="p">:</span><span class="w"> </span><span class="s">"./tmp"</span><span class="w"></span>
<span class="w"> </span><span class="nt">cache</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="s">"FILESYSTEM"</span><span class="w"></span>
<span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">directory</span><span class="p">:</span><span class="w"> </span><span class="s">"./tmp"</span><span class="w"></span>
<span class="nt">scrape_configs</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s">"bookstore"</span><span class="w"></span>
<span class="w"> </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="s">"1m"</span><span class="w"></span>
<span class="w"> </span><span class="nt">scrape_timeout</span><span class="p">:</span><span class="w"> </span><span class="s">"15s"</span><span class="w"></span>
<span class="w"> </span><span class="nt">static_configs</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="w"> </span><span class="s">'bookstore.default:6060'</span><span class="w"> </span><span class="p p-Indicator">]</span><span class="w"></span>
</code></pre></div>
<p>Note, the config above is part of
<a href="https://github.com/parca-dev/parca/releases/download/v0.10.0/kubernetes-manifest.yaml">kubernetes-manifest.yaml</a>.
Check out <a href="https://www.parca.dev/docs/kubernetes">Parca tutorial</a> to learn more.</p>
<h2>BPF profiling agent</h2>
<p>The cool things about BPF profiling agents are that they incur low overhead,
can capture user/kernel-space stack traces, and don't need much configuration
because they discover containers on the Kubernetes node where an agent is running.</p>
<p>A quick summary about BPF.
In the kernel, a BPF program is executed on each event.
The event types could be kprobes, uprobes, tracepoints, sockets, perf_events.
It can use BPF helpers to fetch kernel state, and BPF maps for storage.
In user space BPF maps or a perf buffer are periodically read and some output (summary) is shown to a user.</p>
<p>I was curious how Parca Agent was implemented.
According to its <a href="https://github.com/parca-dev/parca-agent/blob/main/docs/design.md">design doc</a>
it attaches the BPF program to a Linux cgroup using
<a href="https://man7.org/linux/man-pages/man2/perf_event_open.2.html">perf_event_open()</a> system call.
The call creates a file descriptor that allows measuring performance information.
It instructs the kernel to call the BPF program 100 times per second.</p>
<p>Here I annotated
<a href="https://github.com/parca-dev/parca-agent/blob/1e935f4d5f7b4484f6cf3d4ee26340f3718ff37d/pkg/profiler/profiler.go#L270">github.com/parca-dev/parca-agent/pkg/profiler/profiler.go</a> code around the syscall.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// To discover cgroups to profile in Kubernetes,</span><span class="w"></span>
<span class="c1">// Parca Agent first discovers all pods running on the node it is on,</span><span class="w"></span>
<span class="c1">// then discovers the primary PID of the cgroup using Kubernetes CRI (container runtime interface).</span><span class="w"></span>
<span class="c1">// For profiling purposes a perf_event cgroup is required, which is read from /proc/PID/cgroup.</span><span class="w"></span>
<span class="nx">cgroup</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Open</span><span class="p">(</span><span class="o">...</span><span class="p">)</span><span class="w"></span>
<span class="c1">// The pid and cpu arguments allow specifying which process and CPU to monitor.</span><span class="w"></span>
<span class="nx">pid</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">int</span><span class="p">(</span><span class="nx">cgroup</span><span class="p">.</span><span class="nx">Fd</span><span class="p">())</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="nx">cpu</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nx">cpu</span><span class="w"> </span><span class="p"><</span><span class="w"> </span><span class="nx">runtime</span><span class="p">.</span><span class="nx">NumCPU</span><span class="p">();</span><span class="w"> </span><span class="nx">cpu</span><span class="o">++</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">fd</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">PerfEventOpen</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="nx">unix</span><span class="p">.</span><span class="nx">PerfEventAttr</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PERF_TYPE_SOFTWARE event type indicates that</span><span class="w"></span>
<span class="w"> </span><span class="c1">// we are measuring software events provided by the kernel.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Type</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">PERF_TYPE_SOFTWARE</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Config is a Type-specific configuration.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PERF_COUNT_SW_CPU_CLOCK reports the CPU clock, a high-resolution per-CPU timer.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Config</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">PERF_COUNT_SW_CPU_CLOCK</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Size of attribute structure for forward/backward compatibility.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Size</span><span class="p">:</span><span class="w"> </span><span class="nb">uint32</span><span class="p">(</span><span class="nx">unsafe</span><span class="p">.</span><span class="nx">Sizeof</span><span class="p">(</span><span class="nx">unix</span><span class="p">.</span><span class="nx">PerfEventAttr</span><span class="p">{})),</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Sample could mean sampling period (expressed as the number of occurrences of an event)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// or frequency (the average rate of samples per second).</span><span class="w"></span>
<span class="w"> </span><span class="c1">// See https://perf.wiki.kernel.org/index.php/Tutorial#Period_and_rate.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// In order to use frequency PerfBitFreq flag is set below.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The kernel will adjust the sampling period to try and achieve the desired rate.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Sample</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Bits</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">PerfBitDisabled</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">PerfBitFreq</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nx">pid</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">cpu</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// This argument allows event groups to be created.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// A single event on its own is created with groupFd = -1</span><span class="w"></span>
<span class="w"> </span><span class="c1">// and is considered to be a group with only 1 member.</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PERF_FLAG_PID_CGROUP flag activates per-container system-wide monitoring.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// In this mode, the event is measured only if the thread running on the monitored CPU</span><span class="w"></span>
<span class="w"> </span><span class="c1">// belongs to the designated container (cgroup).</span><span class="w"></span>
<span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">PERF_FLAG_PID_CGROUP</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h3>BPF C code</h3>
<p>From looking at BPF C code
<a href="https://github.com/parca-dev/parca-agent/blob/31253527651ebebb74c6200eb68fe9251479ed6b/parca-agent.bpf.c">parca-agent.bpf.c</a>
we can see the BPF program <code>do_sample</code>.
As mentioned above it's executed 100 times per second.
Each time it captures stack traces of the thread that is currently running on CPU.</p>
<div class="highlight"><pre><span></span><code><span class="n">SEC</span><span class="p">(</span><span class="s">"perf_event"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">do_sample</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">bpf_perf_event_data</span><span class="w"> </span><span class="o">*</span><span class="n">ctx</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
</code></pre></div>
<p><code>SEC("perf_event")</code> denotes <code>BPF_PROG_TYPE_PERF_EVENT</code> BPF program type
which specifies the type of events that the BPF program attaches to (perf_events),
and the arguments for the events.</p>
<div class="highlight"><pre><span></span><code><span class="n">u64</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_get_current_pid_tgid</span><span class="p">();</span><span class="w"></span>
<span class="n">u32</span><span class="w"> </span><span class="n">tgid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="mi">32</span><span class="p">;</span><span class="w"></span>
<span class="n">u32</span><span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p><a href="https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/bpf.h#L1777">bpf_get_current_pid_tgid()</a>
BPF helper returns an unsigned 64-bit integer containing
the current TGID (what user space calls the PID) in the upper bits and
the current PID (what user space calls the kernel thread ID) in the lower bits.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Max amount of different stack trace addresses to buffer in the map.</span>
<span class="cp">#define MAX_STACK_ADDRESSES 1024</span>
<span class="c1">// Max depth of each stack trace to track.</span>
<span class="cp">#define MAX_STACK_DEPTH 127</span>
<span class="c1">// Stack trace value is 1 big byte array of the stack addresses.</span>
<span class="k">typedef</span><span class="w"> </span><span class="n">__u64</span><span class="w"> </span><span class="n">stack_trace_type</span><span class="p">[</span><span class="n">MAX_STACK_DEPTH</span><span class="p">];</span><span class="w"></span>
<span class="c1">// The stack_traces map holds an array of memory addresses,</span>
<span class="c1">// e.g., stack_traces[1253] = [0xdeadbeef, 0x123abcde]</span>
<span class="c1">// where 1253 is a stack ID.</span>
<span class="k">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">type</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_MAP_TYPE_STACK_TRACE</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">max_entries</span><span class="p">,</span><span class="w"> </span><span class="n">MAX_STACK_ADDRESSES</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__type</span><span class="p">(</span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="n">u32</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__type</span><span class="p">(</span><span class="n">value</span><span class="p">,</span><span class="w"> </span><span class="n">stack_trace_type</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">stack_traces</span><span class="w"> </span><span class="n">SEC</span><span class="p">(</span><span class="s">".maps"</span><span class="p">);</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">stack_count_key_t</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">u32</span><span class="w"> </span><span class="n">pid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">int32</span><span class="w"> </span><span class="n">user_stack_id</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">int32</span><span class="w"> </span><span class="n">kernel_stack_id</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="c1">// The counts map keeps track of how many times a stack trace has been seen,</span>
<span class="c1">// e.g., counts[{10342, 1253, 0234}] = 45 times.</span>
<span class="k">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">type</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_MAP_TYPE_HASH</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">max_entries</span><span class="p">,</span><span class="w"> </span><span class="mi">10240</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__type</span><span class="p">(</span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">stack_count_key_t</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__type</span><span class="p">(</span><span class="n">value</span><span class="p">,</span><span class="w"> </span><span class="n">u64</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">counts</span><span class="w"> </span><span class="n">SEC</span><span class="p">(</span><span class="s">".maps"</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>The BPF program records stack traces in special <code>stack_traces</code> BPF map
which is indexed by stack IDs.
The map value is an array of memory addresses that represent the code executed.</p>
<table>
<thead>
<tr>
<th>stack ID</th>
<th>memory addresses</th>
</tr>
</thead>
<tbody>
<tr>
<td>1253</td>
<td>[ 0xdeadbeef; 0x123abcde; ... ]</td>
</tr>
<tr>
<td>0234</td>
<td>[ 0x597be95a; 0xae5ee03; ... ]</td>
</tr>
</tbody>
</table>
<p>The <code>counts</code> BPF map keeps track of how many times a stack trace has been seen:</p>
<ul>
<li>its key is a triple of PID, user-space stack ID, and kernel-space stack ID, see <code>struct stack_count_key_t { ... }</code></li>
<li>its value is the amount of times that stack trace has been observed</li>
</ul>
<table>
<thead>
<tr>
<th>stack_count_key_t</th>
<th>seen</th>
</tr>
</thead>
<tbody>
<tr>
<td>{ 10342; 1253; 0234 }</td>
<td>45</td>
</tr>
</tbody>
</table>
<p>The maps are populated using
<a href="https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/bpf.h#L2080">bpf_get_stackid()</a>
and <code>bpf_map_lookup_or_try_init()</code> functions.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Create a key for "counts" map.</span>
<span class="k">struct</span><span class="w"> </span><span class="nc">stack_count_key_t</span><span class="w"> </span><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{.</span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tgid</span><span class="p">};</span><span class="w"></span>
<span class="c1">// Read user-space stack ID and insert memory addresses into stack_traces map.</span>
<span class="c1">// The positive or null stack id is returned on success,</span>
<span class="c1">// or a negative error in case of failure.</span>
<span class="n">key</span><span class="p">.</span><span class="n">user_stack_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_get_stackid</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">stack_traces</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_F_USER_STACK</span><span class="p">);</span><span class="w"></span>
<span class="c1">// Read kernel-space stack ID and insert memory addresses into stack_traces map.</span>
<span class="n">key</span><span class="p">.</span><span class="n">kernel_stack_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_get_stackid</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">stack_traces</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span><span class="w"></span>
<span class="n">u64</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="n">u64</span><span class="w"> </span><span class="o">*</span><span class="n">seen</span><span class="p">;</span><span class="w"></span>
<span class="n">seen</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_map_lookup_or_try_init</span><span class="p">(</span><span class="o">&</span><span class="n">counts</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">zero</span><span class="p">);</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="n">seen</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="c1">// Atomically increments the seen counter.</span>
<span class="n">__sync_fetch_and_add</span><span class="p">(</span><span class="n">seen</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>On Go side the BPF maps are read every 10 seconds,
get processed, and then purged to reset for the next iteration.</p>
<h3>DIY profiler</h3>
<p>Parca Agent relies on <a href="https://github.com/aquasecurity/libbpfgo">cgo bindings for libbpf</a> from Aqua Security.
I wanted to make something similar using
<a href="https://github.com/cilium/ebpf">github.com/cilium/ebpf</a> which doesn't need cgo.</p>
<p>In order to prepare the environment I ran Ubuntu 21.10 in a virtual machine and installed Clang with Go.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ cat > Vagrantfile <<CFG</span>
<span class="go">Vagrant.configure("2") do |config|</span>
<span class="go"> config.vm.box = "ubuntu/impish64"</span>
<span class="go">end</span>
<span class="go">CFG</span>
<span class="go">οΉͺ vagrant up</span>
<span class="go">οΉͺ vagrant ssh</span>
<span class="go">οΉ© sudo apt-get update</span>
<span class="go">οΉ© sudo apt-get install clang</span>
<span class="go">οΉ© sudo snap install go --classic</span>
<span class="go">οΉ© uname -nr</span>
<span class="go">ubuntu-impish 5.13.0-39-generic</span>
<span class="go">οΉ© clang -v</span>
<span class="go">Ubuntu clang version 13.0.0-2</span>
</code></pre></div>
<p>Then I copied
<a href="https://github.com/parca-dev/parca-agent/blob/31253527651ebebb74c6200eb68fe9251479ed6b/parca-agent.bpf.c">parca-agent.bpf.c</a>
from parca-agent repository and tried to generate Go files using bpf2go tool.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© cd /vagrant/</span>
<span class="go">οΉ© mkdir -p cmd/profiler/bpf/</span>
<span class="go">οΉ© curl -o cmd/profiler/bpf/parca-agent.bpf.c https://raw.githubusercontent.com/parca-dev/parca-agent/31253527651ebebb74c6200eb68fe9251479ed6b/parca-agent.bpf.c</span>
<span class="go">οΉ© cat > cmd/profiler/main.go <<<'''</span>
<span class="go">package main</span>
<span class="go">//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cflags $BPF_CFLAGS -cc clang-13 ParcaAgent ./bpf/parca-agent.bpf.c -- -I../../headers</span>
<span class="go">func main() {}</span>
<span class="go">'''</span>
<span class="go">οΉ© go mod init diy-parca-agent</span>
<span class="go">οΉ© go get github.com/cilium/ebpf/cmd/bpf2go</span>
<span class="go">οΉ© go generate ./cmd/profiler/</span>
<span class="go">/vagrant/cmd/profiler/bpf/parca-agent.bpf.c:11:10: fatal error: 'vmlinux.h' file not found</span>
</code></pre></div>
<p><code>vmlinux.h</code> can be generated from the installed kernel using bpftool tool.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© mkdir headers</span>
<span class="go">οΉ© sudo apt-get install linux-tools-$(uname -r) linux-tools-common</span>
<span class="go">οΉ© bpftool btf dump file /sys/kernel/btf/vmlinux format c > ./headers/vmlinux.h</span>
</code></pre></div>
<p>The remaining missing headers I copied from ubuntu-kernel, libbpf, and cilium repositories.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© curl -o ./headers/bpf_core_read.h https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/impish/plain/tools/lib/bpf/bpf_core_read.h</span>
<span class="go">οΉ© curl -o ./headers/bpf_endian.h https://raw.githubusercontent.com/cilium/ebpf/master/examples/headers/bpf_endian.h</span>
<span class="go">οΉ© curl -o ./headers/bpf_helpers.h https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/impish/plain/tools/lib/bpf/bpf_helpers.h</span>
<span class="go">οΉ© curl -o ./headers/bpf_tracing.h https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/impish/plain/tools/lib/bpf/bpf_tracing.h</span>
<span class="go">οΉ© curl -o ./headers/bpf_helper_defs.h https://raw.githubusercontent.com/libbpf/libbpf/master/src/bpf_helper_defs.h</span>
</code></pre></div>
<p>Finally, all the dependencies were resolved and Go code was generated.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© go generate ./cmd/profiler/</span>
<span class="go">Compiled /vagrant/cmd/profiler/parcaagent_bpfel.o</span>
<span class="go">Stripped /vagrant/cmd/profiler/parcaagent_bpfel.o</span>
<span class="go">Wrote /vagrant/cmd/profiler/parcaagent_bpfel.go</span>
<span class="go">Compiled /vagrant/cmd/profiler/parcaagent_bpfeb.o</span>
<span class="go">Stripped /vagrant/cmd/profiler/parcaagent_bpfeb.o</span>
<span class="go">Wrote /vagrant/cmd/profiler/parcaagent_bpfeb.go</span>
</code></pre></div>
<p>Next step was to see whether <code>DoSample</code> BPF program is loaded into the kernel from an ELF.
<code>ParcaAgentObjects{}</code> contains all objects (BPF program and BPF maps) after they have been loaded into the kernel.</p>
<div class="highlight"><pre><span></span><code><span class="nx">objs</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ParcaAgentObjects</span><span class="p">{}</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">LoadParcaAgentObjects</span><span class="p">(</span><span class="o">&</span><span class="nx">objs</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to load BPF program and maps: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
</code></pre></div>
<p>It worked, no errors were printed.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/profiler/</span>
</code></pre></div>
<blockquote>
<p>A call to perf_event_open() creates a file descriptor
that allows measuring performance information.
Each file descriptor corresponds to one event that is measured.
Events can be enabled and disabled via ioctl(2).
https://man7.org/linux/man-pages/man2/perf_event_open.2.html</p>
</blockquote>
<p>It's time to attach <code>DoSample</code> BPF program to perf event using
<code>unix.PerfEventOpen()</code> and <code>unix.IoctlSetInt()</code> functions.
Note, Parca Agent doesn't directly use <code>unix.IoctlSetInt()</code> and relies on
<a href="https://pkg.go.dev/github.com/aquasecurity/libbpfgo#BPFProg.AttachPerfEvent">AttachPerfEvent()</a>
function from <a href="http://github.com/aquasecurity/libbpfgo">github.com/aquasecurity/libbpfgo</a>.
I found cilium related explanation <a href="https://github.com/cilium/ebpf/discussions/548">#548</a>
how to achieve the same thing:</p>
<ol>
<li>Load the BPF program into the kernel: <code>LoadParcaAgentObjects(&objs, nil)</code></li>
<li>Open the perf event: <code>fd, _ := unix.PerfEventOpen(...)</code>.</li>
<li>Set the BPF program on the perf event:
<code>unix.IoctlSetInt(fd, unix.PERF_EVENT_IOC_SET_BPF, objs.DoSample.FD())</code></li>
<li>Enable the perf event:
<code>unix.IoctlSetInt(fd, unix.PERF_EVENT_IOC_ENABLE, 0)</code></li>
</ol>
<p>Those steps worked as well.
I was hopeful to see whether stack traces were collected,
so I periodically printed the content of <code>Counts</code> BPF map.</p>
<div class="highlight"><pre><span></span><code><span class="kd">type</span><span class="w"> </span><span class="nx">stackCountKey</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">PID</span><span class="w"> </span><span class="kt">uint32</span><span class="w"></span>
<span class="w"> </span><span class="nx">UserStackID</span><span class="w"> </span><span class="kt">int32</span><span class="w"></span>
<span class="w"> </span><span class="nx">KernelStackID</span><span class="w"> </span><span class="kt">int32</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="nx">stackCountKey</span><span class="w"></span>
<span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="kt">uint64</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="nx">it</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">ParcaAgentMaps</span><span class="p">.</span><span class="nx">Counts</span><span class="p">.</span><span class="nx">Iterate</span><span class="p">()</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="nx">it</span><span class="p">.</span><span class="nx">Next</span><span class="p">(</span><span class="o">&</span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">value</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"%+v seen %d times\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">it</span><span class="p">.</span><span class="nx">Err</span><span class="p">();</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to read from Counts map: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>The easiest way to reliably get stack traces was to run <code>top</code> in another terminal.
I made the profiler focus on its PID 15958 and it worked π.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© top</span>
<span class="go">οΉ© sudo go run ./cmd/profiler/ -pid 15958</span>
<span class="go">Waiting for stack traces...</span>
<span class="go">{PID:15958 UserStackID:132 KernelStackID:114} seen 1 times</span>
<span class="go">{PID:15958 UserStackID:709 KernelStackID:-14} seen 1 times # -14 indicates bpf_get_stackid() error.</span>
<span class="go">{PID:15958 UserStackID:366 KernelStackID:30} seen 2 times</span>
<span class="go">{PID:15958 UserStackID:674 KernelStackID:943} seen 1 times</span>
</code></pre></div>
<p>The end of my post is an disappointing anticlimax because
I haven't looked yet at how to show the actual stack traces.
I will update <a href="https://github.com/marselester/diy-parca-agent">the repository</a> when I have time.</p>
<p>Update: finally I figured out how to show stack traces, see
<a href="https://marselester.com/diy-cpu-profiler-from-bpf-maps-to-pprof.html">DIY CPU profiler: from BPF maps to pprof</a> post.</p>BPF Go program in Kubernetes2021-11-17T00:00:00+07:002021-11-17T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2021-11-17:/bpf-go-program-in-kubernetes.html<p>BPF opens a lot of possibilities of making observability tools running in Kubernetes.
One can start with <a href="https://github.com/iovisor/bcc/tree/master/libbpf-tools">BCC libbpf-tools</a> written in C,
e.g., launch tcpconnlat program and process its stdout with another program to
detect cases when it took too long to establish a TCP connection.
For example, <code>curl β¦</code></p><p>BPF opens a lot of possibilities of making observability tools running in Kubernetes.
One can start with <a href="https://github.com/iovisor/bcc/tree/master/libbpf-tools">BCC libbpf-tools</a> written in C,
e.g., launch tcpconnlat program and process its stdout with another program to
detect cases when it took too long to establish a TCP connection.
For example, <code>curl http://example.com</code> took 47.8 milliseconds to establish a connection
where source address is <code>10.0.2.15</code> and destination address is <code>93.184.216.34</code>.</p>
<div class="highlight"><pre><span></span><code><span class="go">PID COMM IP SADDR DADDR DPORT LAT(ms)</span>
<span class="go">21500 curl 4 10.0.2.15 93.184.216.34 80 47.80</span>
</code></pre></div>
<p>Another option is to use Go version of
<a href="https://github.com/marselester/libbpf-tools/blob/master/cmd/tcpconnlat/main.go">tcpconnlat</a>
and modify it as you wish, e.g., write the events into Kafka for further analysis.</p>
<p>I have already published a Docker image
<a href="https://github.com/marselester/libbpf-tools/blob/master/Dockerfile">marselester/go-libbpf-tools</a>
containing tcpconnlat and tried to run it on Mac, that didn't go well though.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ docker run --rm -it --privileged marselester/go-libbpf-tools:latest bash</span>
<span class="go">root@552a159ce901:/opt/libbpf-toolsοΌ ./tcpconnlat</span>
<span class="go">failed to load BPF programs and maps: field TcpRcvStateProcess: program tcp_rcv_state_process: CO-RE relocations: no BTF for kernel version 5.10.47-linuxkit: not supported</span>
</code></pre></div>
<p>Minikube with Virtualbox driver didn't help either because it uses an old kernel,
hopefully it will be upgraded soon <a href="https://github.com/kubernetes/minikube/issues/10501">#10501</a>.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ minikube start --driver=virtualbox</span>
<span class="go">οΉͺ minikube ssh</span>
<span class="go">οΉ© uname -nr</span>
<span class="go">Linux minikube 4.19.202</span>
</code></pre></div>
<p>Luckily there is another option called <a href="https://kubespray.io">Kubespray</a>.</p>
<h2>Kubespray</h2>
<p>Kubespray sets up a Kubernetes cluster of 3 nodes using Vagrant and Ansible.
Clone the repository and install Python dependencies for provisioning tasks.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ git clone https://github.com/kubernetes-sigs/kubespray</span>
<span class="go">οΉͺ cd ./kubespray/</span>
<span class="go">οΉͺ virtualenv venv</span>
<span class="go">οΉͺ source ./venv/bin/activate</span>
<span class="gp gp-VirtualEnv">(venv)</span> <span class="go">οΉͺ pip install -r requirements.txt</span>
</code></pre></div>
<p>From looking at <a href="https://github.com/kubernetes-sigs/kubespray/blob/master/Vagrantfile">the Vagrantfile</a>
we see that Kubespray supports Fedora Linux 34, so BTF and CO-RE technologies should be there.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ mkdir vagrant</span>
<span class="go">οΉͺ echo '$os = "fedora34"' > ./vagrant/config.rb</span>
<span class="go">οΉͺ vagrant up</span>
<span class="go">οΉͺ export KUBECONFIG=$(pwd)/.vagrant/provisioners/ansible/inventory/artifacts/admin.conf</span>
<span class="go">οΉͺ kubectl get nodes</span>
<span class="go">NAME STATUS ROLES AGE VERSION</span>
<span class="go">k8s-1 Ready control-plane,master 8m20s v1.22.3</span>
<span class="go">k8s-2 Ready control-plane,master 7m56s v1.22.3</span>
<span class="go">k8s-3 Ready <none> 6m58s v1.22.3</span>
<span class="go">οΉͺ vagrant ssh k8s-1</span>
<span class="go">οΉ© uname -nr</span>
<span class="go">k8s-1 5.11.12-300.fc34.x86_64</span>
</code></pre></div>
<p>See <a href="https://github.com/kubernetes-sigs/kubespray/blob/master/docs/vagrant.md">vagrant.md</a>
if the cluster wasn't provisioned.</p>
<h2>DaemonSet</h2>
<p>An observability tool should run on each node,
and a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">DaemonSet</a>
ensures that all nodes run a copy of a pod.
Let's try to launch tcpconnlat on all 3 nodes.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ kubectl apply -f - <<<'</span>
<span class="go">apiVersion: apps/v1</span>
<span class="go">kind: DaemonSet</span>
<span class="go">metadata:</span>
<span class="go"> name: tcpconnlat-daemon</span>
<span class="go">spec:</span>
<span class="go"> selector:</span>
<span class="go"> matchLabels:</span>
<span class="go"> app: tcpconnlat</span>
<span class="go"> template:</span>
<span class="go"> metadata:</span>
<span class="go"> labels:</span>
<span class="go"> app: tcpconnlat</span>
<span class="go"> spec:</span>
<span class="go"> containers:</span>
<span class="go"> - name: libbpf-tools</span>
<span class="go"> image: marselester/go-libbpf-tools:latest</span>
<span class="go"> command:</span>
<span class="go"> - /opt/libbpf-tools/tcpconnlat</span>
<span class="go">'</span>
</code></pre></div>
<p>Unfortunately pods have crashed because the containers didn't have privileged mode.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ kubectl get pods</span>
<span class="go">NAME READY STATUS RESTARTS AGE</span>
<span class="go">tcpconnlat-daemon-646h2 0/1 CrashLoopBackOff 5 (2m9s ago) 5m13s</span>
<span class="go">tcpconnlat-daemon-hwnzt 0/1 CrashLoopBackOff 5 (118s ago) 5m13s</span>
<span class="go">tcpconnlat-daemon-tmn6r 0/1 CrashLoopBackOff 5 (2m14s ago) 5m13s</span>
<span class="go">οΉͺ kubectl logs -f tcpconnlat-daemon-646h2</span>
<span class="go">failed to set temporary RLIMIT_MEMLOCK: operation not permitted</span>
</code></pre></div>
<p>Let's enable it.</p>
<blockquote>
<p>By default a container is not allowed to access any devices on the host,
but a "privileged" container is given access to all devices on the host.
This allows the container nearly all the same access as processes running on the host.</p>
<p>https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged</p>
</blockquote>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ kubectl apply -f - <<<'</span>
<span class="go">apiVersion: apps/v1</span>
<span class="go">kind: DaemonSet</span>
<span class="go">metadata:</span>
<span class="go"> name: tcpconnlat-daemon</span>
<span class="go">spec:</span>
<span class="go"> selector:</span>
<span class="go"> matchLabels:</span>
<span class="go"> app: tcpconnlat</span>
<span class="go"> template:</span>
<span class="go"> metadata:</span>
<span class="go"> labels:</span>
<span class="go"> app: tcpconnlat</span>
<span class="go"> spec:</span>
<span class="go"> containers:</span>
<span class="go"> - name: libbpf-tools</span>
<span class="go"> image: marselester/go-libbpf-tools:latest</span>
<span class="go"> command:</span>
<span class="go"> - /opt/libbpf-tools/tcpconnlat</span>
<span class="go"> securityContext:</span>
<span class="go"> privileged: true</span>
<span class="go">'</span>
</code></pre></div>
<p>It works! π</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ kubectl get pods</span>
<span class="go">NAME READY STATUS RESTARTS AGE</span>
<span class="go">tcpconnlat-daemon-9sgc5 1/1 Running 0 18s</span>
<span class="go">tcpconnlat-daemon-lrvh5 1/1 Running 0 18s</span>
<span class="go">tcpconnlat-daemon-th9k4 1/1 Running 0 18s</span>
<span class="go">οΉͺ kubectl logs -f tcpconnlat-daemon-9sgc5</span>
<span class="go">PID COMM IP SADDR DADDR DPORT LAT(ms)</span>
<span class="go">5955 coredns 4 127.0.0.1 127.0.0.1 8080 0.02</span>
<span class="go">703 kubelet 4 172.18.8.101 172.18.8.101 6443 0.04</span>
<span class="go">703 kubelet 4 10.233.64.1 10.233.64.4 8181 0.07</span>
<span class="go">703 kubelet 4 169.254.25.10 169.254.25.10 9254 0.03</span>
<span class="go">703 kubelet 4 10.233.64.1 10.233.64.5 8080 0.03</span>
<span class="go">5537 node-cache 4 169.254.25.10 169.254.25.10 9254 0.06</span>
<span class="go">4204 etcd 4 172.18.8.101 172.18.8.103 2380 0.22</span>
<span class="go">4204 etcd 4 172.18.8.101 172.18.8.102 2380 0.21</span>
</code></pre></div>
<p>I also tried to run execsnoop and tcpconnect, but alas they crashed with the corresponding errors.</p>
<div class="highlight"><pre><span></span><code><span class="go">failed to attach the BPF program to sys_enter_execve tracepoint: trace event syscalls/sys_enter_execve: file does not exist</span>
<span class="go">failed to load BPF programs and maps: field TcpV4ConnectRet: program tcp_v4_connect_ret: load program: permission denied: trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.</span>
</code></pre></div>BPF: Go frontend for tcpconnect2021-11-01T00:00:00+07:002021-11-01T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2021-11-01:/bpf-go-frontend-for-tcpconnect.html<p><a href="https://marselester.com/bpf-go-frontend-for-execsnoop.html">In the previos BPF post</a>
I shared my experiment of writing Go frontend for execsnoop.
Here I would like to focus on tcpconnect, a BCC tool to trace new TCP active connections.
It is useful for determining who is connecting to whom.
This works by tracing the <code>tcp_v4_connect()</code> and <code>tcp_v6_connect β¦</code></p><p><a href="https://marselester.com/bpf-go-frontend-for-execsnoop.html">In the previos BPF post</a>
I shared my experiment of writing Go frontend for execsnoop.
Here I would like to focus on tcpconnect, a BCC tool to trace new TCP active connections.
It is useful for determining who is connecting to whom.
This works by tracing the <code>tcp_v4_connect()</code> and <code>tcp_v6_connect()</code> kernel functions.</p>
<p>As before, I tried to make Go tcpconnect's output match the original program,
so I copied the ELF binary <code>tcpconnect</code> from a Docker container to the virtual machine
(Ubuntu 21.04 from the previos post).</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ docker run --name libbpf marselester/libbpf-tools:latest</span>
<span class="go">οΉͺ docker cp libbpf:/opt/tcpconnect .</span>
</code></pre></div>
<p>The tcpconnect captured <code>curl example.com</code> running in another terminal:</p>
<ul>
<li>PID <em>93336</em> is the process ID that opened the connection</li>
<li>COMM <em>curl</em> is the process name that opened the connection, e.g., via a <code>connect()</code> syscall</li>
<li>IP <em>4</em> stands for IP v4 address protocol</li>
<li>SADDR <em>10.0.2.15</em> is the source address</li>
<li>DADDR <em>93.184.216.34</em> is the destination address</li>
<li>DPORT <em>80</em> is the destination port</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo /vagrant/tcpconnect</span>
<span class="go">PID COMM IP SADDR DADDR DPORT</span>
<span class="go">93336 curl 4 10.0.2.15 93.184.216.34 80</span>
</code></pre></div>
<h2>Embedding BPF in Go</h2>
<p>The tcpconnect BPF program comprises of
<a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpconnect.bpf.c">tcpconnect.bpf.c</a>,
<a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpconnect.h">tcpconnect.h</a>,
and <a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/maps.bpf.h">maps.bpf.h</a> files.
Let's copy them from BCC toolkit and generate Go files.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© cd /vagrant/</span>
<span class="go">οΉ© mkdir -p cmd/tcpconnect/bpf/</span>
<span class="go">οΉ© curl -o cmd/tcpconnect/bpf/tcpconnect.bpf.c https://raw.githubusercontent.com/iovisor/bcc/master/libbpf-tools/tcpconnect.bpf.c</span>
<span class="go">οΉ© curl -o cmd/tcpconnect/bpf/tcpconnect.h https://raw.githubusercontent.com/iovisor/bcc/master/libbpf-tools/tcpconnect.h</span>
<span class="go">οΉ© curl -o cmd/tcpconnect/bpf/maps.bpf.h https://raw.githubusercontent.com/iovisor/bcc/master/libbpf-tools/maps.bpf.h</span>
<span class="go">οΉ© cat > cmd/tcpconnect/main.go <<<'''</span>
<span class="go">package main</span>
<span class="go">//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cflags $BPF_CFLAGS -cc clang-12 TCPConnect ./bpf/tcpconnect.bpf.c -- -I../../headers</span>
<span class="go">func main() {}</span>
<span class="go">'''</span>
<span class="go">οΉ© go generate ./cmd/tcpconnect/</span>
<span class="go">/vagrant/cmd/tcpconnect/bpf/tcpconnect.bpf.c:9:10: fatal error: 'bpf/bpf_tracing.h' file not found</span>
</code></pre></div>
<p>Since <code>bpf_tracing.h</code> file is missing, let's get it from ubuntu-kernel repository
and run <code>go generate</code> again,
but this time with <code>BPF_CFLAGS='-D__TARGET_ARCH_x86'</code> env var (a compiler flag) because
<a href="https://elixir.bootlin.com/linux/latest/source/tools/lib/bpf/bpf_tracing.h#L5">bpf_tracing.h</a> expects it.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© curl -o ./headers/bpf/bpf_tracing.h https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/hirsute/plain/tools/lib/bpf/bpf_tracing.h</span>
</code></pre></div>
<p>It worked out π¬.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© BPF_CFLAGS='-D__TARGET_ARCH_x86' go generate ./cmd/tcpconnect/</span>
<span class="go">Compiled /vagrant/cmd/tcpconnect/tcpconnect_bpfel.o</span>
<span class="go">Wrote /vagrant/cmd/tcpconnect/tcpconnect_bpfel.go</span>
<span class="go">Compiled /vagrant/cmd/tcpconnect/tcpconnect_bpfeb.o</span>
<span class="go">Wrote /vagrant/cmd/tcpconnect/tcpconnect_bpfeb.go</span>
<span class="go">οΉ© sudo go run ./cmd/tcpconnect/</span>
</code></pre></div>
<p>BPF programs loading looks very similar to Go execsnoop version,
though instead of calling <code>LoadTCPConnectObjects()</code> the <code>LoadTCPConnect()</code> is called to
parse an ELF file into <code>spec</code> struct (a collection of BPF maps and programs) and
replace constants in the C program to filter connections by PID or UID.
Here is how they are defined in <code>tcpconnect.bpf.c</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">const</span><span class="w"> </span><span class="k">volatile</span><span class="w"> </span><span class="kt">uid_t</span><span class="w"> </span><span class="n">filter_uid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">-1</span><span class="p">;</span><span class="w"></span>
<span class="k">const</span><span class="w"> </span><span class="k">volatile</span><span class="w"> </span><span class="kt">pid_t</span><span class="w"> </span><span class="n">filter_pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>After the constants are rewritten, the BPF programs get loaded into the kernel with <code>spec.LoadAndAssign()</code>.</p>
<div class="highlight"><pre><span></span><code><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"flag"</span><span class="w"></span>
<span class="w"> </span><span class="s">"log"</span><span class="w"></span>
<span class="w"> </span><span class="s">"golang.org/x/sys/unix"</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="c1">//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cflags $BPF_CFLAGS -cc clang-12 TCPConnect ./bpf/tcpconnect.bpf.c -- -I../../headers</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">filterUID</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Int</span><span class="p">(</span><span class="s">"uid"</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="s">"trace this UID only"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">filterPID</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Int</span><span class="p">(</span><span class="s">"pid"</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="s">"trace this PID only"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Increase the resource limit of the current process to provide sufficient space</span><span class="w"></span>
<span class="w"> </span><span class="c1">// for locking memory for the BPF maps.</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">Setrlimit</span><span class="p">(</span><span class="nx">unix</span><span class="p">.</span><span class="nx">RLIMIT_MEMLOCK</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">unix</span><span class="p">.</span><span class="nx">Rlimit</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Cur</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">RLIM_INFINITY</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Max</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">RLIM_INFINITY</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to set temporary RLIMIT_MEMLOCK: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Replace constants in the BPF C program to filter connections by PID or UID.</span><span class="w"></span>
<span class="w"> </span><span class="nx">spec</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">LoadTCPConnect</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to load collection spec: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">bpfConst</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">make</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kd">interface</span><span class="p">{})</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">*</span><span class="nx">filterUID</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="o">-</span><span class="mi">1</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">bpfConst</span><span class="p">[</span><span class="s">"filter_uid"</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">uint32</span><span class="p">(</span><span class="o">*</span><span class="nx">filterUID</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">*</span><span class="nx">filterPID</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">bpfConst</span><span class="p">[</span><span class="s">"filter_pid"</span><span class="p">]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">int32</span><span class="p">(</span><span class="o">*</span><span class="nx">filterPID</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">spec</span><span class="p">.</span><span class="nx">RewriteConstants</span><span class="p">(</span><span class="nx">bpfConst</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to rewrite BPF constants: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Load the BPF program into the kernel from an ELF.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// TCPConnectObjects contains all objects (BPF programs and maps) after they have been loaded into the kernel:</span><span class="w"></span>
<span class="w"> </span><span class="c1">// - TcpV4Connect, TcpV4ConnectRet, TcpV6Connect, TcpV6ConnectRet BPF programs,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// - Events, Ipv4Count, Ipv6Count, Sockets BPF maps.</span><span class="w"></span>
<span class="w"> </span><span class="nx">objs</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">TCPConnectObjects</span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">spec</span><span class="p">.</span><span class="nx">LoadAndAssign</span><span class="p">(</span><span class="o">&</span><span class="nx">objs</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to load BPF programs and maps: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h2>Kprobes</h2>
<p>A BPF program is executed on events such as kprobes.
Kprobes provide kernel dynamic instrumentation for any kernel function,
and they can instrument instructions within functions.
There is also an interface called kretprobes for instrumenting when functions return, and their return values.</p>
<p>The <code>tcpconnect.bpf.c</code> lists four BPF programs that trace events related to creating TCP sessions:
enter and exit of <code>tcp_v4_connect()</code> and <code>tcp_v6_connect()</code> kernel functions.</p>
<div class="highlight"><pre><span></span><code><span class="n">SEC</span><span class="p">(</span><span class="s">"kprobe/tcp_v4_connect"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">BPF_KPROBE</span><span class="p">(</span><span class="n">tcp_v4_connect</span><span class="p">,</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">sock</span><span class="w"> </span><span class="o">*</span><span class="n">sk</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">enter_tcp_connect</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="n">sk</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">SEC</span><span class="p">(</span><span class="s">"kretprobe/tcp_v4_connect"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">BPF_KRETPROBE</span><span class="p">(</span><span class="n">tcp_v4_connect_ret</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">ret</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">exit_tcp_connect</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="n">ret</span><span class="p">,</span><span class="w"> </span><span class="mi">4</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">SEC</span><span class="p">(</span><span class="s">"kprobe/tcp_v6_connect"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">BPF_KPROBE</span><span class="p">(</span><span class="n">tcp_v6_connect</span><span class="p">,</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">sock</span><span class="w"> </span><span class="o">*</span><span class="n">sk</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">enter_tcp_connect</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="n">sk</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">SEC</span><span class="p">(</span><span class="s">"kretprobe/tcp_v6_connect"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">BPF_KRETPROBE</span><span class="p">(</span><span class="n">tcp_v6_connect_ret</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">ret</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">exit_tcp_connect</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="n">ret</span><span class="p">,</span><span class="w"> </span><span class="mi">6</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>On Go side I attached the BPF programs to kprobes/kretprobes.
The <code>link.Kprobe()</code>, <code>link.Kretprobe</code> functions return a link (it represents a program attached to a BPF hook)
which is later used to detach a BPF program.</p>
<div class="highlight"><pre><span></span><code><span class="nx">tcpv4kp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">link</span><span class="p">.</span><span class="nx">Kprobe</span><span class="p">(</span><span class="s">"tcp_v4_connect"</span><span class="p">,</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">TCPConnectPrograms</span><span class="p">.</span><span class="nx">TcpV4Connect</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to attach the BPF program to tcp_v4_connect kprobe: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">tcpv4kp</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="nx">tcpv4krp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">link</span><span class="p">.</span><span class="nx">Kretprobe</span><span class="p">(</span><span class="s">"tcp_v4_connect"</span><span class="p">,</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">TCPConnectPrograms</span><span class="p">.</span><span class="nx">TcpV4ConnectRet</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to attach the BPF program to tcp_v4_connect kretprobe: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">tcpv4krp</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="nx">tcpv6kp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">link</span><span class="p">.</span><span class="nx">Kprobe</span><span class="p">(</span><span class="s">"tcp_v6_connect"</span><span class="p">,</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">TCPConnectPrograms</span><span class="p">.</span><span class="nx">TcpV6Connect</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to attach the BPF program to tcp_v6_connect kprobe: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">tcpv6kp</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="nx">tcpv6krp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">link</span><span class="p">.</span><span class="nx">Kretprobe</span><span class="p">(</span><span class="s">"tcp_v6_connect"</span><span class="p">,</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">TCPConnectPrograms</span><span class="p">.</span><span class="nx">TcpV6ConnectRet</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to attach the BPF program to tcp_v6_connect kretprobe: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">tcpv6krp</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
</code></pre></div>
<p>The <code>bpf_perf_event_output()</code> C function emits records to user space via a BPF map <code>events</code>
that accesses perf per-CPU ring buffers.</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">type</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_MAP_TYPE_PERF_EVENT_ARRAY</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">key_size</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">u32</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">value_size</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">u32</span><span class="p">));</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">events</span><span class="w"> </span><span class="n">SEC</span><span class="p">(</span><span class="s">".maps"</span><span class="p">);</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="w"> </span><span class="n">e</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span><span class="w"></span>
<span class="n">bpf_perf_event_output</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">events</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_F_CURRENT_CPU</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">e</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">e</span><span class="p">));</span><span class="w"></span>
</code></pre></div>
<p>In user space the events are read using <code>github.com/cilium/ebpf/perf</code> package.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Open a perf event reader from user space on the PERF_EVENT_ARRAY map</span><span class="w"></span>
<span class="c1">// defined in the BPF C program.</span><span class="w"></span>
<span class="nx">rd</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">perf</span><span class="p">.</span><span class="nx">NewReader</span><span class="p">(</span><span class="nx">objs</span><span class="p">.</span><span class="nx">TCPConnectMaps</span><span class="p">.</span><span class="nx">Events</span><span class="p">,</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Getpagesize</span><span class="p">())</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to create perf event reader: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">rd</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">record</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">rd</span><span class="p">.</span><span class="nx">Read</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">perf</span><span class="p">.</span><span class="nx">IsClosed</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to read from perf ring buffer: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"received from perf ring buffer: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">RawSample</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Running <code>curl</code> shows that the TCP connect events are captured.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/tcpconnect/</span>
<span class="go">received from perf ring buffer:</span>
<span class="go">]??"curl???0x??P</span>
</code></pre></div>
<h2>Decoding events</h2>
<p>Let's decode the events and print them as the original tcpconnect program does.
As mentioned before, the <code>event</code> Go struct should match its C counterpart.</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">union</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__u32</span><span class="w"> </span><span class="n">saddr_v4</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">__u8</span><span class="w"> </span><span class="n">saddr_v6</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">union</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__u32</span><span class="w"> </span><span class="n">daddr_v4</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">__u8</span><span class="w"> </span><span class="n">daddr_v6</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">task</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="n">__u64</span><span class="w"> </span><span class="n">ts_us</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">__u32</span><span class="w"> </span><span class="n">af</span><span class="p">;</span><span class="w"> </span><span class="c1">// AF_INET or AF_INET6</span>
<span class="w"> </span><span class="n">__u32</span><span class="w"> </span><span class="n">pid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">__u32</span><span class="w"> </span><span class="n">uid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">__u16</span><span class="w"> </span><span class="n">dport</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</code></pre></div>
<p>I wasn't sure how to represent <code>union { ... }</code> in Go, so allocating 16-byte slice worked well,
since the union is only as big as necessary to hold its largest member (IPv6 address).</p>
<div class="highlight"><pre><span></span><code><span class="c1">// event represents a perf event sent to user space from the BPF program running in the kernel.</span><span class="w"></span>
<span class="c1">// Note, that it must match the C event struct, and both C and Go structs must be aligned the same way.</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">event</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// SrcAddr is the source address.</span><span class="w"></span>
<span class="w"> </span><span class="nx">SrcAddr</span><span class="w"> </span><span class="p">[</span><span class="mi">16</span><span class="p">]</span><span class="kt">byte</span><span class="w"></span>
<span class="w"> </span><span class="c1">// DstAddr is the destination address.</span><span class="w"></span>
<span class="w"> </span><span class="nx">DstAddr</span><span class="w"> </span><span class="p">[</span><span class="mi">16</span><span class="p">]</span><span class="kt">byte</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Comm is the process name that opened the connection.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Comm</span><span class="w"> </span><span class="p">[</span><span class="mi">16</span><span class="p">]</span><span class="kt">byte</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Timestamp is the timestamp in microseconds.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Timestamp</span><span class="w"> </span><span class="kt">uint64</span><span class="w"></span>
<span class="w"> </span><span class="c1">// AddrFam is the address family, 2 is AF_INET (IPv4), 10 is AF_INET6 (IPv6).</span><span class="w"></span>
<span class="w"> </span><span class="nx">AddrFam</span><span class="w"> </span><span class="kt">uint32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PID is the process ID that opened the connection.</span><span class="w"></span>
<span class="w"> </span><span class="nx">PID</span><span class="w"> </span><span class="kt">uint32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// UID is the process user ID.</span><span class="w"></span>
<span class="w"> </span><span class="nx">UID</span><span class="w"> </span><span class="kt">uint32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// DstPort is the destination port (uint16 in C struct).</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Note, network byte order is big-endian.</span><span class="w"></span>
<span class="w"> </span><span class="nx">DstPort</span><span class="w"> </span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="kt">byte</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>The source and destination addresses were pretty-printed using <code>net.IP</code> type
(either 4-byte (IPv4) or 16-byte (IPv6) slice) depending on the address family
(<code>2</code> is IPv4, <code>10</code> is IPv6).</p>
<p>The destination port (unsigned 16-bit integer in C struct) was translated with
<code>binary.BigEndian.Uint16()</code> function because <code>DstPort</code> byte order is big-endian
(most significant bits first).</p>
<div class="highlight"><pre><span></span><code><span class="kd">func</span><span class="w"> </span><span class="nx">printEvent</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">Writer</span><span class="p">,</span><span class="w"> </span><span class="nx">e</span><span class="w"> </span><span class="o">*</span><span class="nx">event</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">srcAddr</span><span class="p">,</span><span class="w"> </span><span class="nx">dstAddr</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">IP</span><span class="w"></span>
<span class="w"> </span><span class="nx">ipVer</span><span class="w"> </span><span class="kt">byte</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">switch</span><span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">AddrFam</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">2</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nx">srcAddr</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">IP</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">SrcAddr</span><span class="p">[:</span><span class="mi">4</span><span class="p">])</span><span class="w"></span>
<span class="w"> </span><span class="nx">dstAddr</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">IP</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">DstAddr</span><span class="p">[:</span><span class="mi">4</span><span class="p">])</span><span class="w"></span>
<span class="w"> </span><span class="nx">ipVer</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">4</span><span class="w"></span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="mi">10</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nx">srcAddr</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">IP</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">SrcAddr</span><span class="p">[:])</span><span class="w"></span>
<span class="w"> </span><span class="nx">dstAddr</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">IP</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">DstAddr</span><span class="p">[:])</span><span class="w"></span>
<span class="w"> </span><span class="nx">ipVer</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">6</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprintf</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"%-6d %-12s %-2d %-16s %-16s %d\n"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">PID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">TrimRight</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">Comm</span><span class="p">[:],</span><span class="w"> </span><span class="s">"\x00"</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">ipVer</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">srcAddr</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">dstAddr</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">BigEndian</span><span class="p">.</span><span class="nx">Uint16</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">DstPort</span><span class="p">[:]),</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Putting it all together we've got a Go frontend for tcpconnect with ability
to filter connections by PID and UID π.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/tcpconnect/</span>
<span class="go">PID COMM IP SADDR DADDR DPORT</span>
<span class="go">189217 curl 4 10.0.2.15 93.184.216.34 80</span>
<span class="go">οΉ© ./tcpconnect -h</span>
<span class="go">Usage of tcpconnect:</span>
<span class="go"> -pid int</span>
<span class="go"> trace this PID only</span>
<span class="go"> -uid int</span>
<span class="go"> trace this UID only (default -1)</span>
</code></pre></div>
<p>You can find all the source code at
<a href="https://github.com/marselester/libbpf-tools">github.com/marselester/libbpf-tools</a>.</p>
<div class="highlight"><pre><span></span><code><span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"%-6s %-12s %-2s %-16s %-16s %s\n"</span><span class="p">,</span><span class="w"> </span><span class="s">"PID"</span><span class="p">,</span><span class="w"> </span><span class="s">"COMM"</span><span class="p">,</span><span class="w"> </span><span class="s">"IP"</span><span class="p">,</span><span class="w"> </span><span class="s">"SADDR"</span><span class="p">,</span><span class="w"> </span><span class="s">"DADDR"</span><span class="p">,</span><span class="w"> </span><span class="s">"DPORT"</span><span class="p">)</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">record</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">rd</span><span class="p">.</span><span class="nx">Read</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">perf</span><span class="p">.</span><span class="nx">IsClosed</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to read from perf ring buffer: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">LostSamples</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"ring event perf buffer is full, dropped %d samples"</span><span class="p">,</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">LostSamples</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">continue</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">e</span><span class="w"> </span><span class="nx">event</span><span class="w"></span>
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">Read</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBuffer</span><span class="p">(</span><span class="nx">record</span><span class="p">.</span><span class="nx">RawSample</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">LittleEndian</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="nx">e</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to parse perf event: %#+v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">continue</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">printEvent</span><span class="p">(</span><span class="nx">os</span><span class="p">.</span><span class="nx">Stdout</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">e</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>BPF: Go frontend for execsnoop2021-10-26T00:00:00+07:002021-10-26T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2021-10-26:/bpf-go-frontend-for-execsnoop.html<p>After reading Brendan Gregg's books about BPF I was excited to try and implement something in Go.
At a first glance it turned out I would need to install LLVM, Clang, and kernel header dependencies to run a simple program.
Fortunately <a href="https://www.brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbpf.html">BTF, CO-RE technologies</a>
eliminate those dependencies though kernel >=5 β¦</p><p>After reading Brendan Gregg's books about BPF I was excited to try and implement something in Go.
At a first glance it turned out I would need to install LLVM, Clang, and kernel header dependencies to run a simple program.
Fortunately <a href="https://www.brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbpf.html">BTF, CO-RE technologies</a>
eliminate those dependencies though kernel >=5.8 should be used.</p>
<p>Since I don't know C, I resorted to <a href="https://github.com/iovisor/bcc/tree/master/libbpf-tools">BCC libbpf-tools</a>,
from which I picked a few BPF C programs (tcpconnect.bpf.c, execsnoop.bpf.c) to write Go frontends.
Below I will be focusing on execsnoop,
and you can find my experiments at <a href="https://github.com/marselester/libbpf-tools">github.com/marselester/libbpf-tools</a>.</p>
<h2>Preparing the environment</h2>
<p>First things first, I ran Ubuntu 21.04 in a virtual machine and installed Clang, Go.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ cat > Vagrantfile <<CFG</span>
<span class="go">Vagrant.configure("2") do |config|</span>
<span class="go"> config.vm.box = "ubuntu/hirsute64"</span>
<span class="go">end</span>
<span class="go">CFG</span>
<span class="go">οΉͺ vagrant up</span>
<span class="go">οΉͺ vagrant ssh</span>
<span class="go">οΉ© sudo apt-get update</span>
<span class="go">οΉ© sudo apt-get install clang</span>
<span class="go">οΉ© sudo snap install go --classic</span>
<span class="go">οΉ© uname -nr</span>
<span class="go">ubuntu-hirsute 5.11.0-37-generic</span>
<span class="go">οΉ© clang -v</span>
<span class="go">Ubuntu clang version 12.0.0-3ubuntu1~21.04.2</span>
</code></pre></div>
<p>Then I made sure the execsnoop program from BCC toolkit showed new processes
(it prints one line for every <code>execve</code> syscall),
so I could compare its output later with the Go implementation.</p>
<p>The ELF binary <code>execsnoop</code> was copied from a Docker container to the virtual machine as follows.
In case you wanted to build the image yourself,
here is a <a href="https://gist.github.com/marselester/ac8e4262742c90539c8c53f37d9a6965">Dockerfile</a>.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉͺ docker run --name libbpf marselester/libbpf-tools:latest</span>
<span class="go">οΉͺ docker cp libbpf:/opt/execsnoop .</span>
</code></pre></div>
<p>The execsnoop captured <code>sshd</code> and <code>ls</code> processes, nice!</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo /vagrant/execsnoop</span>
<span class="go">PCOMM PID PPID RET ARGS</span>
<span class="go">sshd 21254 832 0 /usr/sbin/sshd -D -R</span>
<span class="go">ls 21361 21351 0 /usr/bin/ls --color=auto</span>
</code></pre></div>
<p>It can be useful to check for short-lived processes that consume CPU resources,
e.g., slow or failing application/container startup.</p>
<h2>Embedding BPF in Go</h2>
<p>My search for a BPF Go library stopped at <a href="https://github.com/cilium/ebpf">github.com/cilium/ebpf</a>
since it had several examples to build upon, and folks from Cilium were <a href="https://github.com/cilium/ebpf/issues/303">very helpful</a>.</p>
<p>The repository also contains <a href="https://github.com/cilium/ebpf/tree/master/cmd/bpf2go">bpf2go</a> tool to compile a BPF C source file into BPF bytecode.
It emits Go files containing compiled BPF for little and big endian systems.</p>
<p>I copied <a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/execsnoop.bpf.c">execsnoop.bpf.c</a>,
<a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/execsnoop.h">execsnoop.h</a> files from BCC toolkit
and tried to generate Go files.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© cd /vagrant/</span>
<span class="go">οΉ© mkdir -p cmd/execsnoop/bpf/</span>
<span class="go">οΉ© curl -o cmd/execsnoop/bpf/execsnoop.bpf.c https://raw.githubusercontent.com/iovisor/bcc/master/libbpf-tools/execsnoop.bpf.c</span>
<span class="go">οΉ© curl -o cmd/execsnoop/bpf/execsnoop.h https://raw.githubusercontent.com/iovisor/bcc/master/libbpf-tools/execsnoop.h</span>
<span class="go">οΉ© cat > cmd/execsnoop/main.go <<<'''</span>
<span class="go">package main</span>
<span class="go">//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cflags $BPF_CFLAGS -cc clang-12 ExecSnoop ./bpf/execsnoop.bpf.c -- -I../../headers</span>
<span class="go">func main() {}</span>
<span class="go">'''</span>
<span class="go">οΉ© go mod init execsnoop</span>
<span class="go">οΉ© go get github.com/cilium/ebpf/cmd/bpf2go</span>
<span class="go">οΉ© go generate ./cmd/execsnoop/</span>
<span class="go">/vagrant/cmd/execsnoop/bpf/execsnoop.bpf.c:2:10: fatal error: 'vmlinux.h' file not found</span>
</code></pre></div>
<p>Unfortunately that didn't work because the <code>vmlinux.h</code> header was missing.
This file contains kernel's type definitions that can be generated from the installed kernel using bpftool tool.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo apt-get install linux-tools-$(uname -r) linux-tools-common</span>
<span class="go">οΉ© bpftool btf dump file /sys/kernel/btf/vmlinux format c > ./headers/vmlinux.h</span>
</code></pre></div>
<p>There were three more header files missing that I copied from ubuntu-kernel and libbpf repositories.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© mkdir -p headers/bpf/</span>
<span class="go">οΉ© curl -o ./headers/bpf/bpf_helpers.h https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/hirsute/plain/tools/lib/bpf/bpf_helpers.h</span>
<span class="go">οΉ© curl -o ./headers/bpf/bpf_helper_defs.h https://raw.githubusercontent.com/libbpf/libbpf/master/src/bpf_helper_defs.h</span>
<span class="go">οΉ© curl -o ./headers/bpf/bpf_core_read.h https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/hirsute/plain/tools/lib/bpf/bpf_core_read.h</span>
</code></pre></div>
<p>Finally the Go files were generated and the BPF program was loaded into the kernel from an ELF.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© go generate ./cmd/execsnoop/</span>
<span class="go">Compiled /vagrant/cmd/execsnoop/execsnoop_bpfel.o</span>
<span class="go">Wrote /vagrant/cmd/execsnoop/execsnoop_bpfel.go</span>
<span class="go">Compiled /vagrant/cmd/execsnoop/execsnoop_bpfeb.o</span>
<span class="go">Wrote /vagrant/cmd/execsnoop/execsnoop_bpfeb.go</span>
<span class="go">οΉ© sudo go run ./cmd/execsnoop/</span>
</code></pre></div>
<p>Note, bpf2go tool uses Clang to compile the BPF program:</p>
<ul>
<li><code>-cflags $BPF_CFLAGS</code> are flags passed to the compiler from the <code>$BPF_CFLAGS</code> env var,
for example, <code>BPF_CFLAGS='-D__TARGET_ARCH_x86' go generate</code> is expected in
<a href="https://elixir.bootlin.com/linux/latest/source/tools/lib/bpf/bpf_tracing.h#L5">bpf_tracing.h</a> header
(it is not used in execsnoop though)</li>
<li><code>-cc clang-12</code> is a Clang binary name</li>
<li><code>ExecSnoop</code> is a name used by a code generation</li>
<li><code>./bpf/execsnoop.bpf.c</code> is a path to the BPF C file that is compiled</li>
<li><code>-- -I../../headers</code> are compiler options that indicate where to find header files</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"log"</span><span class="w"></span>
<span class="w"> </span><span class="s">"golang.org/x/sys/unix"</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="c1">//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cflags $BPF_CFLAGS -cc clang-12 ExecSnoop ./bpf/execsnoop.bpf.c -- -I../../headers</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Increase the resource limit of the current process to provide sufficient space</span><span class="w"></span>
<span class="w"> </span><span class="c1">// for locking memory for the BPF maps.</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">Setrlimit</span><span class="p">(</span><span class="nx">unix</span><span class="p">.</span><span class="nx">RLIMIT_MEMLOCK</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">unix</span><span class="p">.</span><span class="nx">Rlimit</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Cur</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">RLIM_INFINITY</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Max</span><span class="p">:</span><span class="w"> </span><span class="nx">unix</span><span class="p">.</span><span class="nx">RLIM_INFINITY</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to set temporary RLIMIT_MEMLOCK: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Load the BPF program into the kernel from an ELF.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ExecSnoopObjects contains all objects (BPF programs and maps) after they have been loaded into the kernel:</span><span class="w"></span>
<span class="w"> </span><span class="c1">// TracepointSyscallsSysEnterExecve and TracepointSyscallsSysExitExecve BPF programs,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Events and Execs BPF maps.</span><span class="w"></span>
<span class="w"> </span><span class="nx">objs</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ExecSnoopObjects</span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">LoadExecSnoopObjects</span><span class="p">(</span><span class="o">&</span><span class="nx">objs</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to load BPF programs and maps: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<h2>Tracepoints</h2>
<p>The BPF program is executed on events such as tracepoints.
Tracepoint is a static instrumentation (hard-coded to the source code) of the Linux kernel,
see <code>/sys/kernel/debug/tracing/events</code> to find available tracepoints.</p>
<p>From looking at BPF C source file <code>execsnoop.bpf.c</code> we can see two tracepoints instrumented:
the entry and the return of the <code>execve</code> syscall so that
the process ID, command name, arguments, and the return value can be printed.</p>
<div class="highlight"><pre><span></span><code><span class="n">SEC</span><span class="p">(</span><span class="s">"tracepoint/syscalls/sys_enter_execve"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">tracepoint__syscalls__sys_enter_execve</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">trace_event_raw_sys_enter</span><span class="w"> </span><span class="o">*</span><span class="n">ctx</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="n">SEC</span><span class="p">(</span><span class="s">"tracepoint/syscalls/sys_exit_execve"</span><span class="p">)</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">tracepoint__syscalls__sys_exit_execve</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">trace_event_raw_sys_exit</span><span class="w"> </span><span class="o">*</span><span class="n">ctx</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
</code></pre></div>
<p><code>SEC</code> declares an ELF section named <code>tracepoint/syscalls/sys_enter_execve</code>, followed by a BPF program.
A user-level loader may use this section header to determine where to attach a program.</p>
<p><code>tracepoint__syscalls__sys_enter_execve</code> function is called for the tracepoint event.
The struct <code>trace_event_raw_sys_enter</code> argument contains register state and BPF context.
From the registers, function arguments <code>ctx->args</code> and return value <code>ctx->ret</code> can be read.</p>
<p>On Go side I attached the BPF program to <code>syscalls/sys_enter_execve</code> tracepoint using <code>github.com/cilium/ebpf/link</code> package.
The <code>link.Tracepoint()</code> function returns a link (it represents a program attached to a BPF hook)
which is later used to detach the program.</p>
<div class="highlight"><pre><span></span><code><span class="nx">tpEnter</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">link</span><span class="p">.</span><span class="nx">Tracepoint</span><span class="p">(</span><span class="s">"syscalls"</span><span class="p">,</span><span class="w"> </span><span class="s">"sys_enter_execve"</span><span class="p">,</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">ExecSnoopPrograms</span><span class="p">.</span><span class="nx">TracepointSyscallsSysEnterExecve</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to attach the BPF program to sys_enter_execve tracepoint: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">tpEnter</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="nx">tpExit</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">link</span><span class="p">.</span><span class="nx">Tracepoint</span><span class="p">(</span><span class="s">"syscalls"</span><span class="p">,</span><span class="w"> </span><span class="s">"sys_exit_execve"</span><span class="p">,</span><span class="w"> </span><span class="nx">objs</span><span class="p">.</span><span class="nx">ExecSnoopPrograms</span><span class="p">.</span><span class="nx">TracepointSyscallsSysExitExecve</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to attach the BPF program to sys_exit_execve tracepoint: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">tpExit</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
</code></pre></div>
<p>The <code>bpf_perf_event_output()</code> C function emits records to user space via a BPF map <code>events</code>
that accesses perf per-CPU ring buffers.</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">type</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_MAP_TYPE_PERF_EVENT_ARRAY</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">key_size</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">u32</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">value_size</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">u32</span><span class="p">));</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">events</span><span class="w"> </span><span class="n">SEC</span><span class="p">(</span><span class="s">".maps"</span><span class="p">);</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="w"> </span><span class="n">e</span><span class="p">;</span><span class="w"></span>
<span class="n">bpf_perf_event_output</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">events</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_F_CURRENT_CPU</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">e</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">e</span><span class="p">));</span><span class="w"></span>
</code></pre></div>
<p>In user space the events are read using <code>github.com/cilium/ebpf/perf</code> package.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Open a perf event reader from user space on the PERF_EVENT_ARRAY map</span><span class="w"></span>
<span class="c1">// defined in the BPF C program.</span><span class="w"></span>
<span class="nx">rd</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">perf</span><span class="p">.</span><span class="nx">NewReader</span><span class="p">(</span><span class="nx">objs</span><span class="p">.</span><span class="nx">ExecSnoopMaps</span><span class="p">.</span><span class="nx">Events</span><span class="p">,</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Getpagesize</span><span class="p">())</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to create perf event reader: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">rd</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">record</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">rd</span><span class="p">.</span><span class="nx">Read</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">perf</span><span class="p">.</span><span class="nx">IsClosed</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to read from perf ring buffer: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"received from perf ring buffer: %s"</span><span class="p">,</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">RawSample</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>In order to verify that the events are captured, I ran the Go program and <code>ls</code> in another terminal.
Luckily it worked!</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/execsnoop/</span>
<span class="go">received from perf ring buffer: ?????ls/usr/bin/ls--color=auto</span>
</code></pre></div>
<h2>Decoding events</h2>
<p>I wanted to match the output of the original execsnoop program, for that I needed to decode <code>record.RawSample</code>
which contains bytes of the <code>event</code> C struct into a Go struct.</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">pid_t</span><span class="w"> </span><span class="n">pid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">pid_t</span><span class="w"> </span><span class="n">ppid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">uid_t</span><span class="w"> </span><span class="n">uid</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">int32</span><span class="w"> </span><span class="n">retval</span><span class="p">;</span><span class="w"> </span><span class="c1">// it's int in https://raw.githubusercontent.com/iovisor/bcc/master/libbpf-tools/execsnoop.h</span>
<span class="w"> </span><span class="n">int32</span><span class="w"> </span><span class="n">args_count</span><span class="p">;</span><span class="w"> </span><span class="c1">// int</span>
<span class="w"> </span><span class="n">u32</span><span class="w"> </span><span class="n">args_size</span><span class="p">;</span><span class="w"> </span><span class="c1">// unsigned int</span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">comm</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">args</span><span class="p">[</span><span class="mi">7680</span><span class="p">];</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</code></pre></div>
<p>Go <code>event</code> struct represents a perf event sent to user space from the BPF program running in the kernel.
Note, that it should match the C event struct, and both C and Go structs must be aligned the same way.</p>
<div class="highlight"><pre><span></span><code><span class="kd">type</span><span class="w"> </span><span class="nx">event</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PID is the process ID.</span><span class="w"></span>
<span class="w"> </span><span class="nx">PID</span><span class="w"> </span><span class="kt">int32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// PPID is the process ID of the parent of this process.</span><span class="w"></span>
<span class="w"> </span><span class="nx">PPID</span><span class="w"> </span><span class="kt">int32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// UID is the process user ID, e.g., 1000.</span><span class="w"></span>
<span class="w"> </span><span class="nx">UID</span><span class="w"> </span><span class="kt">uint32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Retval is the return value of the execve().</span><span class="w"></span>
<span class="w"> </span><span class="nx">Retval</span><span class="w"> </span><span class="kt">int32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ArgsCount is a number of arguments.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ArgsCount</span><span class="w"> </span><span class="kt">int32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ArgSize is a size of arguments in bytes.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ArgsSize</span><span class="w"> </span><span class="kt">uint32</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Comm is the parent process/command name, e.g., bash.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Comm</span><span class="w"> </span><span class="p">[</span><span class="mi">16</span><span class="p">]</span><span class="kt">byte</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>The variable length <code>args</code> field is omitted on Go side and decoded manually,
because I kept getting <code>failed to parse perf event: unexpected EOF error</code>.
Perhaps the Go program got trailing garbage from the ring buffer which couldn't be unmarshalled.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// The data submitted via bpf_perf_event_output.</span><span class="w"></span>
<span class="c1">// Due to a kernel bug, this can contain between 0 and 7 bytes of trailing</span><span class="w"></span>
<span class="c1">// garbage from the ring depending on the input sample's length.</span><span class="w"></span>
<span class="c1">// See https://pkg.go.dev/github.com/cilium/ebpf/perf#Record.</span><span class="w"></span>
<span class="nx">RawSample</span><span class="w"> </span><span class="p">[]</span><span class="kt">byte</span><span class="w"></span>
</code></pre></div>
<p>Timo Beckers in Cilium's Slack channel recommended to check alignment and hand-write an unmarshaller.
Here you can find my attempts to verify alignments <a href="https://github.com/marselester/libbpf-tools/pull/3">#3</a>
with alignchecker program.</p>
<blockquote>
<p>Everything up until ArgsCount should be aligned on 8 bytes, but I'd expect the Go compiler to insert 4 bytes of padding between ArgsSize and Args, which would lead to an underrun on the C side. I've found hand-writing binary unmarshalers for these structs to be the most robust, but also the most annoying if your event format changes frequently.</p>
<p>Cilium has a few alignchecker.go files if you're looking to dive deeper, we compare the DWARF info from the eBPF ELFs to Go struct fields annotated with align tags to make sure C and Go alignment corresponds</p>
</blockquote>
<p><code>binary.Read()</code> is used to read structured binary data from <code>record.RawSample</code> into the <code>event</code> struct.
The remaining bytes belong to <code>args</code> field and just printed.</p>
<div class="highlight"><pre><span></span><code><span class="c1">// eventSize is the event struct size in bytes, i.e., unsafe.Sizeof(e).</span><span class="w"></span>
<span class="c1">// It's used to obtain the variable length args from the perf record.</span><span class="w"></span>
<span class="kd">const</span><span class="w"> </span><span class="nx">eventSize</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="mi">40</span><span class="w"></span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"%-16s %-6s %-6s %3s %s\n"</span><span class="p">,</span><span class="w"> </span><span class="s">"PCOMM"</span><span class="p">,</span><span class="w"> </span><span class="s">"PID"</span><span class="p">,</span><span class="w"> </span><span class="s">"PPID"</span><span class="p">,</span><span class="w"> </span><span class="s">"RET"</span><span class="p">,</span><span class="w"> </span><span class="s">"ARGS"</span><span class="p">)</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">record</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">rd</span><span class="p">.</span><span class="nx">Read</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">perf</span><span class="p">.</span><span class="nx">IsClosed</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to read from perf ring buffer: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">LostSamples</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"ring event perf buffer is full, dropped %d samples"</span><span class="p">,</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">LostSamples</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">continue</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">e</span><span class="w"> </span><span class="nx">event</span><span class="w"></span>
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">Read</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBuffer</span><span class="p">(</span><span class="nx">record</span><span class="p">.</span><span class="nx">RawSample</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">binary</span><span class="p">.</span><span class="nx">LittleEndian</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="nx">e</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"failed to parse perf event: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">continue</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">args</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">record</span><span class="p">.</span><span class="nx">RawSample</span><span class="p">[</span><span class="nx">eventSize</span><span class="p">:]</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"%-16s %-6d %-6d %3d %s\n"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">TrimRight</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">Comm</span><span class="p">[:],</span><span class="w"> </span><span class="s">"\x00"</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">PID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">PPID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">Retval</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">ReplaceAll</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">bytes</span><span class="p">.</span><span class="nx">Trim</span><span class="p">(</span><span class="nx">args</span><span class="p">,</span><span class="w"> </span><span class="s">"\x00"</span><span class="p">)),</span><span class="w"></span>
<span class="w"> </span><span class="s">"\x00"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">" "</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>The resulting output of the Go execsnoop program is very close to the original π.</p>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo go run ./cmd/execsnoop/</span>
<span class="go">PCOMM PID PPID RET ARGS</span>
<span class="go">ls 69672 60093 0 /usr/bin/ls --color=auto</span>
<span class="go">sh 69783 69782 0 /bin/sh -c cd / && run-parts --report /etc/cron.hourly</span>
<span class="go">run-parts 69784 69783 0 /usr/bin/run-parts --report /etc/cron.hourly</span>
</code></pre></div>
<p>You can find all the source code at
<a href="https://github.com/marselester/libbpf-tools">github.com/marselester/libbpf-tools</a>.</p>
<h2>Appendix</h2>
<p><a href="https://www.brendangregg.com/bpf-performance-tools-book.html">BPF performance tools</a> book
aided to navigate BPF C code.
Here are a few notes that are relevant to execsnoop program.</p>
<p>A BPF program has to use <a href="https://elixir.bootlin.com/linux/v5.11/source/include/uapi/linux/bpf.h#L712">helpers</a>
because it can't access arbitrary memory (outside of BPF) and can't call arbitrary kernel functions.
The <code>execsnoop.bpf.c</code> makes use of the following helpers.</p>
<p><code>bpf_probe_read_user(dst, size, unsafe_ptr)</code> safely attempts to read <em>size</em> bytes from user space address
<em>unsafe_ptr</em> and stores the data in <em>dst</em>.</p>
<p><code>bpf_get_current_pid_tgid()</code> returns an unsigned 64-bit integer containing the current TGID (what user space calls the PID) in the upper bits
and the current PID (what user space calls the kernel thread ID) in the lower bits.</p>
<div class="highlight"><pre><span></span><code><span class="n">u64</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_get_current_pid_tgid</span><span class="p">();</span><span class="w"></span>
<span class="kt">pid_t</span><span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">pid_t</span><span class="p">)</span><span class="n">id</span><span class="p">;</span><span class="w"></span>
<span class="kt">pid_t</span><span class="w"> </span><span class="n">tgid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">>></span><span class="w"> </span><span class="mi">32</span><span class="p">;</span><span class="w"></span>
<span class="n">e</span><span class="o">-></span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tgid</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p><code>bpf_get_current_uid_gid()</code> returns a u64 integer containing the current GID in the upper bits
and UID in the lower bits.</p>
<div class="highlight"><pre><span></span><code><span class="kt">uid_t</span><span class="w"> </span><span class="n">uid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">u32</span><span class="p">)</span><span class="n">bpf_get_current_uid_gid</span><span class="p">();</span><span class="w"></span>
</code></pre></div>
<p><code>bpf_get_current_task()</code> returns a pointer to the current (on-CPU)
<a href="https://elixir.bootlin.com/linux/v5.11/source/include/linux/sched.h#L649">task struct</a> (a process).
This contains many details about the running process and links to other structs containing system state.</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="w"> </span><span class="o">*</span><span class="n">t</span><span class="p">;</span><span class="w"></span>
<span class="n">t</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">task_struct</span><span class="o">*</span><span class="p">)</span><span class="n">bpf_get_current_task</span><span class="p">();</span><span class="w"></span>
</code></pre></div>
<p><code>bpf_get_current_comm(buf, buf_size)</code> copies the task name to the buffer.</p>
<div class="highlight"><pre><span></span><code><span class="n">bpf_get_current_comm</span><span class="p">(</span><span class="o">&</span><span class="n">e</span><span class="o">-></span><span class="n">comm</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">e</span><span class="o">-></span><span class="n">comm</span><span class="p">));</span><span class="w"></span>
</code></pre></div>
<p><code>bpf_perf_event_output(ctx, map, data, size)</code> writes data to the <code>perf_event</code> ring buffers;
this is used for per-event output.</p>
<p><code>bpf_map_lookup_elem(map, key)</code> finds a key in a map and returns its value (pointer).</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="w"> </span><span class="o">*</span><span class="n">e</span><span class="p">;</span><span class="w"></span>
<span class="n">e</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">bpf_map_lookup_elem</span><span class="p">(</span><span class="o">&</span><span class="n">execs</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">pid</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p><code>bpf_map_update_elem(map, key, value, flags)</code> atomically updates the value of the entry selected by key.</p>
<div class="highlight"><pre><span></span><code><span class="k">static</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="w"> </span><span class="n">empty_event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">type</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_MAP_TYPE_HASH</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__uint</span><span class="p">(</span><span class="n">max_entries</span><span class="p">,</span><span class="w"> </span><span class="mi">10240</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__type</span><span class="p">(</span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="kt">pid_t</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">__type</span><span class="p">(</span><span class="n">value</span><span class="p">,</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">event</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">execs</span><span class="w"> </span><span class="n">SEC</span><span class="p">(</span><span class="s">".maps"</span><span class="p">);</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">bpf_map_update_elem</span><span class="p">(</span><span class="o">&</span><span class="n">execs</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">empty_event</span><span class="p">,</span><span class="w"> </span><span class="n">BPF_NOEXIST</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>The <em>execs</em> is declared of type <code>BPF_MAP_TYPE_HASH</code> (a hash table).
The key is a <code>pid_t</code>, and the value is struct <code>event</code>.
<em>BPF_NOEXIST</em> flag specifies that the entry for key must not exist in the map.</p>
<p>BPF supports the following map types:</p>
<ul>
<li><code>BPF_MAP_TYPE_HASH</code> is a hash table: key/value pairs</li>
<li><code>BPF_MAP_TYPE_ARRAY</code> is an array of elements</li>
<li><code>BPF_MAP_TYPE_PERF_EVENT_ARRAY</code> is an interface to the <code>perf_event</code> ring buffers
for emitting trace records to user space</li>
<li><code>BPF_MAP_TYPE_PERCPU_HASH</code> is a faster hash table maintained on a per-CPU basis</li>
<li><code>BPF_MAP_TYPE_STACK_TRACE</code> is a storage for stack traces, indexed by stack IDs</li>
<li><code>BPF_MAP_TYPE_STACK</code> is a storage for stack traces</li>
</ul>
<p>There are three ways to output data from kernel to user:</p>
<ul>
<li><code>BPF_PERF_OUTPUT()</code> is a way to send per-event details to user space, via a custom struct you define.</li>
<li>BPF maps using <code>bpf_perf_event_output()</code> (periodically read from user space)</li>
<li><code>bpf_trace_printk()</code> writes to trace_pipe (debugging only).
I wasn't sure about <code>pid_t</code> size so I used <code>bpf_printk("the PID is %d", sizeof(pid));</code>
and printed the trace buffer as follows (see <a href="https://nakryiko.com/posts/bpf-tips-printk/">BPF tips & tricks</a>).</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="go">οΉ© sudo cat /sys/kernel/debug/tracing/trace_pipe</span>
<span class="go"><...>-487422 [001] .... 544705.474270: 0: the PID is 4</span>
</code></pre></div>Ambassador as API Gateway2019-03-13T00:00:00+07:002019-03-13T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2019-03-13:/apigate-ambassador.html<p><a class="reference external" href="https://docs.microsoft.com/en-us/azure/architecture/microservices/design/gateway">API gateway</a>
acts as a reverse proxy, routing API requests from clients to services.
Usually it also performs authentication and rate limiting, so the services behind the gate don't have to.
In this short tutorial we'll see how to achieve that with <a class="reference external" href="https://getambassador.io/">Ambassador</a>.</p>
<p>The demo is based on a dummy β¦</p><p><a class="reference external" href="https://docs.microsoft.com/en-us/azure/architecture/microservices/design/gateway">API gateway</a>
acts as a reverse proxy, routing API requests from clients to services.
Usually it also performs authentication and rate limiting, so the services behind the gate don't have to.
In this short tutorial we'll see how to achieve that with <a class="reference external" href="https://getambassador.io/">Ambassador</a>.</p>
<p>The demo is based on a dummy <a class="reference external" href="https://traveling.docs.apiary.io/">Traveling project</a>
where we have services to rent a car and book a hotel.</p>
<p>Firstly, we shall install Ambassador on <a class="reference external" href="http://kubernetes.io/docs/getting-started-guides/minikube/">Minikube</a>
by following <a class="reference external" href="https://www.getambassador.io/user-guide/getting-started/">Ambassador user guide</a>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>minikube start
<span class="go">Starting local Kubernetes v1.13.2 cluster...</span>
<span class="gp">$ </span>kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-rbac.yaml
</pre></div>
<p>Next, create Kubernetes service for Ambassador deployment of type NodePort, so it's easily accessible outside of the cluster.
It will be the entry point of our API gateway.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f - <span class="o"><<<</span><span class="err">'</span>
<span class="go">apiVersion: v1</span>
<span class="go">kind: Service</span>
<span class="go">metadata:</span>
<span class="go"> name: ambassador</span>
<span class="go">spec:</span>
<span class="go"> selector:</span>
<span class="go"> service: ambassador</span>
<span class="go"> type: NodePort</span>
<span class="go"> ports:</span>
<span class="go"> - port: 80</span>
<span class="gp"> # </span>Propagate the original <span class="nb">source</span> IP of the client.
<span class="go"> externalTrafficPolicy: Local</span>
<span class="go">'</span>
<span class="gp">$ </span><span class="nv">AMBASSADORURL</span><span class="o">=</span><span class="k">$(</span>minikube service --url ambassador<span class="k">)</span>
</pre></div>
<p>Note, there is a diagnostic web UI at <cite>$AMBASSADORURL/ambassador/v0/diag/</cite>.
You should restrict access to it from your custom auth service.</p>
<div class="section" id="travel-apps">
<h2>Travel Apps</h2>
<p>Let's deploy the travel apps on Kubernetes. For simplicity, you can use the images uploaded to Docker Hub.
If you prefer to build Docker images yourself, make sure they are accessible from Minikube:
configure Docker client to communicate with the Minikube Docker daemon.
Note, you can restore your local Docker environment with <cite>eval $(minikube docker-env --unset)</cite> later on.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>git clone https://github.com/marselester/apigate.git
<span class="gp">$ </span><span class="nb">cd</span> ./apigate/
<span class="gp">$ </span><span class="nb">eval</span> <span class="k">$(</span>minikube docker-env<span class="k">)</span>
<span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-hotel:v1.0.0 --file<span class="o">=</span>docker/hotel.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/hotel/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/hotel/service.yml
</pre></div>
<p>The hotel app should be running:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl get pods -l <span class="nv">app</span><span class="o">=</span>hotel
<span class="go">hotel-api-8d7d59b69-bcsvh</span>
<span class="gp">$ </span>kubectl port-forward hotel-api-8d7d59b69-bcsvh <span class="m">8000</span>
<span class="gp">$ </span>curl localhost:8000/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">{</span>
<span class="go"> "id": "7b4fc183-ee67-494d-9715-3510c6d8f2ef",</span>
<span class="go"> "hotel_id": "046d471d-70c7-4595-80cc-266d3e6e07fa",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
<p>Now, it's time to put the <cite>hotel</cite> Service behind the gateway by adding annotations to the service.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hotel</span><span class="w"></span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">getambassador.io/config</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span><span class="w"></span>
<span class="w"> </span><span class="no">---</span><span class="w"></span>
<span class="w"> </span><span class="no">apiVersion: ambassador/v1</span><span class="w"></span>
<span class="w"> </span><span class="no">kind: Mapping</span><span class="w"></span>
<span class="w"> </span><span class="no">name: hotel_mapping</span><span class="w"></span>
<span class="w"> </span><span class="no">prefix: /v1/hotels</span><span class="w"></span>
<span class="w"> </span><span class="no">rewrite: ""</span><span class="w"></span>
<span class="w"> </span><span class="no">service: hotel</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hotel</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NodePort</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
<span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8000</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f k8s-ambassador/hotel/service-gate.yml
</pre></div>
<p>By default, Ambassador would rewrite <cite>/v1/hotels</cite> prefix to <cite>/</cite>.
With <cite>rewrite: ""</cite> directive we configure Ambassador to not change the prefix as
it forwards a request to the <cite>hotel</cite> service.
Let's confirm that hotel booking requests are routed to the hotel app:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl <span class="nv">$AMBASSADORURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">{</span>
<span class="go"> "id": "7b4fc183-ee67-494d-9715-3510c6d8f2ef",</span>
<span class="go"> "hotel_id": "046d471d-70c7-4595-80cc-266d3e6e07fa",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
<p>The car rental app is deployed similarly.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">car</span><span class="w"></span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">getambassador.io/config</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span><span class="w"></span>
<span class="w"> </span><span class="no">---</span><span class="w"></span>
<span class="w"> </span><span class="no">apiVersion: ambassador/v1</span><span class="w"></span>
<span class="w"> </span><span class="no">kind: Mapping</span><span class="w"></span>
<span class="w"> </span><span class="no">name: car_mapping</span><span class="w"></span>
<span class="w"> </span><span class="no">prefix: /v1/cars</span><span class="w"></span>
<span class="w"> </span><span class="no">rewrite: ""</span><span class="w"></span>
<span class="w"> </span><span class="no">service: car</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">car</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NodePort</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
<span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8000</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-car:v1.0.0 --file<span class="o">=</span>docker/car.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/car/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/car/service-gate.yml
</pre></div>
<p>You can see that the car booking is available at the gateway.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl <span class="nv">$AMBASSADORURL</span>/v1/cars/bookings/9e0d65f5-9de2-4428-9bee-1f3967f05129
<span class="go">{</span>
<span class="go"> "id": "9e0d65f5-9de2-4428-9bee-1f3967f05129",</span>
<span class="go"> "car_id": "cfb6f7a5-4591-4f5c-8b17-9a1b10f98ada",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="authentication-service">
<h2>Authentication Service</h2>
<p>API requests must be authenticated before reaching the Travel apps.
This work is delegated to <cite>travelauth</cite> Service that performs HTTP Basic authentication
and returns a username in <cite>X-Travel-User</cite> header if credentials matched.
We shall start from deploying it on the cluster.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-auth:v1.0.0 --file<span class="o">=</span>docker/auth.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/auth/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/auth/service.yml
</pre></div>
<p>If the auth server is properly deployed, it will prompt for username/password.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl get pods -l <span class="nv">app</span><span class="o">=</span>auth
<span class="go">auth-api-556685f658-h9qb4</span>
<span class="gp">$ </span>kubectl port-forward auth-api-556685f658-h9qb4 <span class="m">8000</span>
<span class="gp">$ </span>curl -i -u bob:bob localhost:8000/v1/hotels
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">X-Travel-User: bob</span>
</pre></div>
<p>Let's tell Ambassador to
<a class="reference external" href="https://www.getambassador.io/reference/services/auth-service/">forward all requests to the auth server</a>
and copy <cite>X-Travel-User</cite> response header from the auth server to the routed request.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">travelauth</span><span class="w"></span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">getambassador.io/config</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span><span class="w"></span>
<span class="w"> </span><span class="no">---</span><span class="w"></span>
<span class="w"> </span><span class="no">apiVersion: ambassador/v1</span><span class="w"></span>
<span class="w"> </span><span class="no">kind: AuthService</span><span class="w"></span>
<span class="w"> </span><span class="no">name: gate_auth</span><span class="w"></span>
<span class="w"> </span><span class="no"># This is k8s service name; all API requests are sent there. For example,</span><span class="w"></span>
<span class="w"> </span><span class="no"># API request /v1/hotels will be sent to http://travelauth:80/v1/hotels.</span><span class="w"></span>
<span class="w"> </span><span class="no">auth_service: travelauth</span><span class="w"></span>
<span class="w"> </span><span class="no">proto: http</span><span class="w"></span>
<span class="w"> </span><span class="no"># The travelauth service adds a username into a header after successful authentication,</span><span class="w"></span>
<span class="w"> </span><span class="no"># so all the other services know who the user is (ratelimit, hotel, car services).</span><span class="w"></span>
<span class="w"> </span><span class="no">allowed_authorization_headers:</span><span class="w"></span>
<span class="w"> </span><span class="no">- "X-Travel-User"</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">auth</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NodePort</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
<span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8000</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f k8s-ambassador/auth/service-auth.yml
</pre></div>
<p>With updated config Ambassador should enforce authentication on API gateway.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -i <span class="nv">$AMBASSADORURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">HTTP/1.1 401 Unauthorized</span>
<span class="gp">$ </span>curl -i -u bob:bob <span class="nv">$AMBASSADORURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">{</span>
<span class="go"> "id": "7b4fc183-ee67-494d-9715-3510c6d8f2ef",</span>
<span class="go"> "hotel_id": "046d471d-70c7-4595-80cc-266d3e6e07fa",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="rate-limiting-service">
<h2>Rate Limiting Service</h2>
<p>With Ambassador, individual requests can be annotated with metadata, called labels.
These labels can then be passed to a third party rate limiting service through a gRPC interface
which implements actual rate limit logic.
Check out <a class="reference external" href="https://www.getambassador.io/user-guide/rate-limiting-tutorial/">Node.js</a> and
<a class="reference external" href="https://blog.getambassador.io/designing-a-rate-limiting-service-for-ambassador-f460e9fabedb">Java</a> based examples.</p>
<p>In order to build such service, it must support Envoy's <a class="reference external" href="https://github.com/datawire/ambassador/blob/master/ambassador/common/ratelimit/ratelimit.proto">ratelimit.proto</a> interface.
The protocol buffer definition of RateLimitService service is described in <cite>./internal/pb/ratelimit.proto</cite>.
You need to <a class="reference external" href="https://grpc.io/docs/quickstart/go.html">install</a> protoc compiler and protoc plugin for Go
(beware of <a class="reference external" href="https://github.com/golang/protobuf/issues/763">ProtoPackageIsVersion3 issue</a>)
to generate gRPC service code.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>brew install protobuf
<span class="gp">$ </span>go get -u github.com/golang/protobuf/protoc-gen-go
<span class="gp">$ </span><span class="nb">cd</span> <span class="nv">$GOPATH</span>/src/github.com/golang/protobuf/protoc-gen-go
<span class="gp">$ </span>git checkout v1.2.0
<span class="gp">$ </span>go install
</pre></div>
<p>The arguments tell protoc to use <cite>ratelimit.proto</cite> definition,
search for imports in <cite>./internal/pb/</cite> dir, generate Go code using gprc plugin,
and place the result in <cite>./internal/pb/</cite> dir.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>protoc ratelimit.proto -I internal/pb/ --go_out<span class="o">=</span><span class="nv">plugins</span><span class="o">=</span>grpc:internal/pb/
</pre></div>
<p>We now have a newly generated gRPC server and client code in <cite>./internal/pb/ratelimit.pb.go</cite>.
Our ratelimit server already implements <cite>RateLimitServiceServer</cite> interface.</p>
<div class="highlight"><pre><span></span><span class="c1">// ShouldRateLimit must respond to the request with an OK or OVER_LIMIT code.</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">s</span><span class="w"> </span><span class="o">*</span><span class="nx">server</span><span class="p">)</span><span class="w"> </span><span class="nx">ShouldRateLimit</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="w"> </span><span class="o">*</span><span class="nx">pb</span><span class="p">.</span><span class="nx">RateLimitRequest</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="nx">pb</span><span class="p">.</span><span class="nx">RateLimitResponse</span><span class="p">,</span><span class="w"> </span><span class="kt">error</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Descriptors is a list of labels on which the rate limit service can base</span><span class="w"></span>
<span class="w"> </span><span class="c1">// its decision to accept or reject the request.</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"%s: %+v"</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">GetDomain</span><span class="p">(),</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">GetDescriptors</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="nx">resp</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">pb</span><span class="p">.</span><span class="nx">RateLimitResponse</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">OverallCode</span><span class="p">:</span><span class="w"> </span><span class="nx">pb</span><span class="p">.</span><span class="nx">RateLimitResponse_OVER_LIMIT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Let's deploy it on the cluster and send a few requests using
<a class="reference external" href="https://github.com/fullstorydev/grpcurl">grpcurl</a>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-ratelimit:v1.0.0 --file<span class="o">=</span>docker/ratelimit.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/ratelimit/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-ambassador/ratelimit/service.yml
<span class="gp">$ </span>kubectl get pods -l <span class="nv">app</span><span class="o">=</span>ratelimit
<span class="go">ratelimit-api-5554898589-4gmv5</span>
<span class="gp">$ </span>kubectl port-forward ratelimit-api-5554898589-4gmv5 <span class="m">5000</span>
</pre></div>
<p>As we can see the server exposes RateLimitService.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>grpcurl -plaintext localhost:5000 list
<span class="go">grpc.reflection.v1alpha.ServerReflection</span>
<span class="go">pb.lyft.ratelimit.RateLimitService</span>
</pre></div>
<p>Its ShouldRateLimit method should return <cite>OK</cite> or <cite>OVER_LIMIT</cite> reply.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>grpcurl -d <span class="s1">'{"domain":"envoy"}'</span> -plaintext localhost:5000 pb.lyft.ratelimit.RateLimitService/ShouldRateLimit
<span class="go">{</span>
<span class="go"> "overallCode": "OVER_LIMIT"</span>
<span class="go">}</span>
</pre></div>
<p>Now we shall introduce the ratelimit service to Ambassador.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">travelratelimit</span><span class="w"></span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">getambassador.io/config</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span><span class="w"></span>
<span class="w"> </span><span class="no">---</span><span class="w"></span>
<span class="w"> </span><span class="no">apiVersion: ambassador/v1</span><span class="w"></span>
<span class="w"> </span><span class="no">kind: RateLimitService</span><span class="w"></span>
<span class="w"> </span><span class="no">name: gate_ratelimit</span><span class="w"></span>
<span class="w"> </span><span class="no">service: travelratelimit</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ratelimit</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NodePort</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
<span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">5000</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f k8s-ambassador/ratelimit/service-rate.yml
</pre></div>
<p>Finally, I am going to add <cite>labels</cite> to attach rate limiting descriptors as shown in
<a class="reference external" href="https://www.getambassador.io/user-guide/rate-limiting-tutorial/">Rate Limiting tutorial</a>.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hotel</span><span class="w"></span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">getambassador.io/config</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span><span class="w"></span>
<span class="w"> </span><span class="no">---</span><span class="w"></span>
<span class="w"> </span><span class="no">apiVersion: ambassador/v1</span><span class="w"></span>
<span class="w"> </span><span class="no">kind: Mapping</span><span class="w"></span>
<span class="w"> </span><span class="no">name: hotel_mapping</span><span class="w"></span>
<span class="w"> </span><span class="no">prefix: /v1/hotels</span><span class="w"></span>
<span class="w"> </span><span class="no">rewrite: ""</span><span class="w"></span>
<span class="w"> </span><span class="no">service: hotel</span><span class="w"></span>
<span class="w"> </span><span class="no">labels:</span><span class="w"></span>
<span class="w"> </span><span class="no">ambassador:</span><span class="w"></span>
<span class="w"> </span><span class="no">- request_label_group:</span><span class="w"></span>
<span class="w"> </span><span class="no">- x-ambassador-test-allow:</span><span class="w"></span>
<span class="w"> </span><span class="no">header: "x-ambassador-test-allow"</span><span class="w"></span>
<span class="w"> </span><span class="no">omit_if_not_present: true</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hotel</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NodePort</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
<span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8000</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f k8s-ambassador/hotel/service-rate.yml
</pre></div>
<p>If a request made to the hotel API has header <cite>X-Ambassador-Test-Allow</cite>,
it should be eligible for rate limiting.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -H <span class="s2">"x-ambassador-test-allow: probably"</span> -i -u bob:bob <span class="nv">$AMBASSADORURL</span>/v1/hotels
<span class="go">HTTP/1.1 200 OK</span>
</pre></div>
<p>Unfortunately I have not been able to make rate limiting work yet.
Here is the corresponding <a class="reference external" href="https://github.com/datawire/ambassador/issues/1144">GitHub issue</a>.</p>
</div>
Traefik as API Gateway2019-03-12T00:00:00+07:002019-03-12T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2019-03-12:/apigate-traefik.html<p><a class="reference external" href="https://docs.microsoft.com/en-us/azure/architecture/microservices/design/gateway">API gateway</a>
acts as a reverse proxy, routing API requests from clients to services.
Usually it also performs authentication and rate limiting, so the services behind the gate don't have to.
In this short tutorial we'll see how to achieve that with <a class="reference external" href="https://traefik.io/">Traefik</a> reverse-proxy.</p>
<p>The demo is based on a β¦</p><p><a class="reference external" href="https://docs.microsoft.com/en-us/azure/architecture/microservices/design/gateway">API gateway</a>
acts as a reverse proxy, routing API requests from clients to services.
Usually it also performs authentication and rate limiting, so the services behind the gate don't have to.
In this short tutorial we'll see how to achieve that with <a class="reference external" href="https://traefik.io/">Traefik</a> reverse-proxy.</p>
<p>The demo is based on a dummy <a class="reference external" href="https://traveling.docs.apiary.io/">Traveling project</a>
where we have services to rent a car and book a hotel.</p>
<p>Firstly, we shall install Traefik on <a class="reference external" href="http://kubernetes.io/docs/getting-started-guides/minikube/">Minikube</a>
as an <a class="reference external" href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress Controller</a>
by following <a class="reference external" href="https://docs.traefik.io/user-guide/kubernetes/">Traefik user guide</a>.
Ingress Controller is a Pod that's responsible for fulfilling the Ingress:
it watches the <cite>/ingresses</cite> Kubernetes endpoint for updates to satisfy requests for Ingresses.
Ingress exposes HTTP routes from outside the cluster to Services within the cluster.
HTTP traffic routing is controlled by rules defined on the Ingress resource.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>minikube start
<span class="go">Starting local Kubernetes v1.13.2 cluster...</span>
<span class="gp">$ </span>kubectl apply -f https://raw.githubusercontent.com/containous/traefik/master/examples/k8s/traefik-rbac.yaml
<span class="gp">$ </span>kubectl apply -f https://raw.githubusercontent.com/containous/traefik/master/examples/k8s/traefik-deployment.yaml
</pre></div>
<p>Traefik ingress service (port 80, NodePort 32518) is the entry point of our API gateway.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>minikube service list
<span class="go">|-------------|-------------------------|--------------------------------|</span>
<span class="go">| NAMESPACE | NAME | URL |</span>
<span class="go">|-------------|-------------------------|--------------------------------|</span>
<span class="go">| default | kubernetes | No node port |</span>
<span class="go">| kube-system | kube-dns | No node port |</span>
<span class="go">| kube-system | traefik-ingress-service | http://192.168.99.100:32518 |</span>
<span class="go">| | | http://192.168.99.100:31735 |</span>
<span class="go">|-------------|-------------------------|--------------------------------|</span>
<span class="gp">$ </span><span class="nv">TRAEFIKURL</span><span class="o">=</span>http://192.168.99.100:32518
<span class="gp">$ </span>curl <span class="nv">$TRAEFIKURL</span>
<span class="go">404 page not found</span>
</pre></div>
<p>Note, there is also Traefik admin dashboard (port 8080) running at <cite>http://192.168.99.100:31735</cite>.</p>
<img alt="" src="https://marselester.com/images/traefik-providers.png" />
<div class="section" id="travel-apps">
<h2>Travel Apps</h2>
<p>Let's deploy the travel apps on Kubernetes. For simplicity, you can use the images uploaded to Docker Hub.
If you prefer to build Docker images yourself, make sure they are accessible from Minikube:
configure Docker client to communicate with the Minikube Docker daemon.
Note, you can restore your local Docker environment with <cite>eval $(minikube docker-env --unset)</cite> later on.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>git clone https://github.com/marselester/apigate.git
<span class="gp">$ </span><span class="nb">cd</span> ./apigate/
<span class="gp">$ </span><span class="nb">eval</span> <span class="k">$(</span>minikube docker-env<span class="k">)</span>
<span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-hotel:v1.0.0 --file<span class="o">=</span>docker/hotel.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-traefik/hotel/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-traefik/hotel/service.yml
</pre></div>
<p>The hotel app should be running:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl get pods -l <span class="nv">app</span><span class="o">=</span>hotel
<span class="go">hotel-api-8d7d59b69-bcsvh</span>
<span class="gp">$ </span>kubectl port-forward hotel-api-8d7d59b69-bcsvh <span class="m">8000</span>
<span class="gp">$ </span>curl localhost:8000/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">{</span>
<span class="go"> "id": "7b4fc183-ee67-494d-9715-3510c6d8f2ef",</span>
<span class="go"> "hotel_id": "046d471d-70c7-4595-80cc-266d3e6e07fa",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
<p>Now, it's time to put the <cite>hotel</cite> Service behind the gateway by creating an Ingress rule.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f - <span class="o"><<<</span><span class="err">'</span>
<span class="go">apiVersion: extensions/v1beta1</span>
<span class="go">kind: Ingress</span>
<span class="go">metadata:</span>
<span class="go"> name: hotel-ingress</span>
<span class="go">spec:</span>
<span class="go"> rules:</span>
<span class="go"> - http:</span>
<span class="go"> paths:</span>
<span class="go"> - path: /v1/hotels</span>
<span class="go"> backend:</span>
<span class="go"> serviceName: hotel</span>
<span class="go"> servicePort: 80</span>
<span class="go">'</span>
</pre></div>
<p>Hotel booking requests should be routed to the hotel app:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl <span class="nv">$TRAEFIKURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">{</span>
<span class="go"> "id": "7b4fc183-ee67-494d-9715-3510c6d8f2ef",</span>
<span class="go"> "hotel_id": "046d471d-70c7-4595-80cc-266d3e6e07fa",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
<p>The car rental app is deployed similarly.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-car:v1.0.0 --file<span class="o">=</span>docker/car.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-traefik/car/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-traefik/car/service.yml
<span class="gp">$ </span>kubectl apply -f - <span class="o"><<<</span><span class="err">'</span>
<span class="go">apiVersion: extensions/v1beta1</span>
<span class="go">kind: Ingress</span>
<span class="go">metadata:</span>
<span class="go"> name: car-ingress</span>
<span class="go">spec:</span>
<span class="go"> rules:</span>
<span class="go"> - http:</span>
<span class="go"> paths:</span>
<span class="go"> - path: /v1/cars</span>
<span class="go"> backend:</span>
<span class="go"> serviceName: car</span>
<span class="go"> servicePort: 80</span>
<span class="go">'</span>
</pre></div>
<p>You can see that the car booking is available at the gateway.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl <span class="nv">$TRAEFIKURL</span>/v1/cars/bookings/9e0d65f5-9de2-4428-9bee-1f3967f05129
<span class="go">{</span>
<span class="go"> "id": "9e0d65f5-9de2-4428-9bee-1f3967f05129",</span>
<span class="go"> "car_id": "cfb6f7a5-4591-4f5c-8b17-9a1b10f98ada",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="authentication-service">
<h2>Authentication Service</h2>
<p>API requests must be authenticated before reaching the Travel apps.
This work is delegated to <cite>travelauth</cite> Service that performs HTTP Basic authentication
and returns a username in <cite>X-Travel-User</cite> header if credentials matched.
We shall start from deploying it on the cluster.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker build --tag<span class="o">=</span>marselester/travel-auth:v1.0.0 --file<span class="o">=</span>docker/auth.Dockerfile .
<span class="gp">$ </span>kubectl apply -f k8s-traefik/auth/deployment.yml
<span class="gp">$ </span>kubectl apply -f k8s-traefik/auth/service.yml
</pre></div>
<p>If the auth server is properly deployed, it will prompt for username/password.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl get pods -l <span class="nv">app</span><span class="o">=</span>auth
<span class="go">auth-api-556685f658-h9qb4</span>
<span class="gp">$ </span>kubectl port-forward auth-api-556685f658-h9qb4 <span class="m">8000</span>
<span class="gp">$ </span>curl -i -u bob:bob localhost:8000/v1/hotels
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">X-Travel-User: bob</span>
</pre></div>
<p>Let's tell Traefik to
<a class="reference external" href="https://docs.traefik.io/configuration/entrypoints/#forward-authentication">forward all requests to the auth server</a>
and copy <cite>X-Travel-User</cite> response header from the auth server to the routed request.</p>
<div class="highlight"><pre><span></span><span class="k">[entryPoints]</span><span class="w"></span>
<span class="w"> </span><span class="k">[entryPoints.http]</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">":80"</span><span class="w"></span>
<span class="w"> </span><span class="k">[entryPoints.http.auth.forward]</span><span class="w"></span>
<span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"http://travelauth.default/"</span><span class="w"></span>
<span class="w"> </span><span class="n">authResponseHeaders</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"X-Travel-User"</span><span class="p">]</span><span class="w"></span>
</pre></div>
<p>Create a ConfigMap entry for the Traefik config file and
mount <cite>traefik-conf</cite> ConfigMap volume to <cite>traefik-ingress-controller</cite> Pod.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create configmap traefik-conf --from-file<span class="o">=</span>traefik.toml<span class="o">=</span>k8s-traefik/traefik/traefik.toml --namespace<span class="o">=</span>kube-system
<span class="gp">$ </span>kubectl apply -f k8s-traefik/traefik/deployment.yml
</pre></div>
<p>The updated Traefik deployment is now enforces authentication on API gateway.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -i <span class="nv">$TRAEFIKURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">HTTP/1.1 401 Unauthorized</span>
<span class="gp">$ </span>curl -i -u bob:bob <span class="nv">$TRAEFIKURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">{</span>
<span class="go"> "id": "7b4fc183-ee67-494d-9715-3510c6d8f2ef",</span>
<span class="go"> "hotel_id": "046d471d-70c7-4595-80cc-266d3e6e07fa",</span>
<span class="go"> "status": "confirmed"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="rate-limiting">
<h2>Rate Limiting</h2>
<p>Request rate limiting is configured via Ingress <a class="reference external" href="https://docs.traefik.io/configuration/backends/kubernetes/#annotations">annotations</a>.
For example, we allow to send only 2 hotel requests per 15 seconds for each logged in customer.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">extensions/v1beta1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Ingress</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hotel-ingress</span><span class="w"></span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">traefik.ingress.kubernetes.io/rate-limit</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span><span class="w"></span>
<span class="w"> </span><span class="no">extractorfunc: "request.header.X-Travel-User"</span><span class="w"></span>
<span class="w"> </span><span class="no">rateset:</span><span class="w"></span>
<span class="w"> </span><span class="no"># Allow 2 requests every 15 seconds.</span><span class="w"></span>
<span class="w"> </span><span class="no">modest:</span><span class="w"></span>
<span class="w"> </span><span class="no">period: 15s</span><span class="w"></span>
<span class="w"> </span><span class="no">average: 2</span><span class="w"></span>
<span class="w"> </span><span class="no">burst: 2</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">http</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/v1/hotels</span><span class="w"></span>
<span class="w"> </span><span class="nt">backend</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">serviceName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hotel</span><span class="w"></span>
<span class="w"> </span><span class="nt">servicePort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
</pre></div>
<p>Let's apply these limits onto the hotel app.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f k8s-traefik/hotel/ingress.yml
<span class="gp">$ </span><span class="k">for</span> i <span class="k">in</span> <span class="o">{</span><span class="m">1</span>..3<span class="o">}</span><span class="p">;</span> <span class="k">do</span> curl -i -s -u bob:bob <span class="nv">$TRAEFIKURL</span>/v1/hotels/bookings/7b4fc183-ee67-494d-9715-3510c6d8f2ef <span class="p">|</span> head -n <span class="m">1</span><span class="p">;</span> <span class="k">done</span>
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">HTTP/1.1 429 Too Many Requests</span>
</pre></div>
<p>As you can see the third request was throttled.</p>
<p>Although it's convenient enough to apply rate limiting using annotations,
I would've preferred rate limiting to be decoupled from Traefik similar to authentication service.</p>
</div>
How to Structure Go Projects2018-09-28T00:00:00+07:002018-09-28T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2018-09-28:/how-to-structure-go-projects.html<p>I came to Go from Django where the framework defines project layout, thus I wanted to know
how to structure my Go applications. After reading documentation and building a few Django projects,
you get a clear mental picture, as most of the questions are already answered.
That helps to keep β¦</p><p>I came to Go from Django where the framework defines project layout, thus I wanted to know
how to structure my Go applications. After reading documentation and building a few Django projects,
you get a clear mental picture, as most of the questions are already answered.
That helps to keep the projects within a company consistent, so developers don't have to struggle
when they land on a new codebase. I was looking for a similar framework approach in Go,
but none of them felt right to me. For some reason the same concepts do not resonate with Go.
I didn't have better ideas, so I watched a few videos from conferences.
Here are the talks I found most helpful and I use them as guidelines:</p>
<ul class="simple">
<li>Ben Johnson's <a class="reference external" href="https://www.youtube.com/watch?v=LMSbsW1Xpwg">Structuring Applications for Growth</a></li>
<li>Brian Ketelsen's <a class="reference external" href="https://www.youtube.com/watch?v=MzTcsI6tn-0">Write Code Like the Go Team</a>,
<a class="reference external" href="https://talks.bjk.fyi/gcru18-best.html#/">slides</a></li>
<li>Kat Zien's <a class="reference external" href="https://www.youtube.com/watch?v=oL6JBUk6tj0">How Do You Structure Your Go Apps</a></li>
<li>Peter Bourgon's <a class="reference external" href="https://www.youtube.com/watch?v=LHe1Cb_Ud_M">Ways To Do Things</a>.
I tried his ideas in <a class="reference external" href="https://github.com/marselester/rascaldb/blob/master/rascaldb.go">RascalDB π</a>
to serialize concurrent DB actions.</li>
</ul>
<p>I encourage you to watch those talks and go over code examples they provided.</p>
<p>The following is my takeaway which could be completely misleading and far from the original ideas
(just in case, you don't have to apply all of them together).
If you feel something doesn't bring enough value in your case, just skip it, trust your intuition.
Development requires many iterations to bounce ideas until the result becomes good enough.</p>
<div class="section" id="know-your-domain">
<h2>Know Your Domain</h2>
<p>I like to start a new project on A4 paper and dump everything I know in free form
(what the problem are you facing, can you solve it without writing code,
what are possible solutions (pros/cons), terminology, system requirements, API endpoints, sketches,
doodles, whatever gets your train of thoughts moving). Paper helps to get away from constraints of computer,
I can write and draw without any programs which would otherwise have taken time and stolen my brain cycles.
Paper helps me to stay focused β nothing blinks, pops up or rings, there is no urge to multitask
(check email/Slack, switch between editor and console as if there is something new).</p>
<p>Once you wrote everything down, you might realize there is not enough information to make progress.
So you can reach out to stake holders and get more insights using your notes.
Don't forget to write new info as well. It might take a few iterations when you finally establish
common terminology, refine the actual project's goal, and cut unnecessary features/requirements.</p>
<p>Now you have a mental picture of the project with clear deliverables.
Based on that I like to spend time to come up with a concise project/repository name which
summarizes nicely the project's goal and spirit (thesaurus comes to the rescue).
As a next step I usually create README file where insights from the paper are documented.
This time you want everyone in the world to understand what you learned about the project.
This helps you to iterate once more as you're documenting in README, and the end result
can be shared with coworkers (they might not understand your handwriting and doodles).
If they have questions, that means you have a room for improvement, since other people
don't have a context you obtained. Take your time and write it down, this will ensure that
you will be able to get help with the project much easier (when you're on vacation or
have to hand over the code). With some practice you'll anticipate questions and
provide answers in README, so it won't take as much time from your coworkers next time.
Keeping in mind that you are ultimately solving problems for your customers and
designing the system and writing code for your coworkers (not just for yourself)
helps to rigorously document as you go. Or else prepare to answer the same questions
over and over again. In order to get better at documenting, I recommend reading
<a class="reference external" href="http://www.writethedocs.org/guide/writing/beginners-guide-to-docs/">A beginnerβs guide to writing documentation</a>
and <a class="reference external" href="http://www.writethedocs.org/guide/">Documentation Guide</a>.</p>
<p>Let's say you're building an API client, then it makes sense to start from
a usage example in the README, since most of the documentation work is done by API provider.
How will people download your library? How will they import it, does the path look concise and clear?
How will your end user configure a client? What use cases your users might have (custom API domain,
timeouts, logging, instrumentation)? Think of a few workflows and see how would you handle errors.
Best case scenario is using your own library to reveal pain points.
Check out <a class="reference external" href="https://github.com/marselester/bitgo-v2">one of my attempts</a> to find a decent layout of
an API library. The library has a configurable client (you can instantiate many of them, e.g., one per coin)
which allows to use a custom logger, http.Client, etc. It also has embedded services β things that know
how to speak with particular API endpoints to operate API resources.</p>
</div>
<div class="section" id="getting-to-the-point">
<h2>Getting to the Point</h2>
<p>As you can see from the talks the main theme in Go application structure is heavily
influenced by Domain-Driven Design (DDD). Using the research done on your project,
you shall write down entities (as Go structs) of the domain model and services (as Go interfaces)
which perform operations over those entities.</p>
<p>I should mention that for simplicity's sake in my projects I decided to combine DDD "service" and
"repository" concepts under "service" term.</p>
<p>Let's proceed with listing minimal set of operations over the key entities to keep the scope small.
It helps to imagine that there are different service implementations,
an entity can be stored anywhere, for example, in Postgres, Kafka, memory, Redis,
JSON file, remote API. But keep in mind semantics that storage of choice provides:
getting a list of entities from Kafka and Postgres are hard to abstract (streaming vs quering).
Moreover, if you do so, you might create unnecessary constraints for yourself.
In reality, it is very unlikely that a project will change storages often,
since you've chosen them for a reason (the choice of the storage is influenced
by a problem you're solving, its constraints and guarantees you need).
Most likely you've already chosen the storages in advance, so make sure you
don't fight with interfaces you defined in your domain model. If you know your services
need to share the same SQL transaction, embrace it in interfaces, for instance:</p>
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">PaymentService</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">CreatePayment</span><span class="p">(</span><span class="nx">ctx</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span><span class="w"> </span><span class="nx">tx</span><span class="w"> </span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">Tx</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">Payment</span><span class="p">)</span><span class="w"> </span><span class="kt">error</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Do you put all operations (add/list) into one service interface? I don't know.
You have to decide. If you add an entity to Postgres, you expect to list them
from Postgres as well. From this perspective "add" and "list" operations should be
under the same interface. If in-memory storage can't implement the service interface,
perhaps you don't have to enforce operation under the same interface,
then probably it's ok to split it.</p>
<p>Ben Johnson defines entities in a single package where domain model is described β the package
shouldnβt have third party dependencies. Whereas Kat Zien in her demo created a package per service:
"adder" package that adds beers and reviews, "lister" service lists beers and reviews.
Each package defines its own beer and review structs.</p>
<p>In my projects I isolate services implementation in a single package.
For example, if I stored beer reviews in Kafka, I would have a kafka package which exposes
a client and beer/review services embedded in it. The same applies to postgres package β two services use
the same db connection pool and beer/review entities might appear in the same db transaction.</p>
<p>During coding I combine the service implementations into a program (some server or
a ctl tool) kept in cmd directory. That helps me to validate design ideas and notice any awkward
component integrations. Similar to service implementations, try to think where the input and
output could be coming from/to: standard input/output, http, rpc, db.</p>
</div>
<div class="section" id="an-example">
<h2>An Example</h2>
<p>Now let's have a look at <a class="reference external" href="https://github.com/marselester/distributed-payment">distributed payment</a>
demo project where I explored an idea of payment transaction
without atomic commit across 3 Kafka partitions.</p>
<p>The domain model is defined in the repository root (note, you can place your packages
in "internal" directory, so you don't mix them up with unrelated files):</p>
<ul class="simple">
<li><cite>wallet.go</cite> has <cite>Transfer</cite>, <cite>Payment</cite> entities, and services <cite>TransferService</cite>, <cite>PaymentService</cite>
which can create and list the entities. Since the project is based on Kafka,
the interfaces reflect that (<cite>partition</cite>, <cite>offset</cite> params). The services accept <cite>context.Context</cite>
as a first argument, because we should be able to tell implementations to cancel operation.
<a class="reference external" href="https://github.com/opentracing/opentracing-go">OpenTracing</a> can leverage context as well.
Pay attention to a pointer/value semantics (share or not) in the service interfaces.
Since an entity in DDD terms has a unique identity, a pointer semantics was used,
hence <cite>*Transfer</cite> is passed to <cite>CreateTransfer()</cite> and returned from <cite>FromOffset()</cite>. Have a look at
<a class="reference external" href="https://www.ardanlabs.com/blog/2017/06/design-philosophy-on-data-and-semantics.html">Design Philosophy On Data And Semantics</a>
for more insights.</li>
<li><cite>error.go</cite> contains errors which are relevant to the whole domain model,
<a class="reference external" href="https://middlemost.com/failure-is-your-domain/">Failure is your Domain</a>.
On implementation level there could be their own specific errors, for example, HTTP API errors in
<cite>rest/error.go</cite> define JSON and validation errors.</li>
<li><cite>log.go</cite> borrows <cite>Logger</cite> interface from Go kit. Since logging is an integral part of the system,
placing it nearby the domain model seems justified. There is
<a class="reference external" href="https://github.com/go-commons/commons/issues/1">Standard logger interface</a> discussion
where the consensus is to emit events instead of logging in the library.
Best practices and examples of how to emit events is still an
<a class="reference external" href="https://github.com/go-commons/event/issues/1">open topic</a> at the time of writing.</li>
</ul>
<p>Implementations of the services defined in <cite>wallet.go</cite> are isolated in packages
by their dependency name, for example, kafka, rest, mock, rocksdb.</p>
<p>Package kafka implements wallet services and provides the Client access to them.
There were two design options: embed the services to the Client struct or
inject a service into each other. The example below would allow to have
a swappable <cite>PaymentService</cite> ("pg" refers to a Postgres implementation):</p>
<div class="highlight"><pre><span></span><span class="nx">kafka</span><span class="p">.</span><span class="nx">TransferService</span><span class="p">.</span><span class="nx">PaymentService</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">pg</span><span class="p">.</span><span class="nx">NewPaymentService</span><span class="p">()</span><span class="w"></span>
</pre></div>
<p>On the other hand, grouping services in the Client would let services maintain DB transactions
by sharing the same <cite>*sql.DB</cite>. Here is <cite>pg.Client</cite> example:</p>
<div class="highlight"><pre><span></span><span class="c1">// Client represents a client to the underlying PostgreSQL data store.</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">Client</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Transfer</span><span class="w"> </span><span class="o">*</span><span class="nx">TransferService</span><span class="w"></span>
<span class="w"> </span><span class="nx">Payment</span><span class="w"> </span><span class="o">*</span><span class="nx">PaymentService</span><span class="w"></span>
<span class="w"> </span><span class="nx">logger</span><span class="w"> </span><span class="nx">wallet</span><span class="p">.</span><span class="nx">Logger</span><span class="w"></span>
<span class="w"> </span><span class="nx">db</span><span class="w"> </span><span class="o">*</span><span class="nx">sql</span><span class="p">.</span><span class="nx">DB</span><span class="w"></span>
<span class="w"> </span><span class="nx">transferQ</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="w"></span>
<span class="w"> </span><span class="nx">paymentQ</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>In <a class="reference external" href="https://github.com/marselester/distributed-signup/blob/master/pg/user_service.go">distributed-signup</a>
project a Client concept is baked into <cite>UserService</cite>, because it was the only service that
needed access to Postgres.</p>
<div class="highlight"><pre><span></span><span class="c1">// UserService reprensets a service to store signed up users.</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">UserService</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="nx">Config</span><span class="w"></span>
<span class="w"> </span><span class="nx">pool</span><span class="w"> </span><span class="o">*</span><span class="nx">pgx</span><span class="p">.</span><span class="nx">ConnPool</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Package rest is responsible for translating incoming HTTP requests to wallet domain and
then translating results from wallet model back to HTTP responses.
The package doesn't implement <cite>TransferService</cite> per se, it uses one in its Server.
The REST-style API server itself is put together in
<a class="reference external" href="https://github.com/marselester/distributed-payment/blob/master/cmd/transfer-server/main.go">cmd/transfer-server</a>.</p>
<div class="highlight"><pre><span></span><span class="c1">// Server represents an HTTP API handler for wallet services.</span><span class="w"></span>
<span class="c1">// It wraps a TransferService so we can provide different</span><span class="w"></span>
<span class="c1">// implementations, e.g., Kafka or a mock.</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">Server</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">*</span><span class="nx">chi</span><span class="p">.</span><span class="nx">Mux</span><span class="w"></span>
<span class="w"> </span><span class="nx">logger</span><span class="w"> </span><span class="nx">wallet</span><span class="p">.</span><span class="nx">Logger</span><span class="w"></span>
<span class="w"> </span><span class="nx">transferService</span><span class="w"> </span><span class="nx">wallet</span><span class="p">.</span><span class="nx">TransferService</span><span class="w"></span>
<span class="w"> </span><span class="nx">wopts</span><span class="w"> </span><span class="nx">walletOption</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Originally in <a class="reference external" href="https://medium.com/wtf-dial/wtf-dial-http-api-d8800ccd605f">WTF Dial: HTTP API</a>
Ben Johnson explained how to implement API properly and isolate http dependencies in wtf/http package.</p>
<p>Package mock provides mock services to facilitate testing. For example, for most cases
we do not need Kafka implementation of a transfer service to be used in HTTP API testing.</p>
<p>Package rocksdb implements user requests deduplication using RocksDB to
memorise already processed request IDs. Requests deduplication is an integral part of
a distributed system, hence the domain model must embrace it.</p>
<p>Everything is connected in cmd directory. Note, that the domain package is used everywhere.</p>
<ul class="simple">
<li><cite>cmd/transfer-server</cite> is HTTP API server to create money transfers which are stored in Kafka.
It delegates the actual hard work to kafka and rest packages.</li>
<li><cite>cmd/paymentd</cite> program is responsible for creating incoming & outgoing payment pairs based on
money transfer requests stored in Kafka.</li>
<li><cite>cmd/accountantd</cite> is the last program in the pipeline. It sequentially reads Kafka messages
from <cite>wallet.payment</cite> topic, deduplicates messages by request ID, and applies the changes to
the account balances. Deduplication is provided by rocksdb package mentioned above.</li>
</ul>
<p>To wrap up, that's all I managed to recall :) I look forward for more talks on structuring Go applications.</p>
</div>
Forward DogStatsD Metrics to Prometheus2017-06-17T00:00:00+07:002017-06-17T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2017-06-17:/prometheus-via-dogstatsd.html<p><strong>tl;dr:</strong> StatsD doesn't have metric labels, DogStatsD does.</p>
<p>This is a follow up post after
<a class="reference external" href="https://marselester.com/django-prometheus-via-statsd.html">Instrumenting Django with Prometheus and StatsD</a>.</p>
<hr class="docutils" />
<p>You got Prometheus up and running and eager to start instrumenting your Django application.
Don't be hasty and read <a class="reference external" href="https://prometheus.io/docs/practices/instrumentation/">Prometheus Best Practices</a>.</p>
<p>Let's say our application has to β¦</p><p><strong>tl;dr:</strong> StatsD doesn't have metric labels, DogStatsD does.</p>
<p>This is a follow up post after
<a class="reference external" href="https://marselester.com/django-prometheus-via-statsd.html">Instrumenting Django with Prometheus and StatsD</a>.</p>
<hr class="docutils" />
<p>You got Prometheus up and running and eager to start instrumenting your Django application.
Don't be hasty and read <a class="reference external" href="https://prometheus.io/docs/practices/instrumentation/">Prometheus Best Practices</a>.</p>
<p>Let's say our application has to use a Python library to request weather forecast.
Sometimes Weather API server doesn't work as expected and we find HTTP 500 status codes in our application logs.
Out of curiosity we want to know how often that happens.
What if we already have to look for another forecast provider?</p>
<p>First idea could be using a counter "weatherapi_responses_500_total".
From the Prometheus docs:</p>
<blockquote>
When reporting failures, you should generally have some other metric
representing the total number of attempts.
This makes the failure ratio easy to calculate.</blockquote>
<p>Alright, then we need to count "HTTP 200 OK" API responses as well "weatherapi_responses_200_total".
But what about other HTTP status codes? Shall we create metrics for each of them?</p>
<blockquote>
When you have multiple metrics that you want to add/average/sum,
they should usually be one metric with labels rather than multiple metrics.</blockquote>
<p>As this is exactly our case, we should use Prometheus labels. Therefore our
metric name should be "weatherapi_responses_total" with a "code" label for the HTTP response code.</p>
<p>Depending on a scenario, we might have multiple API clients in our application.
For example, we may generalize metric to "api_responses_total" with
the following labels "code=200", "service=weather".</p>
<p>Although we should keep it sane and not overuse labels.</p>
<div class="section" id="statsd-by-datadog">
<h2>StatsD by Datadog</h2>
<p>In <a class="reference external" href="https://marselester.com/django-prometheus-via-statsd.html">the previous post</a>
we used StatsD with statsd_exporter to forward metrics to Prometheus server.
StatsD protocol doesn't have a notion of labels,
but <a class="reference external" href="https://docs.datadoghq.com/guides/dogstatsd/#tags">Datadog's fork</a> has introduced tags.
Moreover statsd_exporter can convert them into Prometheus labels.</p>
<p>Let's take it step by step.
Firstly, we need <a class="reference external" href="https://github.com/DataDog/datadogpy">Datadog Python client</a>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>pip install <span class="nv">datadog</span><span class="o">==</span><span class="m">0</span>.16.0
</pre></div>
<p>Secondly, we should increment "weatherapi.responses.total" counter.
For example:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">weather</span>
<span class="kn">from</span> <span class="nn">datadog</span> <span class="kn">import</span> <span class="n">statsd</span>
<span class="k">def</span> <span class="nf">request_weather_forecast</span><span class="p">(</span><span class="n">location</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">forecast</span> <span class="o">=</span> <span class="n">weather</span><span class="o">.</span><span class="n">Forecast</span><span class="o">.</span><span class="n">retrieve</span><span class="p">(</span><span class="n">location</span><span class="p">)</span>
<span class="k">except</span> <span class="n">weather</span><span class="o">.</span><span class="n">APIError</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="n">statsd</span><span class="o">.</span><span class="n">increment</span><span class="p">(</span><span class="s1">'weatherapi.responses.total'</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span>
<span class="s1">'code:</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">exc</span><span class="o">.</span><span class="n">http_status_code</span><span class="p">),</span>
<span class="p">])</span>
<span class="k">raise</span>
<span class="n">statsd</span><span class="o">.</span><span class="n">increment</span><span class="p">(</span><span class="s1">'weatherapi.responses.total'</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span>
<span class="s1">'code:200'</span><span class="p">,</span>
<span class="p">])</span>
<span class="k">return</span> <span class="n">forecast</span>
</pre></div>
<p>Well, since I made up the weather library, we can just run a Python script:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datadog</span> <span class="kn">import</span> <span class="n">statsd</span>
<span class="n">statsd</span><span class="o">.</span><span class="n">increment</span><span class="p">(</span><span class="s1">'weatherapi.responses.total'</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="p">[</span><span class="s1">'code:200'</span><span class="p">])</span>
</pre></div>
<p>And finally, we should run statsd_exporter with disabled "-statsd.add-suffix" flag
which adds the metric type (counter/gauge/timer) as suffix to the generated Prometheus metric.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>statsd_exporter <span class="se">\</span>
-statsd.listen-address<span class="o">=</span>:8125 <span class="se">\</span>
-statsd.add-suffix<span class="o">=</span><span class="nb">false</span>
</pre></div>
<p>The statsd_exporter should expose our metric at <a class="reference external" href="http://localhost:9102/metrics">http://localhost:9102/metrics</a>.</p>
<div class="highlight"><pre><span></span># HELP weatherapi_responses_total Metric autogenerated by statsd_exporter.
# TYPE weatherapi_responses_total counter
weatherapi_responses_total{code="200"} 1
</pre></div>
<p>I hope this helps. Cheers!</p>
</div>
Instrumenting Django with Prometheus and StatsD2017-06-09T00:00:00+07:002017-06-09T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2017-06-09:/django-prometheus-via-statsd.html<p>If you ever wondered how to monitor your Django application with Prometheus this article is for you.
Quick search on the topic will lead you to <a class="reference external" href="https://github.com/korfuri/django-prometheus/">django-prometheus</a>.
For those who don't want to use it, there is another way to export application metrics
via <a class="reference external" href="https://www.datadoghq.com/blog/statsd/">StatsD</a>.</p>
<p>The idea is to send β¦</p><p>If you ever wondered how to monitor your Django application with Prometheus this article is for you.
Quick search on the topic will lead you to <a class="reference external" href="https://github.com/korfuri/django-prometheus/">django-prometheus</a>.
For those who don't want to use it, there is another way to export application metrics
via <a class="reference external" href="https://www.datadoghq.com/blog/statsd/">StatsD</a>.</p>
<p>The idea is to send metrics from Django by StatsD Python library to StatsD server over UDP.
Here is an example of incrementing "hello.requests.total" metric every time "say_hello" view is run.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">statsd.defaults.django</span> <span class="kn">import</span> <span class="n">statsd</span>
<span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">HttpResponse</span>
<span class="k">def</span> <span class="nf">say_hello</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">statsd</span><span class="o">.</span><span class="n">incr</span><span class="p">(</span><span class="s1">'hello.requests.total'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="s1">'Hello, World!'</span><span class="p">)</span>
</pre></div>
<p>StatsD server aggregates measurements over time and flushes them into monitoring backend.
Here <a class="reference external" href="https://github.com/prometheus/statsd_exporter">statsd_exporter</a> comes into play.
It uses the same UDP protocol as StatsD server and exports StatsD-style metrics as Prometheus metrics,
so "hello.requests.total" becomes "hello_requests_total_counter".</p>
<p>All we need is to configure Django to send metrics to statsd_exporter daemon.
Let's set up "Olympus" Django project to demonstrate StatsD integration or
you can get it from <a class="reference external" href="https://github.com/marselester/django-prometheus-via-statsd">github.com/marselester/django-prometheus-via-statsd</a>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>virtualenv venv
<span class="gp">$ </span><span class="nb">source</span> ./venv/bin/activate
<span class="gp">$ </span>pip install <span class="nv">Django</span><span class="o">==</span><span class="m">1</span>.11.1 <span class="nv">statsd</span><span class="o">==</span><span class="m">3</span>.2.1
<span class="gp">$ </span>django-admin.py startproject olympus .
</pre></div>
<p>These two lines of Django settings <cite>./olympus/settings.py</cite> configure a default StatsD client
that is used in the "say_hello" Django view.</p>
<div class="highlight"><pre><span></span><span class="c1"># statsd_exporter daemon listens to UDP at localhost:8125</span>
<span class="n">STATSD_HOST</span> <span class="o">=</span> <span class="s1">'localhost'</span>
<span class="n">STATSD_PORT</span> <span class="o">=</span> <span class="mi">8125</span>
</pre></div>
<p>Assuming "say_hello" is available at "/hello" URL path, we can start a web server.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>python manage.py runserver
<span class="gp">$ </span>curl http://localhost:8000/hello
<span class="go">Hello, World!</span>
</pre></div>
<p>The Olympus application is ready to emit metrics. To make sure this is the case,
we can use tcpdump to capture UDP packets on loopback interface on port 8125
and print each packet in ASCII.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>tcpdump -i lo0 udp port <span class="m">8125</span> -A
<span class="go">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</span>
<span class="go">listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes</span>
<span class="go">11:19:05.460810 IP localhost.57776 > localhost.8125: UDP, length 24</span>
<span class="go">E..4....@................ .3hello.requests.total:1|c</span>
</pre></div>
<p>As we can see, "hello.requests.total:1|c" is sent every time we hit <cite>http://localhost:8000/hello</cite>.</p>
<div class="section" id="statsd-exporter-1">
<h2>StatsD Exporter</h2>
<p>Since we have StatsD metrics being sent, we can expose them for Prometheus via statsd_exporter.
Let's download the latest version and install it (the binary will be in "$GOPATH/bin/").</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>go get -u github.com/prometheus/statsd_exporter
</pre></div>
<p>Run statsd_exporter so it gets StatsD metrics from our Django application.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>statsd_exporter -statsd.listen-address<span class="o">=</span><span class="s2">"localhost:8125"</span>
</pre></div>
<p>By default it exposes generated Prometheus metrics at <a class="reference external" href="http://localhost:9102/metrics">http://localhost:9102/metrics</a>.
Check out the output, among many metrics there will be our counter from "say_hello" view.</p>
<div class="highlight"><pre><span></span># HELP hello_requests_total_counter Metric autogenerated by statsd_exporter.
# TYPE hello_requests_total_counter counter
hello_requests_total_counter 2
</pre></div>
</div>
<div class="section" id="run-everything-on-kubernetes">
<h2>Run Everything on Kubernetes</h2>
<p>Of course we will deploy everything on Kubernetes.
Minikube will help us here.
Minukube is an easy way to run Kubernetes locally.
When we want to build a Docker image in Minukube (so Kubernetes has an access to it),
we can configure our Docker client to communicate with the Minikube Docker daemon.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>minikube start
<span class="go">Starting local Kubernetes cluster...</span>
<span class="go">Kubectl is now configured to use the cluster.</span>
<span class="gp">$ </span><span class="nb">eval</span> <span class="k">$(</span>minikube docker-env<span class="k">)</span>
</pre></div>
<div class="section" id="django-application">
<h3>Django application</h3>
<p>First, we shall deploy our Django application with statsd_exporter running in the same Kubernetes Pod.
It exposes 8000 (uWSGI) and 9102 (statsd_exporter's generated Prometheus metrics) container ports.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker build -t marselester/olympus:v1.0.0 ./olympus-app/
<span class="gp">$ </span>kubectl apply -f ./kube/olympus-app/deployment.yml
<span class="gp">$ </span>kubectl apply -f ./kube/olympus-app/service.yml
</pre></div>
<p>Though we don't need Nginx in the demo, it's ubiquitous on production servers.
Nginx proxies HTTP requests using uWSGI protocol to "olympus-service" Kubernetes Service
we created above.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create configmap olympus-nginx-conf --from-file<span class="o">=</span>./kube/nginx/olympus.conf
<span class="gp">$ </span>kubectl apply -f ./kube/nginx/deployment.yml
</pre></div>
</div>
<div class="section" id="prometheus">
<h3>Prometheus</h3>
<p>Next is Prometheus server's turn to be deployed. It is configured to scrape
own metrics from 9102 port and metrics from Pods that have "app: olympus" label.</p>
<p>Prometheus server listens on port 9090. On production servers you'll likely run
it behind Nginx with basic authentication and make it accessible via VPN only.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create configmap prometheus-server-conf --from-file<span class="o">=</span>./kube/prometheus/prometheus.yml
<span class="gp">$ </span>kubectl apply -f ./kube/prometheus/deployment.yml
</pre></div>
<p>But in our case we'll use Kubernetes port forwarding to test whether Django
metrics show up in Prometheus dashboard.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl port-forward nginx-deployment-3580857522-tn332 <span class="m">8080</span>:80
<span class="gp">$ </span>kubectl port-forward prometheus-deployment-2456821496-8zdg8 <span class="m">9090</span>
<span class="gp">$ </span>curl http://localhost:8080/hello
<span class="go">Hello, World!</span>
</pre></div>
<p>"hello_requests_total_counter" should be searchable at the expression browser
<a class="reference external" href="http://localhost:9090/graph">http://localhost:9090/graph</a>.</p>
</div>
<div class="section" id="prometheus-helm-chart">
<h3>Prometheus Helm Chart</h3>
<p>There are other ways to install Prometheus on Kubernetes:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/kubernetes/charts/tree/master/stable/prometheus">Helm</a> (recommended Kubernetes >= 1.4.1)</li>
<li><a class="reference external" href="https://github.com/coreos/prometheus-operator">Prometheus Operator</a> (requires Kubernetes >= 1.5.0)</li>
</ul>
<p>Prometheus Operator and Helm look awesome, though I have not played with Operator yet.
Here is how you can set up Prometheus via Helm (package manager for Kubernetes).
You will need the Helm client</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>brew install kubernetes-helm
</pre></div>
<p>and Helm server (Tiller). The following command installs it into the Kubernetes cluster.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>helm init
</pre></div>
<p>Now you can install Prometheus Helm package (chart).</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>helm repo update
<span class="gp">$ </span>helm install --name team-ops stable/prometheus
</pre></div>
<p>Nice, we have full-blown Prometheus "team-ops" chart release running
with alert manager and node exporter.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>helm list
<span class="go">NAME REVISION UPDATED STATUS CHART NAMESPACE</span>
<span class="go">team-ops 1 Thu Jun 8 21:40:29 2017 DEPLOYED prometheus-3.0.2 default</span>
</pre></div>
<p>Let's add one more Prometheus to the cluster (call it "team-dev"),
but this time we want only Prometheus server with a custom prometheus.yml config.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>helm install --name team-dev <span class="se">\</span>
--set alertmanager.enabled<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
--set kubeStateMetrics.enabled<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
--set nodeExporter.enabled<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
stable/prometheus
</pre></div>
<p>The config is stored in "team-dev-prometheus-server" ConfigMap.
Let's overwrite it with our prometheus.yml.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create configmap team-dev-prometheus-server <span class="se">\</span>
--from-file<span class="o">=</span>./kube/prometheus/prometheus.yml <span class="se">\</span>
-o yaml <span class="se">\</span>
--dry-run <span class="p">|</span> kubectl replace -f -
</pre></div>
<p>To see whether "team-dev" Prometheus started, we can run a port forwarding:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl port-forward team-dev-prometheus-server-4131857549-c98j0 <span class="m">9091</span>:9090
</pre></div>
<p>Prometheus "team-dev" release is accessible at <a class="reference external" href="http://localhost:9091/graph">http://localhost:9091/graph</a>.</p>
<p>I hope this helps. Cheers!</p>
</div>
</div>
Minukube & Amazon EC2 Container Registry2016-12-05T00:00:00+07:002016-12-05T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2016-12-05:/minukube-aws-ecr.html<p><a class="reference external" href="http://kubernetes.io/docs/getting-started-guides/minikube/">Minukube</a> is an easy way to run Kubernetes locally.
When we want to build a Docker image in Minukube (so Kubernetes has an access to it),
we can configure our Docker client to communicate with the Minikube Docker daemon.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>minikube start
<span class="go">Starting local Kubernetes cluster...</span>
<span class="go">Kubectl is now configured to β¦</span></pre></div><p><a class="reference external" href="http://kubernetes.io/docs/getting-started-guides/minikube/">Minukube</a> is an easy way to run Kubernetes locally.
When we want to build a Docker image in Minukube (so Kubernetes has an access to it),
we can configure our Docker client to communicate with the Minikube Docker daemon.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>minikube start
<span class="go">Starting local Kubernetes cluster...</span>
<span class="go">Kubectl is now configured to use the cluster.</span>
<span class="gp">$ </span><span class="nb">eval</span> <span class="k">$(</span>minikube docker-env<span class="k">)</span>
<span class="gp">$ </span>docker build -t fizz/bazz:latest .
</pre></div>
<p>What if we need to create a Kubernetes Deployment which pulls a Docker image from AWS ECR?</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create -f - <span class="o"><<<</span><span class="err">'</span>
<span class="go">apiVersion: extensions/v1beta1</span>
<span class="go">kind: Deployment</span>
<span class="go">metadata:</span>
<span class="go"> name: my-deployment</span>
<span class="go">spec:</span>
<span class="go"> replicas: 1</span>
<span class="go"> template:</span>
<span class="go"> metadata:</span>
<span class="go"> labels:</span>
<span class="go"> app: my-app-name</span>
<span class="go"> spec:</span>
<span class="go"> containers:</span>
<span class="go"> - name: my-container-name</span>
<span class="go"> image: 4338991606.dkr.ecr.eu-west-1.amazonaws.com/fizz/bazz:latest</span>
<span class="go">'</span>
</pre></div>
<p>Kubernetes will fail to get the image due to authentication error (no credentials).
One of the solutions is to use <a class="reference external" href="http://kubernetes.io/docs/user-guide/images/#specifying-imagepullsecrets-on-a-pod">imagePullSecrets</a>.
The following command creates a secret for use with a Docker registry.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create secret docker-registry my-docker-credentials <span class="se">\</span>
--docker-server<span class="o">=</span>DOCKER_REGISTRY_SERVER <span class="se">\</span>
--docker-username<span class="o">=</span>DOCKER_USER <span class="se">\</span>
--docker-password<span class="o">=</span>DOCKER_PASSWORD <span class="se">\</span>
--docker-email<span class="o">=</span>DOCKER_EMAIL
</pre></div>
<p>Firstly, let's obtain <strong>DOCKER_USER</strong> and <strong>DOCKER_PASSWORD</strong>.
For that, we'll need <strong>awscli</strong> Python library and environment variables
<strong>AWS_ACCESS_KEY_ID</strong> and <strong>AWS_SECRET_ACCESS_KEY</strong> set up.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>pip install awscli
<span class="gp">$ </span><span class="nb">export</span> <span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s1">'AKIAI44QH8DHBEXAMPLE'</span>
<span class="gp">$ </span><span class="nb">export</span> <span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s1">'je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY'</span>
</pre></div>
<p>Log in to Amazon ECR registry.
This aws command will display <strong>$ docker login</strong> command. From its output we conclude that
<strong>DOCKER_USER</strong> is <strong>AWS</strong>, <strong>DOCKER_PASSWORD</strong> is <strong>SomeVeryLongToken</strong> and <strong>DOCKER_REGISTRY_SERVER</strong>
is <strong>https://4338991606.dkr.ecr.eu-west-1.amazonaws.com</strong> respectively.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>aws ecr get-login --region eu-west-1
<span class="go">docker login -u AWS -p SomeVeryLongToken -e none https://4338991606.dkr.ecr.eu-west-1.amazonaws.com</span>
</pre></div>
<p>Now it is time to create a docker-registry secret and update the Deployment manifest.
Note that we've added <strong>imagePullSecrets</strong> which references our private registry.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f - <span class="o"><<<</span><span class="err">'</span>
<span class="go">apiVersion: extensions/v1beta1</span>
<span class="go">kind: Deployment</span>
<span class="go">metadata:</span>
<span class="go"> name: my-deployment</span>
<span class="go">spec:</span>
<span class="go"> replicas: 1</span>
<span class="go"> template:</span>
<span class="go"> metadata:</span>
<span class="go"> labels:</span>
<span class="go"> app: my-app-name</span>
<span class="go"> spec:</span>
<span class="go"> containers:</span>
<span class="go"> - name: my-container-name</span>
<span class="go"> image: 4338991606.dkr.ecr.eu-west-1.amazonaws.com/fizz/bazz:latest</span>
<span class="go"> imagePullSecrets:</span>
<span class="go"> - name: my-docker-credentials</span>
<span class="go">'</span>
</pre></div>
<p>Kubernetes should be able to pull the image from Amazon Container Registry.</p>
Prometheus on Kubernetes2016-11-13T00:00:00+07:002016-11-13T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2016-11-13:/prometheus-on-kubernetes.html<p><a class="reference external" href="https://prometheus.io/">Prometheus</a> is a monitoring toolkit.
Let's set it up on Kubernetes and test how it works by scraping HTTP request metrics
from <a class="reference external" href="https://github.com/marselester/prometheus-on-kubernetes">hello web application</a>
which also runs in the same cluster.</p>
<p>First of all, we need Kubernetes cluster running. It's easy to bootstrap one via <a class="reference external" href="https://console.cloud.google.com">Google Container Engine</a>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>gcloud β¦</pre></div><p><a class="reference external" href="https://prometheus.io/">Prometheus</a> is a monitoring toolkit.
Let's set it up on Kubernetes and test how it works by scraping HTTP request metrics
from <a class="reference external" href="https://github.com/marselester/prometheus-on-kubernetes">hello web application</a>
which also runs in the same cluster.</p>
<p>First of all, we need Kubernetes cluster running. It's easy to bootstrap one via <a class="reference external" href="https://console.cloud.google.com">Google Container Engine</a>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>gcloud container clusters create my-k8
</pre></div>
<p>The command above might complain that zone is not currently set.
Choose the one you like and try to create a cluster again.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>gcloud compute zones list
<span class="gp">$ </span>gcloud config <span class="nb">set</span> compute/zone asia-east1-b
</pre></div>
<p>You should now have a Kubernetes cluster.
If there are any issues, check out <a class="reference external" href="https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615">Udacity course</a> which helped me run Kubernetes in GKE.</p>
<div class="section" id="first-start">
<h2>First Start</h2>
<p>Prometheus is available as Docker image and can be run locally quickly.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>docker run -p <span class="m">9090</span>:9090 prom/prometheus:v1.2.1
</pre></div>
<p>When a container is started, the Prometheus expression browser should be accessible on <cite>http://localhost:9090</cite>.
Now let's achieve the same results with Kubernetes.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl run prometheus-deployment --image<span class="o">=</span>prom/prometheus:v1.2.1 --port<span class="o">=</span><span class="m">9090</span>
</pre></div>
<p>The command above created <a class="reference external" href="http://kubernetes.io/docs/user-guide/deployments/">Kubernetes Deployment</a>
that runs <cite>prometheus</cite> Docker image.</p>
<p>By default deployed applications are visible only inside the Kubernetes cluster.
To see whether Prometheus started up, we can run a proxy between terminal and Kubernetes cluster.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl proxy
<span class="go">Starting to serve on 127.0.0.1:8001</span>
</pre></div>
<p>In another cloud shell, we can call Kubernetes API.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl get pods
<span class="go">NAME READY STATUS RESTARTS AGE</span>
<span class="go">prometheus-deployment-2234044252-of29y 1/1 Running 0 8m</span>
<span class="gp">$ </span>curl http://127.0.0.1:8001/api/v1/proxy/namespaces/default/pods/prometheus-deployment-2234044252-of29y:9090/metrics
</pre></div>
<p>Another option is to run a port forwarding:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl port-forward prometheus-deployment-2234044252-of29y <span class="m">8080</span>:9090
<span class="gp">$ </span>curl http://127.0.0.1:8080/metrics
</pre></div>
<p>Check Prometheus container logs:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl logs -f po/prometheus-deployment-2234044252-of29y
<span class="go">time="2016-10-29T05:02:38Z" level=info msg="Starting prometheus (version=1.2.1, branch=master, revision=dd66f2e94b2b662804b9aa1b6a50587b990ba8b7)" source="main.go:75"</span>
</pre></div>
<p>And finally we can run a shell inside the Pod's container and have a look at Prometheus config file.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl <span class="nb">exec</span> prometheus-deployment-2234044252-of29y -it /bin/sh
<span class="go">root$ cat /etc/prometheus/prometheus.yml</span>
</pre></div>
<p>Its advisable to describe Deployments in configuration files,
so we can have a better visibility and version control over our cluster.
The following Deployment manifest is similar to what we achieved with <cite>kubectl run</cite> command.</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">extensions/v1beta1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Deployment</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prometheus-deployment</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span><span class="w"> </span><span class="c1"># tells deployment to run 1 pod matching the template below</span><span class="w"></span>
<span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span><span class="c1"># crete pods using pod definition in this template</span><span class="w"></span>
<span class="w"> </span><span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">labels</span><span class="p">:</span><span class="w"> </span><span class="c1"># these key value pairs will be attached to pods</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prometheus-server</span><span class="w"></span>
<span class="w"> </span><span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">containers</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prometheus</span><span class="w"></span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prom/prometheus:v1.2.1</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">9090</span><span class="w"> </span><span class="c1"># port we open in the container</span><span class="w"></span>
</pre></div>
<p>Let's delete <cite>prometheus-deployment</cite> we created via <cite>kubectl run</cite> command</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl delete deployment prometheus-deployment
</pre></div>
<p>and re-create the Deployment from a file (it is available in git repository):</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>git clone https://github.com/marselester/prometheus-on-kubernetes.git
<span class="gp">$ </span><span class="nb">cd</span> ./prometheus-on-kubernetes/
<span class="gp">$ </span>kubectl create -f kube/prometheus/deployment-v1.yml
<span class="gp">$ </span>kubectl get deployments
<span class="go">NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE</span>
<span class="go">prometheus-deployment 1 1 1 0 40s</span>
</pre></div>
</div>
<div class="section" id="prometheus-service">
<h2>Prometheus Service</h2>
<p>We have a Prometheus Pod running.
Now we need <a class="reference external" href="http://kubernetes.io/docs/user-guide/services/">Kubernetes Service</a> to let external clients access it.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl expose deployment prometheus-deployment --type<span class="o">=</span>NodePort --name<span class="o">=</span>prometheus-service
</pre></div>
<p>The assigned port can be found in <cite>NodePort</cite> output of Service description:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl describe service prometheus-service
<span class="gp"># </span>...
<span class="go">NodePort: <unset> 32514/TCP</span>
<span class="gp"># </span>...
</pre></div>
<p>We have exposed the Service on an external port <cite>32514</cite> on all nodes in our cluster.
Now create a firewall rule to allow external traffic.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>gcloud compute firewall-rules create prometheus-nodeport --allow<span class="o">=</span>tcp:32514
</pre></div>
<p>Let's use one of the external IPs from our Kubernetes cluster instances to see Prometheus expression browser.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>gcloud compute instances list
</pre></div>
<p>You should be able to access Prometheus on <cite>http://<EXTERNAL_IP>:32514</cite>.
Let's delete the Service and create it via Service config file.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl delete service prometheus-service
<span class="gp">$ </span>kubectl create -f kube/prometheus/service-v1.yml
</pre></div>
<p>Since the Prometheus Pod exposes <cite>9090</cite> port and has <cite>app: prometheus-server</cite> label,
our config should be as following:</p>
<div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
<span class="nt">metadata</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prometheus-service</span><span class="w"></span>
<span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"> </span><span class="c1"># exposes any pods with the following labels as a service</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prometheus-server</span><span class="w"></span>
<span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NodePort</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"> </span><span class="c1"># this Service's port (cluster-internal IP clusterIP)</span><span class="w"></span>
<span class="w"> </span><span class="nt">targetPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">9090</span><span class="w"> </span><span class="c1"># pods expose this port</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Kubernetes master will allocate a port from a flag-configured range (default: 30000-32767),</span><span class="w"></span>
<span class="w"> </span><span class="c1"># or we can set a specific port number (in our case).</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Each node will proxy 32514 port (the same port number on every node) into this service.</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Note that this Service will be visible as both NodeIP:nodePort and clusterIp:port</span><span class="w"></span>
<span class="w"> </span><span class="nt">nodePort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">32514</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="prometheus-config">
<h2>Prometheus Config</h2>
<p>So far we've been using the default Prometheus config which is part of a Docker image.
For sure we will need to update it so Prometheus can collect metrics from our example app.
Let's take the default config as a starting point and store it in
<a class="reference external" href="http://kubernetes.io/docs/user-guide/configmap/">Kubernetes ConfigMap</a>.
The config can be copied from the running container or from the git repository.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl <span class="nb">exec</span> prometheus-deployment-2234044252-of29y -it cat /etc/prometheus/prometheus.yml
</pre></div>
<p>Next we need to create a ConfigMap entry for the <cite>prometheus.yml</cite> file:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create configmap prometheus-server-conf --from-file<span class="o">=</span>prometheus.yml<span class="o">=</span>kube/prometheus/config-v1.yml
</pre></div>
<p>Now let's mount <cite>prometheus-server-conf</cite> ConfigMap volume to our Prometheus Pod</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f kube/prometheus/deployment-v2.yaml
</pre></div>
<p>and store metrics in emptyDir volume, so we don't lose them when a container in the Pod crashes.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f kube/prometheus/deployment-v3.yml
</pre></div>
</div>
<div class="section" id="sending-app-metrics">
<h2>Sending App Metrics</h2>
<p>We have a Prometheus running in Kubernetes but we have no application to monitor.
I wrote <a class="reference external" href="https://github.com/marselester/prometheus-on-kubernetes/blob/master/hello-app/v1/main.go">hello-app/v1</a> web app that exposes <cite>/hello</cite> HTTP endpoint.</p>
<p><a class="reference external" href="https://youtu.be/HkEZ1LJ7kzQ?list=PLDWZ5uzn69eyh791ZTkEA9OaTxVpGY8_g">BjΓΆrn Rabenstein in his talk</a>
explains how to instrument your code to expose metrics to Prometheus.
Our application is not forced to use Prometheus client to expose metrics.
We can create <cite>/metrics</cite> HTTP endpoint manually in the following text format:</p>
<div class="highlight"><pre><span></span># HELP http_requests_total Number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{code="200",method="get"} 2384
</pre></div>
<p>But it's much easier to use a library
(see <a class="reference external" href="https://github.com/marselester/prometheus-on-kubernetes/blob/master/hello-app/v2/main.go">hello-app/v2</a>).</p>
<div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="s">"github.com/prometheus/client_golang/prometheus/promhttp"</span><span class="w"></span>
<span class="c1">// ...</span><span class="w"></span>
<span class="nx">http</span><span class="p">.</span><span class="nx">Handle</span><span class="p">(</span><span class="s">"/metrics"</span><span class="p">,</span><span class="w"> </span><span class="nx">promhttp</span><span class="p">.</span><span class="nx">Handler</span><span class="p">())</span><span class="w"></span>
</pre></div>
<p>Now the app has <cite>/metrics</cite> endpoint with Go runtime metrics (number of goroutines, GC statistics).</p>
<p>Our app exposes an important endpoint <cite>/hello</cite>.</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">helloHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">status</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">doSomeWork</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">WriteHeader</span><span class="p">(</span><span class="nx">status</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">"Hello, World!\n"</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Let's instrument <cite>helloHandler()</cite> to count HTTP requests and their durations.
First, we need to define metrics.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="s">"github.com/prometheus/client_golang/prometheus"</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="c1">// How often our /hello request durations fall into one of the defined buckets.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We can use default buckets or set ones we are interested in.</span><span class="w"></span>
<span class="w"> </span><span class="nx">duration</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">NewHistogram</span><span class="p">(</span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">HistogramOpts</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Name</span><span class="p">:</span><span class="w"> </span><span class="s">"hello_request_duration_seconds"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Help</span><span class="p">:</span><span class="w"> </span><span class="s">"Histogram of the /hello request duration."</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Buckets</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="kt">float64</span><span class="p">{</span><span class="mf">0.01</span><span class="p">,</span><span class="w"> </span><span class="mf">0.025</span><span class="p">,</span><span class="w"> </span><span class="mf">0.05</span><span class="p">,</span><span class="w"> </span><span class="mf">0.1</span><span class="p">,</span><span class="w"> </span><span class="mf">0.25</span><span class="p">,</span><span class="w"> </span><span class="mf">0.5</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mf">2.5</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Counter vector to which we can attach labels. That creates many key-value</span><span class="w"></span>
<span class="w"> </span><span class="c1">// label combinations. So in our case we count requests by status code separetly.</span><span class="w"></span>
<span class="w"> </span><span class="nx">counter</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">NewCounterVec</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">CounterOpts</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Name</span><span class="p">:</span><span class="w"> </span><span class="s">"hello_requests_total"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Help</span><span class="p">:</span><span class="w"> </span><span class="s">"Total number of /hello requests."</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"status"</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="c1">// init registers Prometheus metrics.</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">init</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">MustRegister</span><span class="p">(</span><span class="nx">duration</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">MustRegister</span><span class="p">(</span><span class="nx">counter</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Second, measure a request duration in seconds and increase the counter in the <cite>helloHandler()</cite> function.</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">helloHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">status</span><span class="w"> </span><span class="kt">int</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">begun</span><span class="w"> </span><span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">duration</span><span class="p">.</span><span class="nx">Observe</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Since</span><span class="p">(</span><span class="nx">begun</span><span class="p">).</span><span class="nx">Seconds</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="c1">// hello_requests_total{status="200"} 2385</span><span class="w"></span>
<span class="w"> </span><span class="nx">counter</span><span class="p">.</span><span class="nx">With</span><span class="p">(</span><span class="nx">prometheus</span><span class="p">.</span><span class="nx">Labels</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"status"</span><span class="p">:</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprint</span><span class="p">(</span><span class="nx">status</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="p">}).</span><span class="nx">Inc</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">}(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Now</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="nx">status</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">doSomeWork</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">WriteHeader</span><span class="p">(</span><span class="nx">status</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">"Hello, World!\n"</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p><a class="reference external" href="https://github.com/marselester/prometheus-on-kubernetes/blob/master/hello-app/v3/main.go">hello-app/v3</a>
is used in further examples.</p>
</div>
<div class="section" id="hello-app-demo">
<h2>Hello App Demo</h2>
<p>Next step is to run the web app in Kubernetes.
A Docker image of the <cite>hello-app/v3</cite> is available on Docker Hub.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f kube/hello/deployment-v1.yml
<span class="gp">$ </span>kubectl port-forward hello-deployment-1471727270-eaknp <span class="m">8000</span>:8000
<span class="gp">$ </span>curl localhost:8000/hello
<span class="go">Hello, World!</span>
<span class="gp">$ </span>curl localhost:8000/metrics
<span class="go">...</span>
<span class="gp"># </span>HELP hello_request_duration_seconds Histogram of the /hello request duration.
<span class="gp"># </span>TYPE hello_request_duration_seconds histogram
<span class="go">hello_request_duration_seconds_bucket{le="0.01"} 0</span>
<span class="go">hello_request_duration_seconds_bucket{le="0.025"} 0</span>
<span class="go">hello_request_duration_seconds_bucket{le="0.05"} 0</span>
<span class="go">hello_request_duration_seconds_bucket{le="0.1"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="0.25"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="0.5"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="1"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="2.5"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="5"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="10"} 1</span>
<span class="go">hello_request_duration_seconds_bucket{le="+Inf"} 1</span>
<span class="go">hello_request_duration_seconds_sum 0.083953974</span>
<span class="go">hello_request_duration_seconds_count 1</span>
<span class="gp"># </span>HELP hello_requests_total Total number of /hello requests.
<span class="gp"># </span>TYPE hello_requests_total counter
<span class="go">hello_requests_total{status="500"} 1</span>
</pre></div>
<p>The Service creation is similar to what we have already done before.
We use <cite>32515</cite> NodePort here.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl apply -f kube/hello/service-v1.yml
<span class="gp">$ </span>gcloud compute firewall-rules create hello-nodeport --allow<span class="o">=</span>tcp:32515
</pre></div>
<p>Now it is possible to see the app's metrics from Internet.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl http://<EXTERNAL_IP>:32515/metrics
</pre></div>
</div>
<div class="section" id="hello-prometheus">
<h2>Hello Prometheus</h2>
<p>Since the web app is run on Kubernetes, we can configure Prometheus
to scrape metrics from <cite>/metrics</cite> HTTP endpoint of hello-app Pods.</p>
<div class="highlight"><pre><span></span><span class="nt">global</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">scrape_interval</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">5s</span><span class="w"></span>
<span class="w"> </span><span class="nt">evaluation_interval</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">5s</span><span class="w"></span>
<span class="nt">scrape_configs</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s">'prometheus'</span><span class="w"></span>
<span class="w"> </span><span class="nt">static_configs</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">targets</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="s">'localhost:9090'</span><span class="p p-Indicator">]</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">job_name</span><span class="p">:</span><span class="w"> </span><span class="s">'hello'</span><span class="w"></span>
<span class="w"> </span><span class="c1"># The information to access the Kubernetes API to discover targets.</span><span class="w"></span>
<span class="w"> </span><span class="nt">kubernetes_sd_configs</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">api_servers</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'https://kubernetes.default.svc'</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Prometheus assumes it is being run inside a Kubernetes pod.</span><span class="w"></span>
<span class="w"> </span><span class="nt">in_cluster</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Only pods should be discovered.</span><span class="w"></span>
<span class="w"> </span><span class="nt">role</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pod</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Prometheus collects metrics from pods with "app: hello-server" label.</span><span class="w"></span>
<span class="w"> </span><span class="c1"># Prometheus gets 'hello_requests_total{status="500"} 1'</span><span class="w"></span>
<span class="w"> </span><span class="c1"># from hello:8000/metrics and adds "job" and "instance" labels, so it becomes</span><span class="w"></span>
<span class="w"> </span><span class="c1"># 'hello_requests_total{instance="10.16.0.10:8000",job="hello",status="500"} 1'.</span><span class="w"></span>
<span class="w"> </span><span class="nt">relabel_configs</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source_labels</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">__meta_kubernetes_pod_label_app</span><span class="p p-Indicator">]</span><span class="w"></span>
<span class="w"> </span><span class="nt">regex</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">hello-server</span><span class="w"></span>
<span class="w"> </span><span class="nt">action</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">keep</span><span class="w"></span>
</pre></div>
<p>Update a ConfigMap of Prometheus config and re-create a Prometheus Pod so it picks up changes.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>kubectl create configmap prometheus-server-conf <span class="se">\</span>
--from-file<span class="o">=</span>prometheus.yml<span class="o">=</span>kube/prometheus/config-v2.yaml <span class="se">\</span>
-o yaml <span class="se">\</span>
--dry-run <span class="p">|</span> kubectl replace -f -
</pre></div>
<p>Finally, the app's metrics should show up in Prometheus expression browser <cite>http://<EXTERNAL_IP>:32514</cite>.</p>
<p>You can try to query <cite>hello_requests_total</cite> which shows how many requests we've served since the beginning.
Let's see how many requests we've served in last 5 minutes normalised per second (QPS)
with <cite>rate(hello_requests_total[5m])</cite> query.</p>
<p>Here I gave an example of how to run Prometheus in Kubernetes cluster
and collect metrics from a simple web app.
However recently I have encountered CoreOS <a class="reference external" href="https://github.com/coreos/kube-prometheus">kube-prometheus</a>
which makes it easier. Have a look at <a class="reference external" href="https://coreos.com/blog/the-prometheus-operator.html">The Prometheus Operator: Managed Prometheus setups for Kubernetes</a> for more details.</p>
</div>
Django REST framework: pagination on PostgreSQL triggers2016-04-02T00:00:00+07:002016-04-02T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2016-04-02:/drf-pagination.html<p>Django and Django REST Framework use SQL COUNT in pagination.
As your database grows SQL COUNT becomes too slow. Fortunately the frameworks
are well designed and allow to customize a way items are count.
Let me illustrate that on a typical "books" example.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Author</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField β¦</span></pre></div><p>Django and Django REST Framework use SQL COUNT in pagination.
As your database grows SQL COUNT becomes too slow. Fortunately the frameworks
are well designed and allow to customize a way items are count.
Let me illustrate that on a typical "books" example.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Author</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">()</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">db_table</span> <span class="o">=</span> <span class="s1">'author'</span>
<span class="k">class</span> <span class="nc">Book</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">Author</span><span class="p">,</span> <span class="n">related_name</span><span class="o">=</span><span class="s1">'books'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">db_table</span> <span class="o">=</span> <span class="s1">'book'</span>
</pre></div>
<p>It has been working fine for some time but authors are so productive and wrote thousands of books.
As a result <tt class="docutils literal">alice.books.count()</tt> takes a few seconds to execute.
One option is to store books count in <tt class="docutils literal">Author</tt> model which might be helpful in
other SQL queries. Another one is to keep count somewhere else, e.g., Redis.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Author</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">()</span>
<span class="n">books_count</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">PositiveIntegerField</span><span class="p">(</span><span class="n">null</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">db_table</span> <span class="o">=</span> <span class="s1">'author'</span>
</pre></div>
<p>Note that <tt class="docutils literal">books_count</tt> is nullable due to existing data on production servers.
Setting a default value <tt class="docutils literal">0</tt> on schema migration would take long to update
every author record, also it's hard to tell whether an author record is
already migrated (<tt class="docutils literal">books_count</tt> is calculated or not).</p>
<p>Now we have <tt class="docutils literal">Author.books_count</tt> field which should keep track of new books.
You can implement that based on Django signals or SQL triggers.</p>
<p>Signals might fail, even with <tt class="docutils literal">send_robust</tt>. Updating a books count in
a signal receiver slows down an application (two to three DB queries and
acquiring a DB lock on author record):</p>
<div class="highlight"><pre><span></span><span class="nd">@receiver</span><span class="p">(</span><span class="n">post_save</span><span class="p">,</span> <span class="n">sender</span><span class="o">=</span><span class="n">Book</span><span class="p">,</span> <span class="n">dispatch_uid</span><span class="o">=</span><span class="s1">'update_author_books_count'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">update_author_books_count</span><span class="p">(</span><span class="n">sender</span><span class="p">,</span> <span class="n">instance</span><span class="p">,</span> <span class="n">created</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">created</span><span class="p">:</span>
<span class="k">return</span>
<span class="n">book</span> <span class="o">=</span> <span class="n">instance</span>
<span class="k">with</span> <span class="n">atomic</span><span class="p">():</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">Author</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">select_for_update</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">pk</span><span class="o">=</span><span class="n">book</span><span class="o">.</span><span class="n">author_id</span><span class="p">)</span>
<span class="k">if</span> <span class="n">author</span><span class="o">.</span><span class="n">books_count</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">author</span><span class="o">.</span><span class="n">books_count</span> <span class="o">=</span> <span class="n">author</span><span class="o">.</span><span class="n">books</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">author</span><span class="o">.</span><span class="n">books_count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">author</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
</pre></div>
<p>Moving an update logic from a signal receiver to Celery task doesn't look
promising -- <tt class="docutils literal">books_count</tt> would be inconsistent between a task trigger and
its execution or simply Celery task might fail. Another issue is that
a model signal is not "attached" to DB transaction. For example,
a Celery task is run before a DB transaction is committed.
That is usually quick fixed with <tt class="docutils literal">task_name.apply_async(countdown=1)</tt>
to delay a task execution, but I would rather recommend something like
<a class="reference external" href="https://django-transaction-hooks.readthedocs.org/en/latest/">django-transaction-hooks</a>.</p>
<p>What about SQL trigger option? A bad thing is that a business logic leaks from an app to DB.
But with a good documentation and tests it's maintainable: SQL code is stored in
a data migration module.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>python manage.py makemigrations --empty yourappname
</pre></div>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">migrations</span>
<span class="n">CREATE_FUNCTION_SQL</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2">CREATE OR REPLACE FUNCTION update_books_count_on_author() RETURNS trigger AS $$</span>
<span class="s2"> BEGIN</span>
<span class="s2"> UPDATE author SET books_count =</span>
<span class="s2"> CASE</span>
<span class="s2"> WHEN books_count IS NULL THEN (</span>
<span class="s2"> SELECT count(*) FROM book WHERE author_id = author.id</span>
<span class="s2"> )</span>
<span class="s2"> ELSE books_count + 1</span>
<span class="s2"> END</span>
<span class="s2"> WHERE id = NEW.author_id;</span>
<span class="s2"> RETURN NEW;</span>
<span class="s2"> END;</span>
<span class="s2">$$ LANGUAGE plpgsql;</span>
<span class="s2">"""</span>
<span class="n">CREATE_TRIGGER_SQL</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2">CREATE TRIGGER books_count_update</span>
<span class="s2"> AFTER INSERT ON book</span>
<span class="s2"> FOR EACH ROW</span>
<span class="s2"> EXECUTE PROCEDURE update_books_count_on_author();</span>
<span class="s2">"""</span>
<span class="k">class</span> <span class="nc">Migration</span><span class="p">(</span><span class="n">migrations</span><span class="o">.</span><span class="n">Migration</span><span class="p">):</span>
<span class="n">dependencies</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">operations</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">migrations</span><span class="o">.</span><span class="n">RunSQL</span><span class="p">(</span><span class="n">CREATE_FUNCTION_SQL</span><span class="p">),</span>
<span class="n">migrations</span><span class="o">.</span><span class="n">RunSQL</span><span class="p">(</span><span class="n">CREATE_TRIGGER_SQL</span><span class="p">),</span>
<span class="p">]</span>
</pre></div>
<p>Here is a pitfall though. When a new book is created and author is updated on the
same DB transaction, then <tt class="docutils literal">books_count</tt> value might be overwritten.</p>
<div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">atomic</span><span class="p">():</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">Author</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">select_for_update</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">pk</span><span class="o">=</span><span class="n">author_id</span><span class="p">)</span>
<span class="n">Book</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">author</span><span class="o">=</span><span class="n">author</span><span class="p">)</span>
<span class="n">author</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="s1">'Bob'</span>
<span class="n">author</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
</pre></div>
<p>You can either explicitly list fields to update <tt class="docutils literal"><span class="pre">author.save(update_fields=['name'])</span></tt>
or use <a class="reference external" href="https://django-save-the-change.readthedocs.org/en/latest/">django-save-the-change</a>. Let's document that in the model docstring.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">save_the_change.mixins</span> <span class="kn">import</span> <span class="n">SaveTheChange</span>
<span class="k">class</span> <span class="nc">Author</span><span class="p">(</span><span class="n">SaveTheChange</span><span class="p">,</span> <span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="sd">"""Author, e.g., Terry Pratchett.</span>
<span class="sd"> :attribute books_count: How many books writer has. SQL COUNT is</span>
<span class="sd"> expensive operation, so we store calculated value and update it</span>
<span class="sd"> by SQL trigger (check a data migration module for details).</span>
<span class="sd"> It's important to save only fields that were updated in the model.</span>
<span class="sd"> Otherwise SQL trigger's results are overwritten by Django ORM.</span>
<span class="sd"> For example:</span>
<span class="sd"> 1. author is requested with a lock (books_count = 1)</span>
<span class="sd"> 2. new book is created</span>
<span class="sd"> 3. SQL trigger updates author's books_count field (now it is 2)</span>
<span class="sd"> 4. author instance is saved with the old value of books_count = 1.</span>
<span class="sd"> SaveTheChange mixin helps to prevent it.</span>
<span class="sd"> """</span>
</pre></div>
<p>To benefit from <tt class="docutils literal">books_count</tt> field in Django REST Framework we need
a custom pagination class which implements <tt class="docutils literal">Paginator.count</tt> property.
The idea is to extract author ID from paginator's SQL, query a books count
from <tt class="docutils literal">Author</tt> model and return it, instead of default
<tt class="docutils literal"><span class="pre">Book.objects.filter(author_id=author_id).count()</span></tt>.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.core.paginator</span> <span class="kn">import</span> <span class="n">Paginator</span>
<span class="kn">from</span> <span class="nn">rest_framework</span> <span class="kn">import</span> <span class="n">pagination</span>
<span class="kn">from</span> <span class="nn">rest_framework.viewsets</span> <span class="kn">import</span> <span class="n">ReadOnlyModelViewSet</span>
<span class="k">class</span> <span class="nc">BookViewSet</span><span class="p">(</span><span class="n">ReadOnlyModelViewSet</span><span class="p">):</span>
<span class="n">pagination_class</span> <span class="o">=</span> <span class="n">BookPagination</span>
<span class="k">class</span> <span class="nc">BookPagination</span><span class="p">(</span><span class="n">pagination</span><span class="o">.</span><span class="n">PageNumberPagination</span><span class="p">):</span>
<span class="n">django_paginator_class</span> <span class="o">=</span> <span class="n">CachedBookCountPaginator</span>
<span class="k">class</span> <span class="nc">CachedBookCountPaginator</span><span class="p">(</span><span class="n">Paginator</span><span class="p">):</span>
<span class="nd">@cached_property</span>
<span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Return the total number of books, across all pages.</span>
<span class="sd"> It parses a SQL and learns what author ID was requested</span>
<span class="sd"> based on ``self.object_list.query``. After that we can get</span>
<span class="sd"> a cached books count from Author model.</span>
<span class="sd"> """</span>
<span class="c1"># There is query.where, but I could't find an author ID easily.</span>
<span class="c1"># Moreover query.where internals might be changed.</span>
<span class="n">sql</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">object_list</span><span class="o">.</span><span class="n">query</span><span class="p">)</span>
<span class="n">author_id</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_author_id_from_sql</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
<span class="n">author</span> <span class="o">=</span> <span class="n">Author</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">pk</span><span class="o">=</span><span class="n">author_id</span><span class="p">)</span><span class="o">.</span><span class="n">only</span><span class="p">(</span><span class="s1">'books_count'</span><span class="p">)</span>
<span class="c1"># In case we got unsynced author, we fallback to SQL COUNT.</span>
<span class="k">if</span> <span class="n">author</span><span class="o">.</span><span class="n">books_count</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query_count</span><span class="p">()</span>
<span class="k">return</span> <span class="n">author</span><span class="o">.</span><span class="n">books_count</span>
<span class="k">def</span> <span class="nf">query_count</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Request books count from DB.</span>
<span class="sd"> We need this method to facilitate testing (mocks).</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">CachedBookCountPaginator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">count</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">_get_author_id_from_sql</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">sql</span><span class="p">):</span>
<span class="k">pass</span>
</pre></div>
<p>I hope this helps. Cheers!</p>
API based on Flask2013-12-09T00:00:00+07:002013-12-09T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2013-12-09:/api-based-on-flask.html<p>Here I want to consider implementation of API best practices which
usually don't follow Fielding's REST strictly. <a class="reference external" href="https://github.com/marselester/api-example-based-on-flask">Example Flask project</a>
is on GitHub.</p>
<div class="section" id="api-versioning">
<h2>API Versioning</h2>
<p>Interfaces are changed hence versioning is mandatory in order to not annoy
your users. You might need to add new resource or field to particular β¦</p></div><p>Here I want to consider implementation of API best practices which
usually don't follow Fielding's REST strictly. <a class="reference external" href="https://github.com/marselester/api-example-based-on-flask">Example Flask project</a>
is on GitHub.</p>
<div class="section" id="api-versioning">
<h2>API Versioning</h2>
<p>Interfaces are changed hence versioning is mandatory in order to not annoy
your users. You might need to add new resource or field to particular resource.
You write code, tests and update documentation. Users are happy.
It's possible that you have to rename or delete field of some resource.
This case is harder and you might make the easiest decision β spawn
a lot of <tt class="docutils literal">if</tt> statements and write more tests consequently.
Code base maintaining is getting worse.</p>
<p>I think it's better to make API versions isolated.
It will keep things simple and tests as well. You may even want to change
framework if legacy API implementation is not good enough.
For example, you have two WSGI applications. Each application implements
certain API version, e.g, <strong>messaging_api_v1</strong>, <strong>messaging_api_v2</strong>.
In order to hide versioning information from applications, e.g., URL prefix,
you can dispatch requests by Werkzeug's <tt class="docutils literal">DispatcherMiddleware</tt>.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">werkzeug.wsgi</span> <span class="kn">import</span> <span class="n">DispatcherMiddleware</span>
<span class="kn">from</span> <span class="nn">werkzeug.exceptions</span> <span class="kn">import</span> <span class="n">NotFound</span>
<span class="kn">import</span> <span class="nn">messaging_api_v1</span>
<span class="kn">import</span> <span class="nn">messaging_api_v2</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">DispatcherMiddleware</span><span class="p">(</span><span class="n">NotFound</span><span class="p">(),</span> <span class="p">{</span>
<span class="s1">'/v1'</span><span class="p">:</span> <span class="n">messaging_api_v1</span><span class="o">.</span><span class="n">app</span><span class="p">,</span>
<span class="s1">'/v2'</span><span class="p">:</span> <span class="n">messaging_api_v2</span><span class="o">.</span><span class="n">app</span><span class="p">,</span>
<span class="p">})</span>
</pre></div>
<p>Dispatcher in work:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>gunicorn messaging_api:app
</pre></div>
<p>Let's request the same API resource from different WSGI applications:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl http://127.0.0.1:8000/v1/messages/1
<span class="go">{</span>
<span class="go"> "content": "hi world from V1"</span>
<span class="go">}</span>
<span class="gp">$ </span>curl http://127.0.0.1:8000/v2/messages/1
<span class="go">{</span>
<span class="go"> "content": "hi world from V2"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="urls">
<h2>URLs</h2>
<p>API resources are nouns, so <tt class="docutils literal">/messages/</tt> URL is for a collection of messages.
You should think about it as UNIX folder. Hence <tt class="docutils literal">/messages</tt> is correct
folder path and Flask will redirect to the canonical URL <tt class="docutils literal">/messages/</tt>.
Certain message's <tt class="docutils literal">/messages/1</tt> URL must not contain trailing slash
in order to look like UNIX file path.</p>
<p>Resources usually have relationships and they might be expressed in URLs,
e.g., get messages from account which id is <tt class="docutils literal">1</tt>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl https://api.example.com/v1/accounts/1/messages/
</pre></div>
</div>
<div class="section" id="representation">
<h2>Representation</h2>
<p>There are two popular approaches to specify response format. First one
is based on <strong>Accept</strong> header, second β URL based. Here I use header based
approach.</p>
<p>I wrote <a class="reference external" href="https://github.com/marselester/flask-api-utils#accept-header-based-response">ResponsiveFlask</a> class which extends <tt class="docutils literal">Flask</tt> by supporting
dictionary response. Views can return dict and it will be represented
based on <strong>Accept</strong> header. When <tt class="docutils literal">ResponsiveFlask.make_response()</tt> receives
dictionary it creates real response object using appropriate formatter.
Formatter is picked by mimetype.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl http://127.0.0.1:8000/v2/messages/1 -i -H <span class="s1">'Accept: application/json'</span>
<span class="go">HTTP/1.1 200 OK</span>
<span class="go">Server: gunicorn/18.0</span>
<span class="go">Date: Tue, 10 Dec 2013 07:52:31 GMT</span>
<span class="go">Connection: close</span>
<span class="go">Content-Type: application/json</span>
<span class="go">Content-Length: 35</span>
<span class="go">{</span>
<span class="go"> "content": "hi world from V2"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="error-handling">
<h2>Error Handling</h2>
<p>Flask shows error pages by default with basic description in
<strong>text/html</strong> format. It would be better if error representation depends
on <strong>Accept</strong> header. <tt class="docutils literal">ResponsiveFlask</tt> class concerns about it.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl http://127.0.0.1:8000/v2/messages/666 -i
<span class="go">HTTP/1.1 404 NOT FOUND</span>
<span class="go">Server: gunicorn/18.0</span>
<span class="go">Date: Fri, 03 Jan 2014 05:16:03 GMT</span>
<span class="go">Connection: close</span>
<span class="go">Content-Type: application/json</span>
<span class="go">Content-Length: 49</span>
<span class="go">{</span>
<span class="go"> "code": 404,</span>
<span class="go"> "message": "404: Not Found"</span>
<span class="go">}</span>
</pre></div>
<p>You can set your own HTTP error handler by using
<a class="reference external" href="https://github.com/marselester/flask-api-utils#error-handling">app.default_errorhandler</a> decorator. Note that it might override
already defined error handlers, so you should declare it before them.</p>
<p>It's convenient to add URL of detailed error description into response.</p>
<div class="highlight"><pre><span></span><span class="go">{</span>
<span class="go"> "code": 404,</span>
<span class="go"> "info_url": "http://developer.example.com/errors.html#error-code-404",</span>
<span class="go"> "message": "404: Not Found"</span>
<span class="go">}</span>
</pre></div>
</div>
<div class="section" id="misc">
<h2>Misc</h2>
<p>It's good idea to keep in mind following:</p>
<ul class="simple">
<li>HTTPS;</li>
<li>response should contain resource url, e.g.,
<tt class="docutils literal">{'url': <span class="pre">'https://api.example.com/v2/messages/1'}</span></tt>;</li>
<li>pagination by <tt class="docutils literal">offset</tt> and <tt class="docutils literal">limit</tt> QS arguments with default values;</li>
<li>filtration and search by QS arguments;</li>
<li>partial response by <tt class="docutils literal">fields=id,lastname</tt> QS argument.</li>
</ul>
</div>
Slides about SaltStack2013-12-03T00:00:00+07:002013-12-03T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2013-12-03:/saltstack-slides.html<p><strong>Update</strong> I gave a talk with <a class="reference external" href="https://twitter.com/shr">Simon Robson</a> at <a class="reference external" href="https://twitter.com/barcampcm">Beercamp</a> in Chiang Mai,
Thailand on 12 Dec 2013. We compared Salt and <a class="reference external" href="http://slid.es/simonrobson/ansible">Ansible</a>.
Here are my <a class="reference external" href="https://slid.es/marselester/saltstack">slides</a>.</p>
<p><strong>Update</strong> I gave a talk with <a class="reference external" href="https://twitter.com/shr">Simon Robson</a> at <a class="reference external" href="https://twitter.com/barcampcm">Beercamp</a> in Chiang Mai,
Thailand on 12 Dec 2013. We compared Salt and <a class="reference external" href="http://slid.es/simonrobson/ansible">Ansible</a>.
Here are my <a class="reference external" href="https://slid.es/marselester/saltstack">slides</a>.</p>
Developing & Deploying Django project with SaltStack2013-11-28T00:00:00+07:002013-11-28T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2013-11-28:/developing-and-deploying-django-project-with-saltstack.html<p>Eventually you will need to deploy project,
but deployment was not considered in the <a class="reference external" href="https://marselester.com/developing-django-project-with-saltstack.html">previous post</a>. Let's find it out.</p>
<p>Server configuration is different from local, thus environments will be needed
(at least production <strong>prod</strong> and development <strong>dev</strong>). Salt uses <strong>base</strong>
environment by default.</p>
<p>Environments are set in <tt class="docutils literal">minion.conf β¦</tt></p><p>Eventually you will need to deploy project,
but deployment was not considered in the <a class="reference external" href="https://marselester.com/developing-django-project-with-saltstack.html">previous post</a>. Let's find it out.</p>
<p>Server configuration is different from local, thus environments will be needed
(at least production <strong>prod</strong> and development <strong>dev</strong>). Salt uses <strong>base</strong>
environment by default.</p>
<p>Environments are set in <tt class="docutils literal">minion.conf</tt> because masterless minion is used.
When no environment is set, Salt assumes following configuration:</p>
<div class="highlight"><pre><span></span><span class="nt">file_roots</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">base</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt</span><span class="w"></span>
<span class="nt">pillar_roots</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">base</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/pillar</span><span class="w"></span>
</pre></div>
<p>Salt will look for states in <tt class="docutils literal">/srv/salt</tt> and pillars in <tt class="docutils literal">/srv/pillar</tt>.</p>
<p>Let's introduce <strong>dev</strong> and <strong>prod</strong> environments.</p>
<div class="highlight"><pre><span></span><span class="nt">file_roots</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">base</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/salt/base</span><span class="w"></span>
<span class="w"> </span><span class="nt">dev</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/salt/dev</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/salt/base</span><span class="w"></span>
<span class="w"> </span><span class="nt">prod</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/salt/prod</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/salt/base</span><span class="w"></span>
<span class="nt">pillar_roots</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">base</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/pillar/base</span><span class="w"></span>
<span class="w"> </span><span class="nt">dev</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/pillar/dev</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/pillar/base</span><span class="w"></span>
<span class="w"> </span><span class="nt">prod</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/pillar/prod</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/salt_roots/pillar/base</span><span class="w"></span>
</pre></div>
<p>Each environment describes paths where to find files.
File is searched in the first directory and if it is not found it is
searched in the second one.</p>
<p>States and pillars which were described in <a class="reference external" href="https://marselester.com/developing-django-project-with-saltstack.html">previous post</a> are splitted
to environment folders. Most of files are located in <tt class="docutils literal">base</tt> folder,
because they are the same to <strong>dev</strong> and <strong>prod</strong> environments.
You can find <a class="reference external" href="https://github.com/marselester/abstract-internal-messaging-deploy">complete example</a> on GitHub.</p>
<div class="section" id="what-production-environment-introduces">
<h2>What Production Environment Introduces</h2>
<p>There is a difference between <strong>prod</strong> and <strong>dev</strong> project's source code
delivering. Development environment assumes that code is shared by Vagrant.
Production environment needs to get code from private git repository.
In order to do it user <em>jon_snow</em> must have ssh keys. Here I don't use
private repository, but keys are still needed to log in to production server
by Fabric.</p>
<div class="highlight"><pre><span></span><span class="nt">jon_snow</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">user.present</span><span class="w"></span>
<span class="nt">jon_snow public key</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">ssh_auth</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">present</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://user/id_rsa.pub</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="nt">jon_snow secret key</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/.ssh/id_rsa</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://user/id_rsa</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">600</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">ssh_auth</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow public key</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">prod/source_code.sls</tt> fetches source code from GitHub.
I have tried to use built in <tt class="docutils literal">git.latest</tt> Salt state but it requires
to configure git name and email on server. I solved this issue by using
git hard reset.</p>
<div class="highlight"><pre><span></span><span class="nt">git</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pkg.installed</span><span class="w"></span>
<span class="nt">github.com</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">ssh_known_hosts</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">present</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">fingerprint</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48</span><span class="w"></span>
<span class="nt">messaging repository clone</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">cmd</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">run</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">unless</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test -d /home/jon_snow/messaging_repository</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">></span><span class="w"></span>
<span class="w"> </span><span class="no">git clone https://github.com/marselester/abstract-internal-messaging.git /home/jon_snow/messaging_repository</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">git</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">ssh_known_hosts</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">github.com</span><span class="w"></span>
<span class="nt">messaging latest source code</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">cmd</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">run</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">cwd</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/messaging_repository</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">></span><span class="w"></span>
<span class="w"> </span><span class="no">git fetch origin &&</span><span class="w"></span>
<span class="w"> </span><span class="no">git reset --hard origin/master</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">cmd</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">messaging repository clone</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">symlink</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_src_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">target</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/messaging_repository</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">force</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="fabric-as-deployment-runner">
<h2>Fabric as Deployment Runner</h2>
<p>Last thing to do is to run deployment of particular Salt environment
<strong>salt_env</strong> in particular host <strong>target</strong>. Fabric is good at it.</p>
<p>I want Fabric to:</p>
<ul>
<li><p class="first">bootstrap Salt on given <strong>target</strong>;</p>
</li>
<li><p class="first">upload <tt class="docutils literal">minion.conf</tt> to given <strong>target</strong> to <tt class="docutils literal">/etc/salt/minion</tt>;</p>
</li>
<li><p class="first">upload Salt configs to given <strong>target</strong> how it is described
in <tt class="docutils literal">minion.conf</tt> (to <tt class="docutils literal">/srv/salt_roots</tt>);</p>
</li>
<li><p class="first">invoke <tt class="docutils literal"><span class="pre">salt-call</span></tt> on given <strong>target</strong> with particular <strong>salt_env</strong>
(here is <tt class="docutils literal">prod</tt>).</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>salt-call state.highstate <span class="nv">env</span><span class="o">=</span>prod
</pre></div>
</li>
</ul>
<p>Let's try to deploy <strong>prod</strong> environment to VM. I assume that you have
<tt class="docutils literal">messaging</tt> directory (see <a class="reference external" href="https://marselester.com/developing-django-project-with-saltstack.html">Developing Django project with SaltStack</a>).
You need to clone <a class="reference external" href="https://github.com/marselester/abstract-internal-messaging-deploy">Deploy Abstract Internal Messaging System</a> in it
and start VM.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nb">cd</span> messaging
<span class="gp">$ </span>git clone https://github.com/marselester/abstract-internal-messaging-deploy.git
<span class="gp">$ </span><span class="nb">cd</span> abstract-internal-messaging-deploy/vagrant
<span class="gp">$ </span>vagrant up
</pre></div>
<p>In order to interact with VM you should add <tt class="docutils literal">11.22.33.44 <span class="pre">messaging-part-2</span></tt>
to <tt class="docutils literal">/etc/hosts</tt>.</p>
<p>Install Fabric and set up Salt masterless minion in VM.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nb">cd</span> messaging/abstract-internal-messaging-deploy
<span class="gp">$ </span>pip install -r requirements.txt
<span class="gp">$ </span>fab target:local setup_masterless_minion
</pre></div>
<p>Finally deploy <strong>prod</strong> environment in VM.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>fab target:local salt_env:prod deploy
</pre></div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Besides convenient developing, we got ability to deploy arbitrary environments
to arbitrary targets. It gives us opportunity to test deployment itself.
You can improve this solution by adding ability to specify git branch
which will be deployed.</p>
<p>As in <a class="reference external" href="https://marselester.com/developing-django-project-with-saltstack.html">previous post</a> project should work:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -i messaging-part-2
</pre></div>
</div>
Developing Django project with SaltStack2013-11-09T00:00:00+07:002013-11-09T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2013-11-09:/developing-django-project-with-saltstack.html<p>Let's use <a class="reference external" href="https://github.com/marselester/abstract-internal-messaging">Messaging System</a> as an example of Django project. I want it to
run in VirtualBox which is managed by Vagrant. Infrastructure management
is provided by SaltStack.</p>
<p>I advise you to create separate folder for repositories (currently there
is only one) of project and clone <a class="reference external" href="https://github.com/marselester/abstract-internal-messaging">Messaging System</a> there.
Also β¦</p><p>Let's use <a class="reference external" href="https://github.com/marselester/abstract-internal-messaging">Messaging System</a> as an example of Django project. I want it to
run in VirtualBox which is managed by Vagrant. Infrastructure management
is provided by SaltStack.</p>
<p>I advise you to create separate folder for repositories (currently there
is only one) of project and clone <a class="reference external" href="https://github.com/marselester/abstract-internal-messaging">Messaging System</a> there.
Also I need a folder for Vagrant files and prefer to name it <tt class="docutils literal">vagrant</tt>.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>mkdir messaging
<span class="gp">$ </span><span class="nb">cd</span> messaging
<span class="gp">$ </span>git clone https://github.com/marselester/abstract-internal-messaging.git
<span class="gp">$ </span>mkdir abstract-internal-messaging/vagrant
</pre></div>
<p>In order to make friends of Vagrant and SaltStack, I need to install
Salty Vagrant:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>vagrant plugin install vagrant-salt
</pre></div>
<p>I want Vagrant to:</p>
<ul class="simple">
<li>share source code folder <tt class="docutils literal"><span class="pre">messaging/abstract-internal-messaging</span></tt>;</li>
<li>share states and pillars of SaltStack
<tt class="docutils literal"><span class="pre">messaging/abstract-internal-messaging/vagrant/salt/roots</span></tt>;</li>
<li>run minion masterless by <tt class="docutils literal"><span class="pre">salt-call</span> state.highstate</tt>;</li>
<li>assign convenient host name, here it is "messaging-part-1".</li>
</ul>
<p><tt class="docutils literal">vagrant/Vagrantfile</tt> config responses for these requirements:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- mode: ruby -*-</span>
<span class="no">Vagrant</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span><span class="s1">'2'</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">box</span> <span class="o">=</span> <span class="s1">'precise64'</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">box_url</span> <span class="o">=</span> <span class="s1">'http://files.vagrantup.com/precise64.box'</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">provider</span> <span class="ss">:virtualbox</span> <span class="k">do</span> <span class="o">|</span><span class="n">v</span><span class="o">|</span>
<span class="n">v</span><span class="o">.</span><span class="n">customize</span> <span class="o">[</span><span class="s1">'modifyvm'</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s1">'--name'</span><span class="p">,</span> <span class="s1">'messaging-part-1'</span><span class="o">]</span>
<span class="n">v</span><span class="o">.</span><span class="n">customize</span> <span class="o">[</span><span class="s1">'modifyvm'</span><span class="p">,</span> <span class="ss">:id</span><span class="p">,</span> <span class="s1">'--memory'</span><span class="p">,</span> <span class="mi">1024</span><span class="o">]</span>
<span class="k">end</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">hostname</span> <span class="o">=</span> <span class="s1">'messaging-part-1'</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">network</span> <span class="ss">:private_network</span><span class="p">,</span> <span class="ss">ip</span><span class="p">:</span> <span class="s1">'1.2.3.4'</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">synced_folder</span> <span class="s1">'..'</span><span class="p">,</span> <span class="s1">'/home/vagrant/abstract-internal-messaging'</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">synced_folder</span> <span class="s1">'salt/roots'</span><span class="p">,</span> <span class="s1">'/srv'</span>
<span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">provision</span> <span class="ss">:salt</span> <span class="k">do</span> <span class="o">|</span><span class="n">salt</span><span class="o">|</span>
<span class="n">salt</span><span class="o">.</span><span class="n">minion_config</span> <span class="o">=</span> <span class="s1">'salt/minion.conf'</span>
<span class="n">salt</span><span class="o">.</span><span class="n">run_highstate</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">salt</span><span class="o">.</span><span class="n">verbose</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
<span class="k">end</span>
</pre></div>
<p>As you could notice, I have not had the <tt class="docutils literal">vagrant/salt</tt> folder yet.
Let's create following directory structure (actually you can just
<tt class="docutils literal">git checkout <span class="pre">salted-vagrant</span></tt>):</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>tree abstract-internal-messaging/vagrant
<span class="go">abstract-internal-messaging/vagrant</span>
<span class="go">|-- Vagrantfile</span>
<span class="go">`-- salt</span>
<span class="go"> |-- minion.conf</span>
<span class="go"> `-- roots</span>
<span class="go"> |-- pillar</span>
<span class="go"> | |-- top.sls</span>
<span class="go"> | |-- postgresql.sls</span>
<span class="go"> | `-- website.sls</span>
<span class="go"> `-- salt</span>
<span class="go"> |-- top.sls</span>
<span class="go"> |-- user.sls</span>
<span class="go"> |-- source_code.sls</span>
<span class="go"> |-- python.sls</span>
<span class="go"> |-- redis.sls</span>
<span class="go"> |-- postgresql</span>
<span class="go"> | |-- init.sls</span>
<span class="go"> | `-- pg_hba.conf</span>
<span class="go"> `-- website</span>
<span class="go"> |-- django.sls</span>
<span class="go"> |-- wsgiserver.sls</span>
<span class="go"> |-- webserver.sls</span>
<span class="go"> |-- local.py.template</span>
<span class="go"> |-- supervisord.conf</span>
<span class="go"> |-- gunicorn.conf.py</span>
<span class="go"> `-- nginx.conf</span>
</pre></div>
<p><tt class="docutils literal">minion.conf</tt> contains only one string:</p>
<div class="highlight"><pre><span></span><span class="nt">file_client</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">local</span><span class="w"></span>
</pre></div>
<p>It means "Don't look for master server when running <tt class="docutils literal"><span class="pre">salt-call</span></tt>".
It will assume that the local system has all of the file and pillar resources.</p>
<p>Rest of files I will consider separately.</p>
<div class="section" id="user-and-source-code-states">
<h2>User and Source Code states</h2>
<p><tt class="docutils literal">salt/user.sls</tt> creates user "jon_snow".</p>
<div class="highlight"><pre><span></span><span class="nt">jon_snow</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">user.present</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">salt/source_code.sls</tt> makes shared source code available in jon_snow's home
by creating symlink.</p>
<div class="highlight"><pre><span></span><span class="nt">messaging source code</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">symlink</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/abstract-internal-messaging</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">target</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/vagrant/abstract-internal-messaging</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">force</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
</pre></div>
<p>Hardcoded path can be replaced by <tt class="docutils literal">{{ <span class="pre">pillar['website_src_dir']</span> }}</tt>,
which will be introduced further.</p>
</div>
<div class="section" id="python-state">
<h2>Python state</h2>
<p><tt class="docutils literal">salt/python.sls</tt> installs Python 2, pip and Virtualenv.</p>
<div class="highlight"><pre><span></span><span class="nt">python2</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">names</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python-dev</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python</span><span class="w"></span>
<span class="nt">pip</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python-pip</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python2</span><span class="w"></span>
<span class="nt">virtualenv</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">pip</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pip</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="redis-and-postgresql-states">
<h2>Redis and PostgreSQL states</h2>
<p><tt class="docutils literal">salt/redis.sls</tt> installs Redis.</p>
<div class="highlight"><pre><span></span><span class="nt">redis-server</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pkg.installed</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">salt/postgresql/init.sls</tt> installs PostgreSQL 9.1, copies <tt class="docutils literal">pg_hba.conf</tt>,
starts <tt class="docutils literal">postgresql</tt> service, creates user and database based on pillar's
data.</p>
<div class="highlight"><pre><span></span><span class="nt">postgresql</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">names</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql-9.1</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python-dev</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">libpq-dev</span><span class="w"></span>
<span class="w"> </span><span class="nt">service.running</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">watch</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/postgresql/9.1/main/pg_hba.conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span><span class="w"></span>
<span class="w"> </span><span class="nt">file.managed</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/postgresql/9.1/main/pg_hba.conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://postgresql/pg_hba.conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">644</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span><span class="w"></span>
<span class="nt">postgresql-database-setup</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">postgres_user</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">present</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'postgresql_user'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'postgresql_password'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">createdb</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">service</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span><span class="w"></span>
<span class="w"> </span><span class="nt">postgres_database</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">present</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'postgresql_db'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">encoding</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">UTF8</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">lc_ctype</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">en_US.UTF8</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">lc_collate</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">en_US.UTF8</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">template0</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'postgresql_user'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">postgres_user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql-database-setup</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">pillar/postgresql.sls</tt></p>
<div class="highlight"><pre><span></span><span class="nt">postgresql_user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="nt">postgresql_password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ghost</span><span class="w"></span>
<span class="nt">postgresql_db</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">salt/postgresql/pg_hba.conf</tt></p>
<div class="highlight"><pre><span></span><span class="c1"># This file controls: which hosts are allowed to connect, how clients</span><span class="w"></span>
<span class="c1"># are authenticated, which PostgreSQL user names they can use, which</span><span class="w"></span>
<span class="c1"># databases they can access. Records take one of these forms:</span><span class="w"></span>
<span class="c1">#</span><span class="w"></span>
<span class="c1"># local DATABASE USER METHOD [OPTIONS]</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">local jon_snow jon_snow md5</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain"># Database administrative login by Unix domain socket</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">local all postgres peer</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain"># TYPE DATABASE USER ADDRESS METHOD</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain"># "local" is for Unix domain socket connections only</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">local all all trust</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain"># IPv4 local connections</span><span class="p p-Indicator">:</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">host all all 127.0.0.1/32 md5</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="website-s-django-state">
<h2>Website's Django state</h2>
<p><tt class="docutils literal">salt/website/django.sls</tt> creates virtual environment and installs
project's dependencies there. It copies django settings, collects static
files and migrate db as well.</p>
<div class="highlight"><pre><span></span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_venv_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">directory</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">makedirs</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="nt">virtualenv</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">no_site_packages</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">distribute</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">requirements</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_requirements_path'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">no_chown</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pip</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">virtualenv</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_venv_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="nt">django settings</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_settings_path'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://website/local.py.template</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jinja</span><span class="w"></span>
<span class="nt">django-admin collectstatic</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">module</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">run</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">django.collectstatic</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">bin_env</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_venv_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">settings_module</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">messaging.settings.local</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pythonpath</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_src_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">noinput</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">virtualenv</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_venv_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">django settings</span><span class="w"></span>
<span class="nt">django-admin migrate</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">module</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">run</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">django.syncdb</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">bin_env</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_venv_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">settings_module</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">messaging.settings.local</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pythonpath</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_src_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">migrate</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">virtualenv</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="p p-Indicator">[</span><span class="s">'website_venv_dir'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">django settings</span><span class="w"></span>
</pre></div>
<p>State above actively uses pillar file <tt class="docutils literal">pillar/website.sls</tt>:</p>
<div class="highlight"><pre><span></span><span class="nt">website_venv_dir</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/venv</span><span class="w"></span>
<span class="nt">website_venv_activate_path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/venv/bin/activate</span><span class="w"></span>
<span class="nt">website_src_dir</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/abstract-internal-messaging</span><span class="w"></span>
<span class="nt">website_requirements_path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/abstract-internal-messaging/requirements_tests.txt</span><span class="w"></span>
<span class="nt">website_settings_path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/abstract-internal-messaging/messaging/settings/local.py</span><span class="w"></span>
<span class="nt">website_static_dir</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/abstract-internal-messaging/collected_static/</span><span class="w"></span>
<span class="nt">website_gunicorn_bin_path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/venv/bin/gunicorn</span><span class="w"></span>
<span class="nt">website_gunicorn_conf_path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/home/jon_snow/gunicorn.conf.py</span><span class="w"></span>
</pre></div>
<p>Here is <tt class="docutils literal">salt/website/local.py.template</tt>:</p>
<div class="highlight"><pre><span></span><span class="c1"># coding: utf-8</span>
<span class="kn">from</span> <span class="nn">.dev</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'default'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'ENGINE'</span><span class="p">:</span> <span class="s2">"django.db.backends.postgresql_psycopg2"</span><span class="p">,</span>
<span class="s1">'NAME'</span><span class="p">:</span> <span class="s2">"{{ pillar['postgresql_db'] }}"</span><span class="p">,</span>
<span class="s1">'USER'</span><span class="p">:</span> <span class="s2">"{{ pillar['postgresql_user'] }}"</span><span class="p">,</span>
<span class="s1">'PASSWORD'</span><span class="p">:</span> <span class="s2">"{{ pillar['postgresql_password'] }}"</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">SECRET_KEY</span> <span class="o">=</span> <span class="s1">'some secret key'</span>
</pre></div>
</div>
<div class="section" id="website-s-wsgi-server-state">
<h2>Website's WSGI Server state</h2>
<p><tt class="docutils literal">salt/website/wsgiserver.sls</tt> provides supervisored gunicorn running.</p>
<div class="highlight"><pre><span></span><span class="nt">supervisord conf</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/supervisor/conf.d/website_gunicorn.conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://website/supervisord.conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jinja</span><span class="w"></span>
<span class="nt">gunicorn conf</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">pillar</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="s">'website_gunicorn_conf_path'</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="p p-Indicator">}}</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://website/gunicorn.conf.py</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jon_snow</span><span class="w"></span>
<span class="nt">supervisor</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installed</span><span class="w"></span>
<span class="nt">supervisored gunicorn</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">supervisord</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">running</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">website_gunicorn</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">update</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">watch</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">supervisord conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gunicorn conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">supervisor</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">salt/website/gunicorn.conf.py</tt></p>
<div class="highlight"><pre><span></span><span class="c1"># coding: utf-8</span>
<span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="n">bind</span> <span class="o">=</span> <span class="s1">'127.0.0.1:5000'</span>
<span class="n">workers</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">cpu_count</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span>
</pre></div>
<p><tt class="docutils literal">salt/website/supervisord.conf</tt></p>
<div class="highlight"><pre><span></span><span class="p p-Indicator">[</span><span class="nv">program</span><span class="p p-Indicator">:</span><span class="nv">website_gunicorn</span><span class="p p-Indicator">]</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">command = {{ pillar['website_gunicorn_bin_path'] }} -c {{ pillar['website_gunicorn_conf_path'] }} messaging.wsgi:application</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">directory = {{ pillar['website_src_dir'] }}</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">user = vagrant</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">autostart = true</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">autorestart = true</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">redirect_stderr = True</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">stdout_logfile = /var/log/supervisor/website_gunicorn.log</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="website-s-web-server-state">
<h2>Website's Web Server state</h2>
<p><tt class="docutils literal">salt/website/webserver.sls</tt> installs latest stable Nginx, copies config
file and runs service.</p>
<div class="highlight"><pre><span></span><span class="nt">nginx</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">pkgrepo</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">deb http://nginx.org/packages/ubuntu/ precise nginx</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">key_url</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">http://nginx.org/keys/nginx_signing.key</span><span class="w"></span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">installed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkgrepo</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span><span class="w"></span>
<span class="w"> </span><span class="nt">service</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">running</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">watch</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/nginx/nginx.conf</span><span class="w"></span>
<span class="nt">/etc/nginx/nginx.conf</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">managed</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://website/nginx.conf</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">root</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">root</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">644</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">jinja</span><span class="w"></span>
</pre></div>
<p>Notable in <tt class="docutils literal">salt/website/nginx.conf</tt> is <tt class="docutils literal">sendfile off</tt>. It fixes
<a class="reference external" href="http://jeremyfelt.com/code/2013/01/08/clear-nginx-cache-in-vagrant/">trouble</a>
when Nginx runs in a virtual machine environment.</p>
<div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">worker_processes 2;</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">events {</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">worker_connections 1024;</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">}</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">http {</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">include mime.types;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">default_type application/octet-stream;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sendfile off;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">keepalive_timeout 65;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">server {</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">listen 80;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">server_name messaging-part-1;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">location /static/ {</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">alias {{ pillar['website_static_dir'] }};</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">}</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">location / {</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">proxy_pass http://localhost:5000;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">proxy_redirect off;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">proxy_set_header Host $host;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">proxy_set_header X-Real-IP $remote_addr;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">}</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain"># Redirects server error pages to the static page /50x.html</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">error_page 500 502 503 504 /50x.html;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">location = /50x.html {</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">root html;</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">}</span><span class="w"></span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">}</span><span class="w"></span>
<span class="l l-Scalar l-Scalar-Plain">}</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="top-states">
<h2>Top states</h2>
<p>There is only one thing to do β to add top files and this tree is over.</p>
<p><tt class="docutils literal">salt/top.sls</tt></p>
<div class="highlight"><pre><span></span><span class="nt">base</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="s">'*'</span><span class="p p-Indicator">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">user</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">source_code</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">redis</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">website.django</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">website.wsgiserver</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">website.webserver</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">pillar/top.sls</tt></p>
<div class="highlight"><pre><span></span><span class="nt">base</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="s">'*'</span><span class="p p-Indicator">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgresql</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">website</span><span class="w"></span>
</pre></div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>This article was mostly about states examples.
It's time to test configuration.</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span><span class="nb">cd</span> abstract-internal-messaging/vagrant
<span class="gp">$ </span>vagrant up
</pre></div>
<p>In order to reach website you can add <tt class="docutils literal">1.2.3.4 <span class="pre">messaging-part-1</span></tt> to
<tt class="docutils literal">/etc/hosts</tt>. Now it should work:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>curl -i messaging-part-1
</pre></div>
<p>Tests should be passed as well:</p>
<div class="highlight"><pre><span></span><span class="gp">$ </span>vagrant ssh
<span class="gp">$ </span><span class="nb">source</span> /home/jon_snow/venv/bin/activate
<span class="gp gp-VirtualEnv">(venv)</span><span class="gp">$ </span><span class="nb">cd</span> /home/jon_snow/abstract-internal-messaging
<span class="gp gp-VirtualEnv">(venv)</span><span class="gp">$ </span>./manage.py <span class="nb">test</span>
</pre></div>
</div>
Preparation to Python Interview2012-11-02T00:00:00+07:002012-11-02T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-11-02:/preparation-to-python-interview.html<p>I decided to collect a little more information and experience during
preparation to Python developer interview. These are some information and
links which seemed important to me. Maybe it will be helpful.</p>
<div class="section" id="how-does-it-usually-go">
<h2>How does it usually go?</h2>
<div class="section" id="what-kind-of-projects-did-you-participate-in">
<h3>What kind of projects did you participate in?</h3>
<p>What did you do at β¦</p></div></div><p>I decided to collect a little more information and experience during
preparation to Python developer interview. These are some information and
links which seemed important to me. Maybe it will be helpful.</p>
<div class="section" id="how-does-it-usually-go">
<h2>How does it usually go?</h2>
<div class="section" id="what-kind-of-projects-did-you-participate-in">
<h3>What kind of projects did you participate in?</h3>
<p>What did you do at your previous job? It is expected that you will told
the essence in simple words.</p>
</div>
<div class="section" id="tricky-question">
<h3>Tricky question</h3>
<p>It is expected that you will search for solution of task independently.
Reasonings aloud are welcomed.</p>
</div>
<div class="section" id="writing-code">
<h3>Writing code</h3>
<p>Interviewer is interested in critical analysis of code. For example,
efficiency of used data structures, algorithm's complexity evaluation.</p>
</div>
<div class="section" id="design">
<h3>Design</h3>
<p>In this step it is important to ask as much as possible about a task before
starting to look for solution.</p>
</div>
<div class="section" id="provocative-question">
<h3>Provocative question</h3>
<p>You have to stay in your lane.</p>
</div>
</div>
<div class="section" id="questions">
<h2>Questions</h2>
<div class="section" id="basic">
<h3>Basic</h3>
<p>Probably interviewer starts with basic questions. Let us see example:</p>
<div class="highlight"><pre><span></span><span class="n">funcs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
<span class="nb">print</span> <span class="n">i</span>
<span class="n">funcs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">funcs</span><span class="p">:</span>
<span class="n">f</span><span class="p">()</span>
</pre></div>
<p>I guess he examines knowledge about namespace. It will print <tt class="docutils literal">4</tt> five times.
To explain that you supposed to know about <a class="reference external" href="http://stackoverflow.com/questions/291978/short-description-of-python-scoping-rules">LEGB</a> rule. Also you should know
that variable search in enclosed scope will be done later, <strong>after call</strong> of
enclosed functions. They all get same value -- value of <tt class="docutils literal">i</tt> in last
iteration.</p>
<p>Next example prints numbers from 0 to 4:</p>
<div class="highlight"><pre><span></span><span class="n">funcs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">i</span><span class="o">=</span><span class="n">i</span><span class="p">):</span>
<span class="nb">print</span> <span class="n">i</span>
<span class="n">funcs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">funcs</span><span class="p">:</span>
<span class="n">f</span><span class="p">()</span>
</pre></div>
<p>It happens because default value is stored <strong>when enclosed function was
created</strong>.</p>
<p>There is another popular question related with default value of function.
N.B. <tt class="docutils literal">names</tt> is mutable:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">names</span><span class="o">=</span><span class="p">[]):</span>
<span class="n">names</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'some name'</span><span class="p">)</span>
</pre></div>
<p><a class="reference external" href="http://pythontutor.com">Visualizing code execution</a> and <a class="reference external" href="https://alexbers.com/python_quiz/">quiz of non-trivial features of Python</a>
will be helpful to prepare for questions which are mentioned above.</p>
</div>
<div class="section" id="what-do-you-think-about-this-code">
<h3>What do you think about this code?</h3>
<div class="highlight"><pre><span></span><span class="n">d</span> <span class="o">=</span> <span class="p">{</span><span class="mi">1</span><span class="p">:</span> <span class="s1">'a'</span><span class="p">,</span> <span class="mi">2</span><span class="p">:</span> <span class="s1">'b'</span><span class="p">,</span> <span class="mi">3</span><span class="p">:</span> <span class="s1">'c'</span><span class="p">}</span>
<span class="k">def</span> <span class="nf">safe_get</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">d</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="k">return</span> <span class="n">d</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
<span class="k">return</span> <span class="n">value</span>
<span class="nb">print</span> <span class="n">safe_get</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="s1">'blah'</span><span class="p">)</span>
</pre></div>
<p>Apparently interviewer expects these answers:</p>
<ul class="simple">
<li>it repeats functionality of <tt class="docutils literal">d.get(0, 'blah')</tt>.</li>
<li>it uses <tt class="docutils literal">if key in d.keys()</tt> instead of <tt class="docutils literal">if key in d</tt>.</li>
<li>actually it is looking for <tt class="docutils literal">key</tt> in <tt class="docutils literal">[1, 2, 3]</tt> (for Python 2).
Therefore O(n) is worse than O(1) for dictionary lookup.</li>
</ul>
<p>It would be appropriate to read about <a class="reference external" href="http://habrahabr.ru/post/128457/">data structures</a> and their
<a class="reference external" href="http://wiki.python.org/moin/TimeComplexity">time complexity</a>, about <a class="reference external" href="http://docs.python.org/3/library/collections.html">collections</a>, <a class="reference external" href="http://docs.python.org/3/library/itertools">itertools</a> and <a class="reference external" href="http://wiki.python.org/moin/HowTo/Sorting/">sorting</a>.</p>
</div>
<div class="section" id="advanced-knowledge">
<h3>Advanced knowledge</h3>
<p>These articles are for complete picture of Python:</p>
<ul class="simple">
<li><a class="reference external" href="http://www.cafepy.com/article/python_types_and_objects/">Python Types and Objects</a></li>
<li><a class="reference external" href="http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python/6581949#6581949">What is a metaclass in Python?</a></li>
<li><a class="reference external" href="http://www.cafepy.com/article/python_attributes_and_methods/">Python Attributes and Methods</a>, <a class="reference external" href="http://habrahabr.ru/post/137415/">ΠΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»ΡΡΠΊΠΈΠ΅ Π°ΡΡΠΈΠ±ΡΡΡ Π² Python</a></li>
</ul>
</div>
<div class="section" id="misc">
<h3>Misc</h3>
<p>Questions:</p>
<ul class="simple">
<li><a class="reference external" href="http://pragmaticstartup.wordpress.com/2012/08/25/what-every-new-pythondjango-web-developer-should-know-in-3-months/">What Every New Python/Django Web Developer Should Know in 3 Months</a></li>
<li><a class="reference external" href="https://groups.google.com/d/topic/comp.lang.python/rhW_rIYY5HM/discussion">Python Interview Questions</a></li>
<li><a class="reference external" href="http://www.techinterviews.com/python-interview-questions-and-answers">Python interview questions and answers</a></li>
<li><a class="reference external" href="http://habrahabr.ru/qa/5783/">Π§ΡΠΎ ΡΠΏΡΠ°ΡΠΈΠ²Π°ΡΡ Π½Π° ΡΠΎΠ±Π΅ΡΠ΅Π΄ΠΎΠ²Π°Π½ΠΈΠΈ Π² Π―Π½Π΄Π΅ΠΊΡ?</a></li>
<li><a class="reference external" href="http://pyobject.ru/blog/2010/02/04/python-quiz/">ΠΠΎΠΏΡΠΎΡΡ ΠΈ Π·Π°Π΄Π°Π½ΠΈΡ ΠΏΠΎ Python</a></li>
</ul>
<p>Useful articles:</p>
<ul class="simple">
<li><a class="reference external" href="http://programmers.stackexchange.com/questions/43409/dealing-with-engineers-that-frequently-leave-their-jobs">Dealing with engineers that frequently leave their jobs</a></li>
<li><a class="reference external" href="http://steve-yegge.blogspot.ch/2008/03/get-that-job-at-google.html">Get that job at Google</a></li>
<li><a class="reference external" href="http://stackoverflow.com/questions/101268/hidden-features-of-python">Hidden features of Python</a></li>
</ul>
</div>
</div>
Django TODO: ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π²ΠΎ Π²ΡΠ΅ΠΌΡ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ2012-06-29T13:00:00+07:002012-06-29T13:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-29:/django-todo-testing-during-construction.html<p>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅, Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΠΌΠΎΠ΅ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ°ΠΌΠΈ -- ΠΎΠ΄ΠΈΠ½ ΠΈΠ· Π²Π°ΠΆΠ½Π΅ΠΉΡΠΈΡ
ΡΠ»Π΅ΠΌΠ΅Π½ΡΠΎΠ² ΠΏΠΎΠ»Π½ΠΎΠΉ
ΡΡΡΠ°ΡΠ΅Π³ΠΈΠΈ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ.</p>
<p>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΌΠΎΠΆΠ΅Ρ ΡΠΊΠ°Π·Π°ΡΡ ΡΠΎΠ»ΡΠΊΠΎ Π½Π° ΠΎΡΠ΄Π΅Π»ΡΠ½ΡΠ΅ Π΄Π΅ΡΠ΅ΠΊΡΠ½ΡΠ΅ ΠΎΠ±Π»Π°ΡΡΠΈ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ --
ΠΎΠ½ΠΎ Π½Π΅ ΡΠ΄Π΅Π»Π°Π΅Ρ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ ΡΠ΄ΠΎΠ±Π½Π΅Π΅ Π² ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠΈ, Π±ΠΎΠ»Π΅Π΅ Π±ΡΡΡΡΠΎΠΉ, ΠΊΠΎΠΌΠΏΠ°ΠΊΡΠ½ΠΎΠΉ,
ΡΠ΄ΠΎΠ±ΠΎΡΠΈΡΠ°Π΅ΠΌΠΎΠΉ ΠΈΠ»ΠΈ ΡΠ°ΡΡΠΈΡΡΠ΅ΠΌΠΎΠΉ.</p>
<p>Π¦Π΅Π»Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΡΠΎΡΠΈΠ²ΠΎΠΏΠΎΠ»ΠΎΠΆΠ½Π° ΡΠ΅Π»ΡΠΌ Π΄ΡΡΠ³ΠΈΡ
ΡΡΠ°ΠΏΠΎΠ² ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ. ΠΠ³ΠΎ ΡΠ΅Π»ΡΡ
ΡΠ²Π»ΡΠ΅ΡΡΡ Π½Π°Ρ
ΠΎΠΆΠ΄Π΅Π½ΠΈΠ΅ ΠΎΡΠΈΠ±ΠΎΠΊ. Π£ΡΠΏΠ΅ΡΠ½ΡΠΌ ΡΡΠΈΡΠ°Π΅ΡΡΡ ΡΠ΅ΡΡ, Π½Π°ΡΡΡΠ°ΡΡΠΈΠΉ ΡΠ°Π±ΠΎΡΡ ΠΠ β¦</p><p>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅, Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΠΌΠΎΠ΅ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ°ΠΌΠΈ -- ΠΎΠ΄ΠΈΠ½ ΠΈΠ· Π²Π°ΠΆΠ½Π΅ΠΉΡΠΈΡ
ΡΠ»Π΅ΠΌΠ΅Π½ΡΠΎΠ² ΠΏΠΎΠ»Π½ΠΎΠΉ
ΡΡΡΠ°ΡΠ΅Π³ΠΈΠΈ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ.</p>
<p>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΌΠΎΠΆΠ΅Ρ ΡΠΊΠ°Π·Π°ΡΡ ΡΠΎΠ»ΡΠΊΠΎ Π½Π° ΠΎΡΠ΄Π΅Π»ΡΠ½ΡΠ΅ Π΄Π΅ΡΠ΅ΠΊΡΠ½ΡΠ΅ ΠΎΠ±Π»Π°ΡΡΠΈ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ --
ΠΎΠ½ΠΎ Π½Π΅ ΡΠ΄Π΅Π»Π°Π΅Ρ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ ΡΠ΄ΠΎΠ±Π½Π΅Π΅ Π² ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠΈ, Π±ΠΎΠ»Π΅Π΅ Π±ΡΡΡΡΠΎΠΉ, ΠΊΠΎΠΌΠΏΠ°ΠΊΡΠ½ΠΎΠΉ,
ΡΠ΄ΠΎΠ±ΠΎΡΠΈΡΠ°Π΅ΠΌΠΎΠΉ ΠΈΠ»ΠΈ ΡΠ°ΡΡΠΈΡΡΠ΅ΠΌΠΎΠΉ.</p>
<p>Π¦Π΅Π»Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΡΠΎΡΠΈΠ²ΠΎΠΏΠΎΠ»ΠΎΠΆΠ½Π° ΡΠ΅Π»ΡΠΌ Π΄ΡΡΠ³ΠΈΡ
ΡΡΠ°ΠΏΠΎΠ² ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ. ΠΠ³ΠΎ ΡΠ΅Π»ΡΡ
ΡΠ²Π»ΡΠ΅ΡΡΡ Π½Π°Ρ
ΠΎΠΆΠ΄Π΅Π½ΠΈΠ΅ ΠΎΡΠΈΠ±ΠΎΠΊ. Π£ΡΠΏΠ΅ΡΠ½ΡΠΌ ΡΡΠΈΡΠ°Π΅ΡΡΡ ΡΠ΅ΡΡ, Π½Π°ΡΡΡΠ°ΡΡΠΈΠΉ ΡΠ°Π±ΠΎΡΡ ΠΠ. ΠΡΠ΅
ΠΎΡΡΠ°Π»ΡΠ½ΡΠ΅ ΡΡΠ°ΠΏΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ Π½Π°ΠΏΡΠ°Π²Π»Π΅Π½Ρ Π½Π° ΠΏΡΠ΅Π΄ΠΎΡΠ²ΡΠ°ΡΠ΅Π½ΠΈΠ΅ ΠΎΡΠΈΠ±ΠΎΠΊ ΠΈ Π½Π΅Π΄ΠΎΠΏΡΡΠ΅Π½ΠΈΠ΅
Π½Π°ΡΡΡΠ΅Π½ΠΈΡ ΡΠ°Π±ΠΎΡΡ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ <a class="footnote-reference" href="#mcconnell" id="footnote-reference-1">[1]</a>.</p>
<div class="section" id="section-1">
<h2>ΠΠΈΠ΄Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ</h2>
<p>ΠΠΈΠ΄Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ, Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΠΌΡΠ΅ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠΎΠΌ: Π±Π»ΠΎΡΠ½ΠΎΠ΅ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅,
ΠΈΠ½ΡΠ΅Π³ΡΠ°ΡΠΈΠΎΠ½Π½ΠΎΠ΅ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅.</p>
<p>ΠΠ»ΠΎΡΠ½ΡΠΌ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ΠΌ Π½Π°Π·ΡΠ²Π°ΡΡ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΏΠΎΠ»Π½ΠΎΠ³ΠΎ ΠΊΠ»Π°ΡΡΠ°, ΠΌΠ΅ΡΠΎΠ΄Π° ΠΈΠ»ΠΈ
Π½Π΅Π±ΠΎΠ»ΡΡΠΎΠ³ΠΎ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ, Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΠΌΠΎΠ΅ ΠΎΡΠ΄Π΅Π»ΡΠ½ΠΎ ΠΎΡ ΠΏΡΠΎΡΠΈΡ
ΡΠ°ΡΡΠ΅ΠΉ ΡΠΈΡΡΠ΅ΠΌΡ. ΠΠ°Π½Π½ΡΠΉ
ΡΠΈΠΏ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΡΡΡΡΠΊΡΡΡΠΈΡΠΎΠ²Π°ΡΡ ΠΊΠΎΠ΄.</p>
<p>ΠΠ½ΡΠ΅Π³ΡΠ°ΡΠΈΠΎΠ½Π½ΠΎΠ΅ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ -- ΡΡΠΎ ΡΠΎΠ²ΠΌΠ΅ΡΡΠ½ΠΎΠ΅ Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΠ΅ Π΄Π²ΡΡ
ΠΈΠ»ΠΈ Π±ΠΎΠ»Π΅Π΅
ΠΊΠ»Π°ΡΡΠΎΠ², ΠΏΠ°ΠΊΠ΅ΡΠΎΠ², ΠΊΠΎΠΌΠΏΠΎΠ½Π΅Π½ΡΠΎΠ² ΠΈΠ»ΠΈ ΠΏΠΎΠ΄ΡΠΈΡΡΠ΅ΠΌ. ΠΡΠΎΡ Π²ΠΈΠ΄ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π½Π°ΡΠΈΠ½Π°ΡΡ
Π²Π²ΠΎΠ΄ΠΈΡΡ, ΠΊΠ°ΠΊ ΡΠΎΠ»ΡΠΊΠΎ ΡΠΎΠ·Π΄Π°Π½Ρ Π΄Π²Π° ΠΊΠ»Π°ΡΡΠ°, ΠΊΠΎΡΠΎΡΡΠ΅ ΠΌΠΎΠΆΠ½ΠΎ ΠΏΡΠΎΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ. Π’Π°ΠΊΠΎΠ΅
ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΠΈΡΡ ΡΠ΅Π³ΡΠ΅ΡΡΠΈΡ, Π½ΠΎ ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΌΠ΅Π΄Π»Π΅Π½Π½ΡΠΌ ΠΈ ΠΌΠ΅Π½Π΅Π΅
ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠ²Π½ΡΠΌ, ΡΠ΅ΠΌ Π±Π»ΠΎΡΠ½ΠΎΠ΅ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅.</p>
<p>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΡΠ°Π·Π΄Π΅Π»ΡΡΡ Π½Π° Π΄Π²Π΅ ΠΎΠ±ΡΠΈΡΠ½ΡΠ΅ ΠΊΠ°ΡΠ΅Π³ΠΎΡΠΈΠΈ: "ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΌΠ΅ΡΠΎΠ΄ΠΎΠΌ
ΡΠ΅ΡΠ½ΠΎΠ³ΠΎ ΡΡΠΈΠΊΠ°" ΠΈ "ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΌΠ΅ΡΠΎΠ΄ΠΎΠΌ Π±Π΅Π»ΠΎΠ³ΠΎ (ΠΏΡΠΎΠ·ΡΠ°ΡΠ½ΠΎΠ³ΠΎ) ΡΡΠΈΠΊΠ°". Π ΠΏΠ΅ΡΠ²ΠΎΠΌ
ΡΠ»ΡΡΠ°Π΅ ΡΠ΅ΡΡΠΈΡΠΎΡΠΈΠΊ Π½Π΅ Π²Π»Π°Π΄Π΅Π΅Ρ ΡΠ²Π΅Π΄Π΅Π½ΠΈΡΠΌΠΈ ΠΎ Π²Π½ΡΡΡΠ΅Π½Π½Π΅ΠΉ ΡΠ°Π±ΠΎΡΠ΅ ΡΠ΅ΡΡΠΈΡΡΠ΅ΠΌΠΎΠ³ΠΎ
ΡΠ»Π΅ΠΌΠ΅Π½ΡΠ°. ΠΡΠΈ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠΈ ΠΌΠ΅ΡΠΎΠ΄ΠΎΠΌ Π±Π΅Π»ΠΎΠ³ΠΎ ΡΡΠΈΠΊΠ° Π²Π½ΡΡΡΠ΅Π½Π½ΡΡ ΡΠ΅Π°Π»ΠΈΠ·Π°ΡΠΈΡ
ΡΠ΅ΡΡΠΈΡΡΠ΅ΠΌΠΎΠ³ΠΎ ΡΠ»Π΅ΠΌΠ΅Π½ΡΠ° ΡΠ΅ΡΡΠΈΡΠΎΠ²ΡΠΈΠΊΡ ΠΈΠ·Π²Π΅ΡΡΠ½Π°. Π’Π΅ΡΡΠΈΡΡΡ ΡΠΎΠ±ΡΡΠ²Π΅Π½Π½ΡΠΉ ΠΊΠΎΠ΄,
ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅Ρ ΠΈΠΌΠ΅Π½Π½ΠΎ ΡΡΠΎΡ Π²ΠΈΠ΄ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ.</p>
<p>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΡΡΠ΅Π±ΡΠ΅Ρ, ΡΡΠΎΠ±Ρ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΡ ΡΠ°ΡΡΡΠΈΡΡΠ²Π°Π» Π½Π°ΠΉΡΠΈ ΠΎΡΠΈΠ±ΠΊΠΈ Π² ΡΠ²ΠΎΠ΅ΠΌ ΠΊΠΎΠ΄Π΅
<a class="footnote-reference" href="#mcconnell" id="footnote-reference-2">[1]</a>.</p>
</div>
<div class="section" id="django">
<h2>Π Π΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΠΈ ΠΏΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ</h2>
<p>ΠΠ°ΡΠ» ΠΠ°ΠΉΠ΅Ρ Π΄Π°Π» ΡΠ»Π΅Π΄ΡΡΡΠΈΠ΅ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΠΈ ΠΏΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ
<a class="footnote-reference" href="#meyer" id="footnote-reference-3">[2]</a>:</p>
<ul class="simple">
<li>Π΄Π»Ρ ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΠΉ ΡΠ»Π΅Π΄ΡΠ΅Ρ ΠΏΠΈΡΠ°ΡΡ ΠΈΠ½ΡΠ΅Π³ΡΠ°ΡΠΈΠΎΠ½Π½ΡΠ΅ ΡΠ΅ΡΡΡ;</li>
<li>Π΄Π»Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Ajax ΠΈ Π΄ΡΡΠ³ΠΈΡ
JavaScript Π²Π·Π°ΠΈΠΌΠΎΠ΄Π΅ΠΉΡΡΠ²ΠΈΠΉ Π½ΡΠΆΠ½ΠΎ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ
<em>Selenium</em>, ΠΊΠΎΡΠΎΡΡΠΉ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·ΠΈΡΠΎΠ²Π°ΡΡ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π²Π΅Π±-ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π²
Π±ΡΠ°ΡΠ·Π΅ΡΠ΅;</li>
<li>Π΄Π»Ρ Π²ΡΠ΅Ρ
ΠΎΡΡΠ°Π»ΡΠ½ΡΡ
ΡΠ»ΡΡΠ°Π΅Π² Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ Π±Π»ΠΎΡΠ½ΠΎΠ΅ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅.</li>
</ul>
<p>ΠΠ½ ΡΠ°ΠΊΠΆΠ΅ Π²ΡΡΡΡΠΏΠΈΠ» ΠΏΡΠΎΡΠΈΠ² ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΡ <em>fixtures</em> (Π½Π°Π±ΠΎΡ Π΄Π°Π½Π½ΡΡ
, ΠΊΠΎΡΠΎΡΡΠ΅
Django ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅Ρ Π΄Π»Ρ ΠΈΠΌΠΏΠΎΡΡΠ° Π² ΠΠ), Π°ΡΠ³ΡΠΌΠ΅Π½ΡΠΈΡΡΡ ΡΠ΅ΠΌ, ΡΡΠΎ ΠΎΠ½ΠΈ ΠΌΠ΅Π΄Π»Π΅Π½Π½ΠΎ
Π·Π°Π³ΡΡΠΆΠ°ΡΡΡΡ, ΠΈΡ
ΡΡΡΠ΄Π½ΠΎ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°ΡΡ ΠΈ ΠΎΠ±Π½ΠΎΠ²Π»ΡΡΡ. Π’Π°ΠΊΠΆΠ΅ ΠΏΡΠΈΠΌΠ΅Π½Π΅Π½ΠΈΠ΅ <em>fixtures</em>
ΠΏΡΠΈΠ²ΠΎΠ΄ΠΈΡ ΠΊ ΡΠΎΡΡΡ Π²Π·Π°ΠΈΠΌΠΎΠ·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠΈ ΡΠ΅ΡΡΠΎΠ². ΠΠ»Ρ Π·Π°ΠΌΠ΅Π½Ρ <em>fixtures</em> Π±ΡΠ»
ΡΠ°Π·ΡΠ°Π±ΠΎΡΠ°Π½ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ <em>factory_boy</em>, ΠΊΠΎΡΠΎΡΡΠΉ ΠΈΠΌΠ΅Π΅Ρ ΡΠ»Π΅Π΄ΡΡΡΠΈΠ΅ ΠΏΡΠ΅ΠΈΠΌΡΡΠ΅ΡΡΠ²Π°:
Π±Π»ΠΈΠ·ΠΎΡΡΡ ΡΠ΅ΡΡΠΎΠ²ΡΡ
Π΄Π°Π½Π½ΡΡ
ΠΊ ΡΠ΅ΡΡΠΎΠ²ΠΎΠΌΡ ΠΊΠΎΠ΄Ρ, Π½Π΅ ΡΡΠ΅Π±ΡΠ΅Ρ ΠΎΡ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π²Π²Π΅Π΄Π΅Π½ΠΈΠ΅
ΠΈΠ·Π±ΡΡΠΎΡΠ½ΡΡ
Π΄Π°Π½Π½ΡΡ
, ΠΏΡΠΎΡΡ Π² ΠΎΠ±ΡΠ»ΡΠΆΠΈΠ²Π°Π½ΠΈΠΈ.</p>
<div class="section" id="section-2">
<h3>Π§ΡΠΎ ΠΈΠΌΠ΅Π½Π½ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ?</h3>
<p>ΠΡΠ²Π΅Ρ Π½Π° Π²ΠΎΠΏΡΠΎΡ <em>"Π§ΡΠΎ ΠΈΠΌΠ΅Π½Π½ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ Π² Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΈ?"</em> Π΄Π°Π» ΠΡΠ½ΠΈΡΠ»
ΠΠΈΠ½Π΄ΡΠ»ΠΈ <a class="footnote-reference" href="#lindsley" id="footnote-reference-4">[3]</a>:</p>
<ul class="simple">
<li>Π½Π΅ ΡΠ»Π΅Π΄ΡΠ΅Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ ΡΡΠ°Π½Π΄Π°ΡΡΠ½ΡΠ΅ ΡΡΠ½ΠΊΡΠΈΠΈ Python ΠΈ Django;</li>
<li>Π΅ΡΠ»ΠΈ ΠΌΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»ΡΡΠΊΠΈΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ, ΠΈΡ
Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ
ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ;</li>
<li>Π½ΡΠΆΠ½ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»ΡΡΠΊΠΈΠ΅ ΡΠΎΡΠΌΡ, ΡΠ°Π±Π»ΠΎΠ½Π½ΡΠ΅ ΡΠ΅Π³ΠΈ, ΠΊΠΎΠ½ΡΠ΅ΠΊΡΡΠ½ΡΠ΅
ΠΏΡΠΎΡΠ΅ΡΡΠΎΡΡ, ΠΊΠΎΠΌΠ°Π½Π΄Ρ ΡΠΏΡΠ°Π²Π»Π΅Π½ΠΈΡ;</li>
<li>ΡΠ»Π΅Π΄ΡΠ΅Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ Π±ΠΈΠ·Π½Π΅Ρ Π»ΠΎΠ³ΠΈΠΊΡ Π² ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΡΡ
.</li>
</ul>
</div>
</div>
<div class="section" id="section-3">
<h2>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ Π΄Π°Π½Π½ΡΡ
</h2>
<p>ΠΠ»Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Django ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡΡ Π±Π»ΠΎΡΠ½ΠΎΠ΅ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅. Π ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅
ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½ΡΠ° Π±Π»ΠΎΡΠ½ΠΎΠ³ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π°Π²ΡΠΎΡ Π²ΡΠ±ΡΠ°Π» Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΡ <em>unittest</em>, ΡΠ°ΠΊ ΠΊΠ°ΠΊ
ΠΎΠ½Π° Π²Ρ
ΠΎΠ΄ΠΈΡ Π² ΡΡΠ°Π½Π΄Π°ΡΡΠ½ΡΡ Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΡ Python. Π’Π΅ΡΡΡ, Π½Π°ΠΏΠΈΡΠ°Π½Π½ΡΠ΅ Π½Π° <em>unittest</em>
ΡΠ°Π±ΠΎΡΠ°ΡΡ Π±ΡΡΡΡΠ΅Π΅ ΠΏΡΠΈ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠΈ Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ <a class="footnote-reference" href="#lindsley" id="footnote-reference-5">[3]</a>.</p>
<p>Π ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅ ΠΏΡΠΈΠΌΠ΅ΡΠ° ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΡΠ΅ΡΡΠΎΠ² ΠΏΡΠΈΠ²ΠΎΠ΄ΡΡΡΡ ΠΌΠ΅ΡΠΎΠ΄Ρ
<em>days_quantity_after_deadline</em> (ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΡ
Π΄Π½Π΅ΠΉ
Π·Π°Π΄Π°ΡΠΈ) ΠΈ <em>start_date</em> (ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ).</p>
<div class="section" id="section-4">
<h3>ΠΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΡ
Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</h3>
<p>ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²Π° ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΡ
Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΠ»Ρ ΡΡΠΎΠ³ΠΎ ΡΠ°ΡΡΠΌΠΎΡΡΠΈΠΌ Π²ΡΠ΅
Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΠ΅ ΡΠ»ΡΡΠ°ΠΈ Π½Π°ΡΡΡΠΏΠ»Π΅Π½ΠΈΡ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°:</p>
<ul class="simple">
<li>ΠΏΡΠΎΡΡΠΎΡΠ΅Π½ Π΄Π΅Π΄Π»Π°ΠΉΠ½ Ρ ΠΎΠΆΠΈΠ΄Π°ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ ΠΈΠ·-Π·Π° ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ
Π·Π°Π΄Π°ΡΠ° ΠΏΡΠ΅Π²ΡΡΠΈΠ»Π° ΡΠ²ΠΎΠΉ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li>ΡΠ°Π±ΠΎΡΠ°ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΏΡΠ΅Π²ΡΡΠΈΠ»Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½;</li>
<li>Π·Π°Π΄Π°ΡΠ° Π²ΡΠΏΠΎΠ»Π½Π΅Π½Π° Ρ ΠΏΡΠ΅Π²ΡΡΠ΅Π½ΠΈΠ΅ΠΌ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°;</li>
<li>ΠΏΡΠΎΡΡΠΎΡΠ΅Π½ Π΄Π΅Π΄Π»Π°ΠΉΠ½ Ρ ΠΎΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½Π½ΠΎΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΠ»Π°Π΄Π΅Π»Π΅Ρ ΡΠ΅ΠΏΠΎΡΠΊΠΈ Π½Π΅ ΡΠ΅ΡΠΈΠ» ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ
ΠΎΡΡΠ°Π½ΠΎΠ²ΠΊΠΈ Π·Π°Π΄Π°ΡΠΈ.</li>
</ul>
<p>ΠΠ· Π²ΡΡΠ΅ΠΏΠ΅ΡΠ΅ΡΠΈΡΠ»Π΅Π½Π½ΡΡ
ΡΠ»ΡΡΠ°Π΅Π² ΡΠ»Π΅Π΄ΡΠ΅Ρ Π΄Π²Π° ΠΏΡΠ°Π²ΠΈΠ»Π° ΡΠ°ΡΡΠ΅ΡΠ° ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²Π°
ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΡ
Π΄Π½Π΅ΠΉ:</p>
<ul class="simple">
<li>Π΅ΡΠ»ΠΈ ΡΡΠ°ΡΡΡ <strong>DONE</strong> ΠΈ Π΄Π°ΡΠ° ΠΎΠΊΠΎΠ½ΡΠ°Π½ΠΈΡ Π·Π°Π΄Π°ΡΠΈ Π±ΠΎΠ»ΡΡΠ΅ ΠΈΠ»ΠΈ ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ, ΡΠΎ
ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ ΡΠ°Π²Π½ΠΎ ΡΠ°Π·Π½ΠΎΡΡΠΈ Π΄Π°ΡΡ ΠΎΠΊΠΎΠ½ΡΠ°Π½ΠΈΡ ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°, ΠΏΠ»ΡΡ ΠΎΠ΄ΠΈΠ½ Π΄Π΅Π½Ρ
(Π΄Π΅Π½Ρ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°);</li>
<li>Π΅ΡΠ»ΠΈ ΡΡΠ°ΡΡΡ <strong>WAIT/WORK/STOP</strong> ΠΈ ΡΠ΅ΠΊΡΡΠ°Ρ Π΄Π°ΡΠ° Π±ΠΎΠ»ΡΡΠ΅ ΠΈΠ»ΠΈ ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ, ΡΠΎ
ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ ΡΠ°Π²Π½ΠΎ ΡΠ°Π·Π½ΠΎΡΡΠΈ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π΄Π°ΡΡ ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°, ΠΏΠ»ΡΡ ΠΎΠ΄ΠΈΠ½ Π΄Π΅Π½Ρ (Π΄Π΅Π½Ρ
Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°).</li>
</ul>
<p>ΠΠΏΡΠ΅Π΄Π΅Π»ΠΈΠ² Π²ΡΠ΅ Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΠ΅ ΡΠ»ΡΡΠ°ΠΈ Π½Π°ΡΡΡΠΏΠ»Π΅Π½ΠΈΡ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π° ΠΌΠΎΠΆΠ½ΠΎ ΡΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°ΡΡ
"ΡΠΈΡΡΡΠ΅ ΡΠ΅ΡΡΡ" Π΄Π»Ρ ΠΌΠ΅ΡΠΎΠ΄Π° <em>days_quantity_after_deadline</em>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DeadlineDaysTest</span><span class="p">(</span><span class="n">TaskTest</span><span class="p">):</span>
<span class="sd">"""Π’Π΅ΡΡΠΈΡΡΠ΅Ρ ΠΎΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²Π° ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΡ
Π΄Π½Π΅ΠΉ."""</span>
<span class="k">def</span> <span class="nf">task_wait_overdue</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""ΠΡΠΎΡΡΠΎΡΠ΅Π½ Π΄Π΅Π΄Π»Π°ΠΉΠ½ Ρ ΠΎΠΆΠΈΠ΄Π°ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ ΠΈΠ·-Π·Π° ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ.</span>
<span class="sd"> ΠΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΏΡΠ΅Π²ΡΡΠΈΠ»Π° ΡΠ²ΠΎΠΉ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">task_work_overdue</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Π Π°Π±ΠΎΡΠ°ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΏΡΠ΅Π²ΡΡΠΈΠ»Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½."""</span>
<span class="k">def</span> <span class="nf">task_done_overdue</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""ΠΠ°Π΄Π°ΡΠ° Π²ΡΠΏΠΎΠ»Π½Π΅Π½Π° Ρ ΠΏΡΠ΅Π²ΡΡΠ΅Π½ΠΈΠ΅ΠΌ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°."""</span>
<span class="k">def</span> <span class="nf">task_stop_overdue</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""ΠΡΠΎΡΡΠΎΡΠ΅Π½ Π΄Π΅Π΄Π»Π°ΠΉΠ½ Ρ ΠΎΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½Π½ΠΎΠΉ Π·Π°Π΄Π°ΡΠΈ.</span>
<span class="sd"> ΠΠ»Π°Π΄Π΅Π»Π΅Ρ ΡΠ΅ΠΏΠΎΡΠΊΠΈ Π½Π΅ ΡΠ΅ΡΠΈΠ» ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ ΠΎΡΡΠ°Π½ΠΎΠ²ΠΊΠΈ Π·Π°Π΄Π°ΡΠΈ.</span>
<span class="sd"> """</span>
</pre></div>
<p>ΠΠΎΠ΄ "ΡΠΈΡΡΡΠΌΠΈ ΡΠ΅ΡΡΠ°ΠΌΠΈ" ΠΏΠΎΠ΄ΡΠ°Π·ΡΠΌΠ΅Π²Π°ΡΡΡΡ ΡΠ΅ΡΡΡ, ΠΊΠΎΡΠΎΡΡΠ΅ ΠΏΡΠΎΠ²Π΅ΡΡΡΡ ΡΠ°Π±ΠΎΡΠ°Π΅Ρ Π»ΠΈ
ΠΊΠΎΠ΄, Π° Π½Π΅ ΠΏΡΡΠ°ΡΡΡΡ Π½Π°ΡΡΡΠΈΡΡ Π΅Π³ΠΎ ΡΠ°Π±ΠΎΡΡ Π²ΡΠ΅Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΠΌΠΈ ΡΠΏΠΎΡΠΎΠ±Π°ΠΌΠΈ ("Π³ΡΡΠ·Π½ΡΠ΅
ΡΠ΅ΡΡΡ"). Π ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΡΡ
ΡΠΎ Π·ΡΠ΅Π»ΡΠΌ ΠΏΡΠΎΡΠ΅ΡΡΠΎΠΌ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π½Π° ΠΊΠ°ΠΆΠ΄ΡΠΉ "ΡΠΈΡΡΡΠΉ
ΡΠ΅ΡΡ" ΠΎΠ±ΡΡΠ½ΠΎ ΠΏΡΠΈΡ
ΠΎΠ΄ΡΡΡΡ ΠΏΡΡΡ "Π³ΡΡΠ·Π½ΡΡ
" <a class="footnote-reference" href="#mcconnell" id="footnote-reference-6">[1]</a>.</p>
</div>
<div class="section" id="section-5">
<h3>ΠΠ°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ</h3>
<p>Π Π°ΡΡΠΌΠΎΡΡΠΈΠΌ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΡΠ΅ΡΡΠΎΠ² Π΄Π»Ρ Π±ΠΎΠ»Π΅Π΅ ΡΠ»ΠΎΠΆΠ½ΠΎΠ³ΠΎ ΠΌΠ΅ΡΠΎΠ΄Π° -- ΠΌΠ΅ΡΠΎΠ΄Π°, ΠΊΠΎΡΠΎΡΡΠΉ
ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ.</p>
<p>Π§ΡΠΎΠ±Ρ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΠΈΡΡ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ ΡΠ°ΡΡΠΌΠΎΡΡΠ΅ΡΡ Π²ΡΠ΅
Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΠ΅ ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΠΈ ΠΏΡΠ΅Π΄ΡΠ΅ΡΡΠ²ΡΡΡΠΈΡ
Π·Π°Π΄Π°Ρ ΠΏΠΎ ΡΡΠ°ΡΡΡΠ°ΠΌ.</p>
<div class="section" id="wait">
<h4>Π‘ΡΠ°ΡΡΡ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ WAIT</h4>
<p>ΠΠ°Π΄Π°ΡΠ° ΡΡΠΎΠΈΡ ΠΏΠ΅ΡΠ²ΠΎΠΉ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅, Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ Π½Π΅ Π½Π°ΡΡΡΠΏΠΈΠ»Π°.
Π ΡΡΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΏΠΎΡΠΊΠΈ.</p>
<p>ΠΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΈΠΌΠ΅Π΅Ρ ΡΡΠ°ΡΡΡ <em>WAIT</em>, ΠΊ ΡΠΎΠΌΡ ΠΆΠ΅:</p>
<ul class="simple">
<li><em>Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</em> Π½Π΅ Π½Π°ΡΡΡΠΏΠΈΠ»Π°. Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π°
ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li>Π½Π°ΡΡΡΠΏΠΈΠ»Π° <em>Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</em>, Π½ΠΎ Π΅ΡΠ΅ Π½Π΅ Π½Π°ΡΡΡΠΏΠΈΠ» <em>Π΄Π΅Π΄Π»Π°ΠΉΠ½
ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</em>. Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠ°Π²Π½Π°
Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li>Π½Π°ΡΡΡΠΏΠΈΠ» <em>Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</em>. Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΊΡΡΠ΅ΠΉ
Π·Π°Π΄Π°ΡΠΈ Π½Π΅ ΠΏΡΠΎΠ³Π½ΠΎΠ·ΠΈΡΡΠ΅ΠΌΠ°.</li>
</ul>
<p>ΠΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΈΠΌΠ΅Π΅Ρ ΡΡΠ°ΡΡΡ <em>WORK</em> ΠΈΠ»ΠΈ <em>STOP</em>, ΠΊ ΡΠΎΠΌΡ ΠΆΠ΅:</p>
<ul class="simple">
<li>Π½Π΅ Π½Π°ΡΡΡΠΏΠΈΠ» <em>Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</em>. Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΊΡΡΠ΅ΠΉ
Π·Π°Π΄Π°ΡΠΈ ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li>Π½Π°ΡΡΡΠΏΠΈΠ» <em>Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ</em>. Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΊΡΡΠ΅ΠΉ
Π·Π°Π΄Π°ΡΠΈ Π½Π΅ ΠΏΡΠΎΠ³Π½ΠΎΠ·ΠΈΡΡΠ΅ΠΌΠ°.</li>
</ul>
</div>
<div class="section" id="work-done-stop">
<h4>Π‘ΡΠ°ΡΡΡ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ WORK ΠΈΠ»ΠΈ DONE ΠΈΠ»ΠΈ STOP</h4>
<p>ΠΠ°Π΄Π°ΡΠ° ΡΡΠΎΠΈΡ ΠΏΠ΅ΡΠ²ΠΎΠΉ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅, Π½Π°ΡΡΡΠΏΠΈΠ»Π° Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ.
Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΏΠΎΡΠΊΠΈ.</p>
<p>ΠΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΈΠΌΠ΅Π΅Ρ ΡΡΠ°ΡΡΡ <em>DONE</em>. Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° Π·Π°Π΄Π°ΡΠΈ
ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ.</p>
</div>
<div class="section" id="section-6">
<h4>ΠΡΠ°Π²ΠΈΠ»ΠΎ ΠΎΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ Π΄Π°ΡΡ</h4>
<p>ΠΡΠΎΠ°Π½Π°Π»ΠΈΠ·ΠΈΡΠΎΠ²Π°Π² Π²ΡΡΠ΅ΠΏΠ΅ΡΠ΅ΡΠΈΡΠ»Π΅Π½Π½ΡΠ΅ ΡΠ»ΡΡΠ°ΠΈ Π°Π²ΡΠΎΡ ΡΡΠΎΡΠΌΡΠ»ΠΈΡΠΎΠ²Π°Π» ΠΏΡΠ°Π²ΠΈΠ»Π°
ΠΎΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ:</p>
<ul class="simple">
<li>Π΄Π»Ρ ΠΏΠ΅ΡΠ²ΠΎΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ;</li>
<li>Π΄Π»Ρ ΡΡΠ°ΡΡΡΠ° <em>WAIT</em> ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΡΠ»ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½,
Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° Π·Π°Π΄Π°ΡΠΈ Π½Π΅ ΠΏΡΠΎΠ³Π½ΠΎΠ·ΠΈΡΡΠ΅ΠΌΠ°;</li>
<li>Π΄Π»Ρ ΡΡΠ°ΡΡΡΠΎΠ² <em>WORK</em>, <em>DONE</em>, <em>STOP</em> ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ ΠΎΠΊΠΎΠ½ΡΠ°Π½ΠΈΡ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ.</li>
</ul>
<p>ΠΠ΅ΠΌΠΎΠ½ΡΡΡΠ°ΡΠΈΡ ΡΡΠ°Π³ΠΌΠ΅Π½ΡΠ° Π±Π»ΠΎΡΠ½ΠΎΠ³ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π΄Π»Ρ ΠΌΠ΅ΡΠΎΠ΄Π° <em>start_date</em> ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½Π°
Π½ΠΈΠΆΠ΅:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">import</span> <span class="nn">datetime</span>
<span class="kn">from</span> <span class="nn">django.test</span> <span class="kn">import</span> <span class="n">TestCase</span>
<span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="kn">import</span> <span class="n">User</span>
<span class="kn">from</span> <span class="nn">todo.models</span> <span class="kn">import</span> <span class="n">Chain</span><span class="p">,</span> <span class="n">Task</span>
<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">factories</span>
<span class="k">class</span> <span class="nc">TaskTest</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">factories</span><span class="o">.</span><span class="n">make_fixtures</span><span class="p">()</span>
<span class="c1"># Π‘ΠΎΡΡΡΠ΄Π½ΠΈΠΊΠΈ.</span>
<span class="bp">self</span><span class="o">.</span><span class="n">manager</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="s1">'alexander'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">designer</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="s1">'kazimir'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">programmer</span> <span class="o">=</span> <span class="n">User</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="s1">'ada'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">StartDateTest</span><span class="p">(</span><span class="n">TaskTest</span><span class="p">):</span>
<span class="sd">"""Π’Π΅ΡΡΠΈΡΡΠ΅Ρ ΠΎΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ."""</span>
<span class="k">def</span> <span class="nf">test_first_task</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Π’Π΅ΡΡΠΈΡΡΠ΅Ρ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ ΠΏΠ΅ΡΠ²ΠΎΠΉ Π·Π°Π΄Π°ΡΠΈ.</span>
<span class="sd"> ΠΠ°ΡΠ° Π½Π°ΡΠ°Π»Π° ΠΏΠ΅ΡΠ²ΠΎΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠΎΠ²ΠΏΠ°Π΄Π°Π΅Ρ Ρ Π΄Π°ΡΠΎΠΉ Π½Π°ΡΠ°Π»Π° ΡΠ΅ΠΏΠΎΡΠΊΠΈ. ΠΡΠΎ ΡΡΠ»ΠΎΠ²ΠΈΠ΅</span>
<span class="sd"> Π²Π΅ΡΠ½ΠΎ Π΄Π»Ρ Π·Π°Π΄Π°Ρ Ρ Π»ΡΠ±ΡΠΌ ΡΡΠ°ΡΡΡΠΎΠΌ.</span>
<span class="sd"> """</span>
<span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="n">chain_start_date</span> <span class="o">=</span> <span class="n">today</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">chain</span> <span class="o">=</span> <span class="n">Chain</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'Chain'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">chain_start_date</span><span class="p">,</span>
<span class="n">owner</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">manager</span><span class="p">)</span>
<span class="n">deadline</span> <span class="o">=</span> <span class="n">chain_start_date</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="n">first_task</span> <span class="o">=</span> <span class="n">Task</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">worker</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">designer</span><span class="p">,</span> <span class="n">task</span><span class="o">=</span><span class="s1">'Design'</span><span class="p">,</span>
<span class="n">deadline</span><span class="o">=</span><span class="n">deadline</span><span class="p">,</span> <span class="n">chain</span><span class="o">=</span><span class="n">chain</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">first_task</span><span class="o">.</span><span class="n">start_date</span><span class="p">(),</span> <span class="n">chain</span><span class="o">.</span><span class="n">start_date</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="section-7">
<h3>ΠΠ°ΠΊΠ»ΡΡΠ΅Π½ΠΈΠ΅</h3>
<p>ΠΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΡ Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠΈ <em>unittest</em> ΠΏΠΎ ΠΊΠ»Π°ΡΡΠ°ΠΌ ΠΈ ΠΌΠ΅ΡΠΎΠ΄Π°ΠΌ ΠΏΠΎΠ΄Ρ
ΠΎΠ΄ΠΈΡ Π² ΡΠ»ΡΡΠ°Π΅, ΠΊΠΎΠ³Π΄Π°
Π΅ΡΡΡ ΠΏΠΎΡΡΠ΅Π±Π½ΠΎΡΡΡ Π² Π½Π°ΠΏΠΈΡΠ°Π½ΠΈΠΈ ΡΠ΅ΡΡΠΎΠ², ΠΊΠΎΡΠΎΡΡΠ΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡΡ ΠΎΠ΄ΠΈΠ½Π°ΠΊΠΎΠ²ΡΠΉ ΠΊΠΎΠ΄. Π’Π°ΠΊΠΎΠΉ
ΠΏΠΎΠ΄Ρ
ΠΎΠ΄ ΡΠΏΡΠΎΡΠ°Π΅Ρ Π°Π±ΡΡΡΠ°ΠΊΡΠΈΡ ΠΎΠ±ΡΠΈΡ
Π·Π°Π΄Π°Ρ Π² ΠΎΠ±ΡΠΈΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ. ΠΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠ° ΡΠ°ΠΊΠΆΠ΅
ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°Π΅Ρ ΡΠ²Π½ΡΠ΅ ΠΏΡΠΎΡΠ΅Π΄ΡΡΡ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ ΠΈ ΠΎΡΠΈΡΡΠΊΠΈ, ΠΊΠΎΡΠΎΡΡΠ΅ ΠΏΡΠ΅Π΄ΠΎΡΡΠ°Π²Π»ΡΡΡ
Π²ΡΡΠΎΠΊΠΈΠΉ ΡΡΠΎΠ²Π΅Π½Ρ ΠΊΠΎΠ½ΡΡΠΎΠ»Ρ Π½Π°Π΄ ΡΡΠ΅Π΄ΠΎΠΉ, Π² ΠΊΠΎΡΠΎΡΠΎΠΉ ΠΏΡΠΎΠΈΡΡ
ΠΎΠ΄ΠΈΡ Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΠ΅ ΡΠ΅ΡΡΠΎΠ².</p>
</div>
</div>
<div class="section" id="section-8">
<h2>Π’Π΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΠΉ</h2>
<p>ΠΠ»Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Django ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΠΉ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅ΡΡΡ <a class="footnote-reference" href="#korobov" id="footnote-reference-7">[4]</a> ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ
Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΡ <em>WebTest</em>. ΠΠ»ΠΈΠΆΠ°ΠΉΡΠΈΠΌ Π°Π½Π°Π»ΠΎΠ³ΠΎΠΌ <em>WebTest</em> ΡΠ²Π»ΡΠ΅ΡΡΡ <em>twill</em>, Π½ΠΎ ΠΎΠ½ Π½Π΅
ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°Π΅Ρ ΡΠ½ΠΈΠΊΠΎΠ΄ ΠΈ Π΄Π°Π²Π½ΠΎ Π½Π΅ ΡΠ°Π·Π²ΠΈΠ²Π°Π΅ΡΡΡ (ΠΏΠΎΡΠ»Π΅Π΄Π½ΠΈΠΉ ΡΠ΅Π»ΠΈΠ· Π±ΡΠ» Π² 2007 Π³ΠΎΠ΄Ρ).</p>
<p>Π ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅ ΠΏΡΠΈΠΌΠ΅ΡΠ° ΠΏΡΠΈΠ²ΠΎΠ΄ΠΈΡΡΡ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΏΠΎΡΠ΅ΡΠ΅Π½ΠΈΡ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Π΅ΠΌ ΡΡΡΠ°Π½ΠΈΡΡ
Π°ΠΊΡΡΠ°Π»ΡΠ½ΡΡ
Π·Π°Π΄Π°Ρ:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">from</span> <span class="nn">django_webtest</span> <span class="kn">import</span> <span class="n">WebTest</span>
<span class="kn">from</span> <span class="nn">django.core.urlresolvers</span> <span class="kn">import</span> <span class="n">reverse</span>
<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">factories</span>
<span class="k">class</span> <span class="nc">ActualTasksTest</span><span class="p">(</span><span class="n">WebTest</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">factories</span><span class="o">.</span><span class="n">make_fixtures</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_user_not_logined</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">app</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">reverse</span><span class="p">(</span><span class="s1">'todo_actual_tasks'</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_int</span><span class="p">,</span> <span class="mi">302</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_designer_logined</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">app</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">reverse</span><span class="p">(</span><span class="s1">'todo_actual_tasks'</span><span class="p">),</span> <span class="n">user</span><span class="o">=</span><span class="s1">'kazimir'</span><span class="p">)</span>
<span class="k">assert</span> <span class="s1">'ΠΠ°Π·ΠΈΠΌΠΈΡ ΠΠ°Π»Π΅Π²ΠΈΡ'</span> <span class="ow">in</span> <span class="n">response</span>
</pre></div>
<p>Π ΠΏΠ΅ΡΠ²ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ Π½Π΅ Π°Π²ΡΠΎΡΠΈΠ·ΠΎΠ²Π°Π½ (ΠΌΠ΅ΡΠΎΠ΄ <em>test_user_not_logined</em>) ΠΈ
Π±ΡΠ°ΡΠ·Π΅Ρ Π΄ΠΎΠ»ΠΆΠ΅Π½ Π²Π΅ΡΠ½ΡΡΡ ΡΡΠ°ΡΡΡ <em>302</em> (ΠΏΠ΅ΡΠ΅Π½Π°ΠΏΡΠ°Π²Π»Π΅Π½ΠΈΠ΅ Π½Π° ΡΡΡΠ°Π½ΠΈΡΡ Π°Π²ΡΠΎΡΠΈΠ·Π°ΡΠΈΠΈ),
Π²ΠΎ Π²ΡΠΎΡΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ (ΠΌΠ΅ΡΠΎΠ΄ <em>test_designer_logined</em>) ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ Π°Π²ΡΠΎΡΠΈΠ·ΠΎΠ²Π°Π½ ΠΏΠΎΠ΄
ΠΈΠΌΠ΅Π½Π΅ΠΌ <em>ΠΠ°Π·ΠΈΠΌΠΈΡ ΠΠ°Π»Π΅Π²ΠΈΡ</em>. ΠΠ°Π½Π½ΡΠ΅ ΡΠ΅ΡΡΡ Π½Π΅ ΡΠ°ΠΊΠΈΠ΅ ΠΏΠΎΠ»Π΅Π·Π½ΡΠ΅, ΠΊΠ°ΠΊ Π±Π»ΠΎΡΠ½ΡΠ΅ ΡΠ΅ΡΡΡ.
ΠΠΎ Π΄Π°ΠΆΠ΅ Π΅ΡΠ»ΠΈ ΠΎΠ½ΠΈ ΠΏΡΠΎΡΡΠΎ ΠΏΡΠΎΠ²Π΅ΡΡΡ ΠΎΡΠ½ΠΎΠ²Π½ΡΠ΅ ΡΡΡΠ°Π½ΠΈΡΡ ΡΠΈΡΡΠ΅ΠΌΡ Π½Π° ΠΎΡΡΡΡΡΡΠ²ΠΈΠ΅
ΡΠΎΠΎΠ±ΡΠ΅Π½ΠΈΠΉ ΠΎΠ± ΠΈΡΠΊΠ»ΡΡΠ΅Π½ΠΈΠΈ, ΡΠΎ ΠΎΠ½ΠΈ ΡΠΆΠ΅ ΠΏΡΠΈΠ½Π΅ΡΡΡ Π±ΠΎΠ»ΡΡΡΡ ΠΏΠΎΠ»ΡΠ·Ρ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΡ.</p>
<p>ΠΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΡ ΠΈΠΌΠ΅ΡΡ ΠΌΠ½ΠΎΠ³ΠΎ ΡΠ²ΡΠ·Π΅ΠΉ ΠΈ Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠ΅ΠΉ (ΡΠ°Π±Π»ΠΎΠ½Ρ, Π±Π°Π·Π° Π΄Π°Π½Π½ΡΡ
,
ΠΊΠΎΠ½ΡΠΈΠ³ΡΡΠ°ΡΠΈΡ URL), ΠΏΠΎΡΡΠΎΠΌΡ ΠΈΡ
ΡΡΡΠ΄Π½ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°ΡΡ. ΠΠ°ΡΠ» ΠΠ°ΠΉΠ΅Ρ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅Ρ
ΠΏΠΈΡΠ°ΡΡ ΠΊΠ°ΠΊ ΠΌΠΎΠΆΠ½ΠΎ ΠΌΠ΅Π½ΡΡΠ΅ ΠΊΠΎΠ΄Π° Π½Π° ΡΡΠΎΠ²Π½Π΅ ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΠΉ <a class="footnote-reference" href="#meyer" id="footnote-reference-8">[2]</a>.</p>
<table class="docutils footnote" frame="void" id="mcconnell" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#footnote-reference-1">1</a>, <a class="fn-backref" href="#footnote-reference-2">2</a>, <a class="fn-backref" href="#footnote-reference-6">3</a>)</em> ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π» Π‘. Π‘ΠΎΠ²Π΅ΡΡΠ΅Π½Π½ΡΠΉ ΠΊΠΎΠ΄. ΠΠ°ΡΡΠ΅Ρ-ΠΊΠ»Π°ΡΡ /
ΠΠ΅Ρ. Ρ Π°Π½Π³Π». β Π. : ΠΠ·Π΄Π°ΡΠ΅Π»ΡΡΡΠ²ΠΎ "Π ΡΡΡΠΊΠ°Ρ ΡΠ΅Π΄Π°ΠΊΡΠΈΡ", 2012. β 896 ΡΡΡ. : ΠΈΠ».</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="meyer" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[2]</td><td><em>(<a class="fn-backref" href="#footnote-reference-3">1</a>, <a class="fn-backref" href="#footnote-reference-8">2</a>)</em> Meyer C. <a class="reference external" href="http://carljm.github.com/django-testing-slides/">Testing and Django</a> at PyCon US 2012.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="lindsley" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[3]</td><td><em>(<a class="fn-backref" href="#footnote-reference-4">1</a>, <a class="fn-backref" href="#footnote-reference-5">2</a>)</em> ΠΠΈΠ½Π΄ΡΠ»ΠΈ Π. <a class="reference external" href="http://toastdriven.com/blog/2011/apr/10/guide-to-testing-in-django/">Guide to Testing in Django</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="korobov" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-7">[4]</a></td><td>ΠΠΎΡΠΎΠ±ΠΎΠ² Π. <a class="reference external" href="http://habrahabr.ru/post/91471/">ΠΠΈΡΠ΅ΠΌ ΡΡΠ½ΠΊΡΠΈΠΎΠ½Π°Π»ΡΠ½ΡΠ΅/ΠΈΠ½ΡΠ΅Π³ΡΠ°ΡΠΈΠΎΠ½Π½ΡΠ΅ ΡΠ΅ΡΡΡ Π΄Π»Ρ
ΠΏΡΠΎΠ΅ΠΊΡΠ° Π½Π° Django</a>.</td></tr>
</tbody>
</table>
</div>
Django TODO: ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΡΠΈΡΡΠ΅ΠΌΡ2012-06-29T12:00:00+07:002012-06-29T12:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-29:/django-todo-system-construction.html<p>ΠΡΠΈ ΡΠ°Π±ΠΎΡΠ΅ Π½Π°Π΄ ΠΏΡΠΎΠ΅ΠΊΡΠΎΠΌ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π²ΠΊΠ»ΡΡΠ°Π΅Ρ Π΄ΡΡΠ³ΠΈΠ΅ ΠΏΡΠΎΡΠ΅ΡΡΡ, Π² ΡΠΎΠΌ ΡΠΈΡΠ»Π΅
ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅. Π€ΠΎΡΠΌΠ°Π»ΡΠ½Π°Ρ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ° Π΄Π°Π΅Ρ ΠΎΡΠ²Π΅ΡΡ ΡΠΎΠ»ΡΠΊΠΎ Π½Π° Π²ΠΎΠΏΡΠΎΡΡ
ΡΠΈΡΡΠ΅ΠΌΠ½ΠΎΠ³ΠΎ ΡΡΠΎΠ²Π½Ρ, ΠΏΡΠΈ ΡΡΠΎΠΌ Π·Π½Π°ΡΠΈΡΠ΅Π»ΡΠ½Π°Ρ ΡΠ°ΡΡΡ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ
Π½Π°ΠΌΠ΅ΡΠ΅Π½Π½ΠΎ ΠΎΡΡΠ°Π²Π»Π΅Π½Π° Π½Π° ΡΡΠ°ΠΏ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ. ΠΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ -- ΡΡΠΎ
"ΠΏΠΎΡΡΠ΅ΠΏΠ΅Π½Π½ΡΠΉ" ΠΏΡΠΎΡΠ΅ΡΡ. ΠΡΠΎΠ΅ΠΊΡΡ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π΅ Π²ΠΎΠ·Π½ΠΈΠΊΠ°ΡΡ Π² ΡΠΌΠ°Ρ
ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠΎΠ²
ΡΡΠ°Π·Ρ Π² Π³ΠΎΡΠΎΠ²ΠΎΠΌ Π²ΠΈΠ΄Π΅. ΠΠ½ΠΈ ΡΠ°Π·Π²ΠΈΠ²Π°ΡΡΡΡ β¦</p><p>ΠΡΠΈ ΡΠ°Π±ΠΎΡΠ΅ Π½Π°Π΄ ΠΏΡΠΎΠ΅ΠΊΡΠΎΠΌ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π²ΠΊΠ»ΡΡΠ°Π΅Ρ Π΄ΡΡΠ³ΠΈΠ΅ ΠΏΡΠΎΡΠ΅ΡΡΡ, Π² ΡΠΎΠΌ ΡΠΈΡΠ»Π΅
ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅. Π€ΠΎΡΠΌΠ°Π»ΡΠ½Π°Ρ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ° Π΄Π°Π΅Ρ ΠΎΡΠ²Π΅ΡΡ ΡΠΎΠ»ΡΠΊΠΎ Π½Π° Π²ΠΎΠΏΡΠΎΡΡ
ΡΠΈΡΡΠ΅ΠΌΠ½ΠΎΠ³ΠΎ ΡΡΠΎΠ²Π½Ρ, ΠΏΡΠΈ ΡΡΠΎΠΌ Π·Π½Π°ΡΠΈΡΠ΅Π»ΡΠ½Π°Ρ ΡΠ°ΡΡΡ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ
Π½Π°ΠΌΠ΅ΡΠ΅Π½Π½ΠΎ ΠΎΡΡΠ°Π²Π»Π΅Π½Π° Π½Π° ΡΡΠ°ΠΏ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ. ΠΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ -- ΡΡΠΎ
"ΠΏΠΎΡΡΠ΅ΠΏΠ΅Π½Π½ΡΠΉ" ΠΏΡΠΎΡΠ΅ΡΡ. ΠΡΠΎΠ΅ΠΊΡΡ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π΅ Π²ΠΎΠ·Π½ΠΈΠΊΠ°ΡΡ Π² ΡΠΌΠ°Ρ
ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠΎΠ²
ΡΡΠ°Π·Ρ Π² Π³ΠΎΡΠΎΠ²ΠΎΠΌ Π²ΠΈΠ΄Π΅. ΠΠ½ΠΈ ΡΠ°Π·Π²ΠΈΠ²Π°ΡΡΡΡ ΠΈ ΡΠ»ΡΡΡΠ°ΡΡΡΡ Π² Ρ
ΠΎΠ΄Π΅ ΠΎΠ±Π·ΠΎΡΠΎΠ²,
Π½Π΅ΡΠΎΡΠΌΠ°Π»ΡΠ½ΡΡ
ΠΎΠ±ΡΡΠΆΠ΄Π΅Π½ΠΈΠΉ, Π½Π°ΠΏΠΈΡΠ°Π½ΠΈΡ ΠΊΠΎΠ΄Π° ΠΈ Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ Π΅Π³ΠΎ ΡΠ΅Π²ΠΈΠ·ΠΈΠΉ.</p>
<p>ΠΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈ Π²ΠΎ Π²ΡΠ΅Ρ
ΡΠ»ΡΡΠ°ΡΡ
ΠΏΡΠΎΠ΅ΠΊΡ Π½Π΅ΡΠΊΠΎΠ»ΡΠΊΠΎ ΠΌΠ΅Π½ΡΠ΅ΡΡΡ Π²ΠΎ Π²ΡΠ΅ΠΌΡ ΠΏΠ΅ΡΠ²ΠΎΠ½Π°ΡΠ°Π»ΡΠ½ΠΎΠΉ
ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ ΡΠΈΡΡΠ΅ΠΌΡ ΠΈ Π΅ΡΠ΅ Π±ΠΎΠ»ΡΡΠ΅ -- ΠΏΡΠΈ Π΅Π΅ ΠΌΠΎΠ΄Π΅ΡΠ½ΠΈΠ·Π°ΡΠΈΠΈ.</p>
<p>Π£ΠΏΡΠ°Π²Π»Π΅Π½ΠΈΠ΅ ΡΠ»ΠΎΠΆΠ½ΠΎΡΡΡΡ -- ΡΠ°ΠΌΡΠΉ Π²Π°ΠΆΠ½ΡΠΉ ΡΠ΅Ρ
Π½ΠΈΡΠ΅ΡΠΊΠΈΠΉ Π°ΡΠΏΠ΅ΠΊΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ ΠΠ.
Π£ΠΏΡΠ°Π²Π»ΡΡΡ ΡΠ»ΠΎΠΆΠ½ΠΎΡΡΡΡ Π³ΠΎΡΠ°Π·Π΄ΠΎ Π»Π΅Π³ΡΠ΅, Π΅ΡΠ»ΠΈ ΠΏΡΠΈ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠΈ ΡΡΡΠ΅ΠΌΠΈΡΡΡΡ ΠΊ
ΠΏΡΠΎΡΡΠΎΡΠ΅. ΠΡΡΡ Π΄Π²Π° ΠΎΠ±ΡΠΈΡ
ΡΠΏΠΎΡΠΎΠ±Π° Π΄ΠΎΡΡΠΈΠΆΠ΅Π½ΠΈΡ ΠΏΡΠΎΡΡΠΎΡΡ: ΠΌΠΈΠ½ΠΈΠΌΠΈΠ·Π°ΡΠΈΡ ΠΎΠ±ΡΠ΅ΠΌΠ°
ΡΡΡΠ΅ΡΡΠ²Π΅Π½Π½ΠΎΠΉ ΡΠ»ΠΎΠΆΠ½ΠΎΡΡΠΈ, Ρ ΠΊΠΎΡΠΎΡΠΎΠΉ ΠΏΡΠΈΡ
ΠΎΠ΄ΠΈΡΡΡ ΠΈΠΌΠ΅ΡΡ Π΄Π΅Π»ΠΎ Π² Π»ΡΠ±ΠΎΠΉ ΠΊΠΎΠ½ΠΊΡΠ΅ΡΠ½ΡΠΉ
ΠΌΠΎΠΌΠ΅Π½Ρ Π²ΡΠ΅ΠΌΠ΅Π½ΠΈ, ΠΈ ΠΏΠΎΠ΄Π°Π²Π»Π΅Π½ΠΈΠ΅ Π½Π΅ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ³ΠΎ ΡΠΎΡΡΠ° Π½Π΅ΡΡΡΠ΅ΡΡΠ²Π΅Π½Π½ΠΎΠΉ ΡΠ»ΠΎΠΆΠ½ΠΎΡΡΠΈ.</p>
<p>ΠΠ΄Π½ΠΎΠΉ ΠΈΠ· ΡΠ°ΠΌΡΡ
ΠΏΠΎΠ»Π΅Π·Π½ΡΡ
ΠΊΠΎΠ½ΡΠ΅ΠΏΡΠΈΠΉ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΡΠ²Π»ΡΠ΅ΡΡΡ ΡΠΎΠΊΡΡΡΠΈΠ΅ ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠΈ.
ΠΠ½ΠΎ ΠΏΠΎΠ»Π΅Π·Π½ΠΎ Π½Π° Π²ΡΠ΅Ρ
ΡΡΠΎΠ²Π½ΡΡ
ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ: ΠΎΡ ΠΏΡΠΈΠΌΠ΅Π½Π΅Π½ΠΈΡ ΠΈΠΌΠ΅Π½ΠΎΠ²Π°Π½Π½ΡΡ
ΠΊΠΎΠ½ΡΡΠ°Π½Ρ
Π²ΠΌΠ΅ΡΡΠΎ Π»ΠΈΡΠ΅ΡΠ°Π»ΠΎΠ² Π΄ΠΎ ΡΠΎΠ·Π΄Π°Π½ΠΈΡ ΡΠΈΠΏΠΎΠ² Π΄Π°Π½Π½ΡΡ
ΠΈ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΊΠ»Π°ΡΡΠΎΠ², ΠΌΠ΅ΡΠΎΠ΄ΠΎΠ² ΠΈ
ΠΏΠΎΠ΄ΡΠΈΡΡΠ΅ΠΌ <a class="footnote-reference" href="#mcconnell" id="footnote-reference-1">[1]</a>.</p>
<p>ΠΠ»Ρ ΡΠ½ΠΈΠΆΠ΅Π½ΠΈΡ ΡΠ»ΠΎΠΆΠ½ΠΎΡΡΠΈ Π‘ΡΠΈΠ² ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π» ΠΎΠΏΡΠ΅Π΄Π΅Π»ΠΈΠ» ΡΡΠ΄ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΠΉ
<a class="footnote-reference" href="#mcconnell" id="footnote-reference-2">[1]</a>:</p>
<ul class="simple">
<li>ΡΡΠ°ΡΠ΅Π»ΡΠ½ΠΎ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠΉΡΠ΅ ΠΈΠ½ΡΠ΅ΡΡΠ΅ΠΉΡΡ ΠΊΠ»Π°ΡΡΠΎΠ², ΡΡΠΎΠ±Ρ ΠΌΠΎΠΆΠ½ΠΎ Π±ΡΠ»ΠΎ ΠΈΠ³Π½ΠΎΡΠΈΡΠΎΠ²Π°ΡΡ
Π²Π½ΡΡΡΠ΅Π½Π½Π΅Π΅ ΡΡΡΡΠΎΠΉΡΡΠ²ΠΎ ΠΊΠ»Π°ΡΡΠΎΠ²;</li>
<li>ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°ΠΉΡΠ΅ Π°Π±ΡΡΡΠ°ΠΊΡΠΈΡ, ΡΠΎΡΠΌΠΈΡΡΠ΅ΠΌΡΡ ΠΈΠ½ΡΠ΅ΡΡΠ΅ΠΉΡΠΎΠΌ ΠΊΠ»Π°ΡΡΠ°, ΡΡΠΎΠ±Ρ Π½Π΅
Π·Π°ΠΏΠΎΠΌΠΈΠ½Π°ΡΡ Π½Π΅Π½ΡΠΆΠ½ΡΡ
Π΄Π΅ΡΠ°Π»Π΅ΠΉ;</li>
<li>ΠΈΠ·Π±Π΅Π³Π°ΠΉΡΠ΅ Π³Π»ΠΎΠ±Π°Π»ΡΠ½ΡΡ
Π΄Π°Π½Π½ΡΡ
, ΠΏΠΎΡΠΎΠΌΡ ΡΡΠΎ ΠΈΡ
ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠ΅ Π·Π½Π°ΡΠΈΡΠ΅Π»ΡΠ½ΠΎ
ΡΠ²Π΅Π»ΠΈΡΠΈΠ²Π°Π΅Ρ ΠΏΡΠΎΡΠ΅Π½Ρ ΠΊΠΎΠ΄Π°, ΠΊΠΎΡΠΎΡΡΠΉ Π½ΡΠΆΠ½ΠΎ ΡΠ΄Π΅ΡΠΆΠΈΠ²Π°ΡΡ Π² ΡΠΌΠ΅ Π² Π»ΡΠ±ΠΎΠΉ ΠΌΠΎΠΌΠ΅Π½Ρ
Π²ΡΠ΅ΠΌΠ΅Π½ΠΈ;</li>
<li>ΠΈΠ·Π±Π΅Π³Π°ΠΉΡΠ΅ Π³Π»ΡΠ±ΠΎΠΊΠΈΡ
ΠΈΠ΅ΡΠ°ΡΡ
ΠΈΠΉ Π½Π°ΡΠ»Π΅Π΄ΠΎΠ²Π°Π½ΠΈΡ, ΠΏΠΎΡΠΎΠΌΡ ΡΡΠΎ ΠΎΠ½ΠΈ ΠΏΡΠ΅Π΄ΡΡΠ²Π»ΡΡΡ Π²ΡΡΠΎΠΊΠΈΠ΅
ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ ΠΊ ΠΈΠ½ΡΠ΅Π»Π»Π΅ΠΊΡΡ;</li>
<li>ΠΈΠ·Π±Π΅Π³Π°ΠΉΡΠ΅ Π³Π»ΡΠ±ΠΎΠΊΠΎΠΉ Π²Π»ΠΎΠΆΠ΅Π½Π½ΠΎΡΡΠΈ ΡΠΈΠΊΠ»ΠΎΠ² ΠΈ ΡΡΠ»ΠΎΠ²Π½ΡΡ
ΠΎΠΏΠ΅ΡΠ°ΡΠΎΡΠΎΠ², ΠΏΠΎΡΠΊΠΎΠ»ΡΠΊΡ ΠΈΡ
ΠΌΠΎΠΆΠ½ΠΎ Π·Π°ΠΌΠ΅Π½ΠΈΡΡ Π½Π° Π±ΠΎΠ»Π΅Π΅ ΠΏΡΠΎΡΡΡΠ΅ ΡΠΏΡΠ°Π²Π»ΡΡΡΠΈΠ΅ ΡΡΡΡΠΊΡΡΡΡ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡΡΠΈΠ΅ Π±Π΅ΡΠ΅ΠΆΠ½Π΅Π΅
ΡΠ°ΡΡ
ΠΎΠ΄ΠΎΠ²Π°ΡΡ ΡΠΌΡΡΠ²Π΅Π½Π½ΡΠ΅ ΡΠ΅ΡΡΡΡΡ;</li>
<li>ΡΡΠ°ΡΠ΅Π»ΡΠ½ΠΎ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΠΈΡΠ΅ ΠΏΠΎΠ΄Ρ
ΠΎΠ΄ ΠΊ ΠΎΠ±ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΎΡΠΈΠ±ΠΎΠΊ, Π²ΠΌΠ΅ΡΡΠΎ ΡΠΎΠ³ΠΎ, ΡΡΠΎΠ±Ρ
ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ ΠΏΡΠΎΠΈΠ·Π²ΠΎΠ»ΡΠ½ΡΡ ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΡ ΠΏΡΠΎΠΈΠ·Π²ΠΎΠ»ΡΠ½ΡΡ
ΠΌΠ΅ΡΠΎΠ΄ΠΈΠΊ;</li>
<li>Π½Π΅ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠΉΡΠ΅ ΠΊΠ»Π°ΡΡΠ°ΠΌ Π΄ΠΎΡΡΠΈΠ³Π°ΡΡ ΡΠ°Π·ΠΌΠ΅ΡΠΎΠ² ΡΠ΅Π»ΡΡ
ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌ;</li>
<li>ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°ΠΉΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΊΠΎΡΠΎΡΠΊΠΈΠΌΠΈ;</li>
<li>ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠΉΡΠ΅ ΡΡΠ½ΡΠ΅, ΠΎΡΠ΅Π²ΠΈΠ΄Π½ΡΠ΅ ΠΈΠΌΠ΅Π½Π° ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΡΡ
;</li>
<li>ΠΌΠΈΠ½ΠΈΠΌΠΈΠ·ΠΈΡΡΠΉΡΠ΅ ΡΠΈΡΠ»ΠΎ ΠΏΠ°ΡΠ°ΠΌΠ΅ΡΡΠΎΠ², ΠΏΠ΅ΡΠ΅Π΄Π°Π²Π°Π΅ΠΌΡΡ
Π² ΠΌΠ΅ΡΠΎΠ΄, ΠΈΠ»ΠΈ, ΡΡΠΎ Π΅ΡΠ΅ Π²Π°ΠΆΠ½Π΅Π΅,
ΠΏΠ΅ΡΠ΅Π΄Π°Π²Π°ΠΉΡΠ΅ ΡΠΎΠ»ΡΠΊΠΎ ΡΠ΅ ΠΏΠ°ΡΠ°ΠΌΠ΅ΡΡΡ, ΠΊΠΎΡΠΎΡΡΠ΅ Π½ΡΠΆΠ½Ρ Π΄Π»Ρ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠ°Π½ΠΈΡ Π°Π±ΡΡΡΠ°ΠΊΡΠΈΠΈ,
ΡΠΎΡΠΌΠΈΡΡΠ΅ΠΌΠΎΠΉ ΠΈΠ½ΡΠ΅ΡΡΠ΅ΠΉΡΠΎΠΌ ΠΌΠ΅ΡΠΎΠ΄Π°;</li>
<li>ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠΉΡΠ΅ ΡΠΎΠ³Π»Π°ΡΠ΅Π½ΠΈΡ, ΡΡΠΎΠ±Ρ Π½Π΅ Π·Π°ΠΏΠΎΠΌΠΈΠ½Π°ΡΡ ΠΏΡΠΎΠΈΠ·Π²ΠΎΠ»ΡΠ½ΡΠ΅, Π½Π΅ΡΡΡΠ΅ΡΡΠ²Π΅Π½Π½ΡΠ΅
ΡΠ°Π·Π»ΠΈΡΠΈΡ ΠΌΠ΅ΠΆΠ΄Ρ ΡΠ°Π·Π½ΡΠΌΠΈ ΡΡΠ°Π³ΠΌΠ΅Π½ΡΠ°ΠΌΠΈ ΠΊΠΎΠ΄Π°;</li>
<li>ΡΠΎΠ·Π΄Π°Π²Π°Ρ Π΄Π»Ρ ΡΠ»ΠΎΠΆΠ½ΠΎΠ³ΠΎ ΡΠ΅ΡΡΠ° Π±ΡΠ»Π΅Π²Ρ ΡΡΠ½ΠΊΡΠΈΡ ΠΈ Π°Π±ΡΡΡΠ°Π³ΠΈΡΡΡ ΡΡΡΡ ΡΠ΅ΡΡΠ°
ΡΠΏΡΠΎΡΠ°Π΅ΡΡΡ ΠΊΠΎΠ΄.</li>
</ul>
<p>ΠΡΠΈ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠΈ ΡΠΈΡΡΠ΅ΠΌΡ <a class="reference external" href="https://github.com/marselester/django-todo">Django TODO</a> Ρ ΡΡΠ°ΡΠ°Π»ΡΡ ΠΏΡΠΈΠ΄Π΅ΡΠΆΠΈΠ²Π°ΡΡΡΡ Π²ΡΡΠ΅ΠΏΠ΅ΡΠ΅ΡΠΈΡΠ»Π΅Π½Π½ΡΡ
ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΠΉ.</p>
<div class="section" id="section-1">
<h2>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
</h2>
<p>Π£ΡΠΎΠ²Π΅Π½Ρ Π΄ΠΎΡΡΡΠΏΠ° ΠΊ Π΄Π°Π½Π½ΡΠΌ Π²ΡΠ½Π΅ΡΠ΅Π½ Π² ΠΎΡΠ΄Π΅Π»ΡΠ½ΡΡ ΡΠ°ΡΡΡ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ, Π½Π°Π·ΡΠ²Π°Π΅ΠΌΡΠΉ
ΠΌΠΎΠ΄Π΅Π»ΡΡ. ΠΠΎΠ΄Π΅Π»ΠΈ ΡΠ°Π·ΠΌΠ΅ΡΠ°ΡΡΡΡ Π² ΡΠ°ΠΉΠ»Π΅ <tt class="docutils literal">models.py</tt>.</p>
<p>ΠΠΎΠ΄Π΅Π»Ρ Π² Django ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»ΡΠ΅Ρ ΡΠΎΠ±ΠΎΠΉ ΠΎΠΏΠΈΡΠ°Π½ΠΈΠ΅ Π΄Π°Π½Π½ΡΡ
Π² Π±Π°Π·Π΅, ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½Π½ΠΎΠ΅ Π½Π°
ΡΠ·ΡΠΊΠ΅ Python. Π Django ΠΌΠΎΠ΄Π΅Π»Ρ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ, ΡΡΠΎΠ±Ρ Π²ΡΠΏΠΎΠ»Π½ΠΈΡΡ SQL-Π·Π°ΠΏΡΠΎΡ ΠΈ
Π²Π΅ΡΠ½ΡΡΡ ΡΠ΄ΠΎΠ±Π½ΡΠ΅ ΡΡΡΡΠΊΡΡΡΡ Π΄Π°Π½Π½ΡΡ
Python. ΠΡΠΎΠΌΠ΅ ΡΠΎΠ³ΠΎ, ΠΌΠΎΠ΄Π΅Π»ΠΈ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΡΡ
ΠΏΡΠ΅Π΄ΡΡΠ°Π²ΠΈΡΡ Π²ΡΡΠΎΠΊΠΎΡΡΠΎΠ²Π½Π΅Π²ΡΠ΅ ΠΊΠΎΠ½ΡΠ΅ΠΏΡΠΈΠΈ, Π΄Π»Ρ ΠΊΠΎΡΠΎΡΡΡ
Π² SQL ΠΌΠΎΠΆΠ΅Ρ Π½Π΅ Π±ΡΡΡ
Π°Π½Π°Π»ΠΎΠ³ΠΎΠ².</p>
<p>ΠΠΏΠΈΡΠ°Π½ΠΈΠ΅ ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ Π½Π° Python Π»ΡΡΡΠ΅ ΠΏΠΎ ΡΡΠ΄Ρ ΠΏΡΠΈΡΠΈΠ½:</p>
<ul class="simple">
<li>ΡΠΎΠΊΡΠ°ΡΠ°Π΅ΡΡΡ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ ΠΌΡΡΠ»Π΅Π½Π½ΡΡ
"ΠΏΠ΅ΡΠ΅ΠΊΠ»ΡΡΠ΅Π½ΠΈΠΉ ΠΊΠΎΠ½ΡΠ΅ΠΊΡΡΠ°". ΠΠΎΠ³Π΄Π° ΠΏΡΠΈΡ
ΠΎΠ΄ΠΈΡΡΡ
ΠΏΠΈΡΠ°ΡΡ ΠΊΠΎΠ΄ Π½Π° SQL, ΠΏΠΎΡΠΎΠΌ Π½Π° Python, Π° ΠΏΠΎΡΠΎΠΌ ΡΠ½ΠΎΠ²Π° Π½Π° SQL, ΠΏΡΠΎΠ΄ΡΠΊΡΠΈΠ²Π½ΠΎΡΡΡ
ΠΏΠ°Π΄Π°Π΅Ρ;</li>
<li>ΡΠ°Π·ΠΌΠ΅ΡΠ΅Π½ΠΈΠ΅ ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ Π΄Π°Π½Π½ΡΡ
Π² ΠΊΠΎΠ΄Π΅ ΡΠΏΡΠΎΡΠ°Π΅Ρ ΠΈΡ
Ρ
ΡΠ°Π½Π΅Π½ΠΈΠ΅ Π² ΡΠΈΡΡΠ΅ΠΌΠ΅ ΡΠΏΡΠ°Π²Π»Π΅Π½ΠΈΡ
Π²Π΅ΡΡΠΈΡΠΌΠΈ;</li>
<li>ΡΠ΄ΠΎΠ±Π½ΡΠ΅ ΡΡΠ΅Π΄ΡΡΠ²Π° ΠΌΠΈΠ³ΡΠ°ΡΠΈΠΈ ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ (Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, South).</li>
</ul>
<p>ΠΠ°ΠΆΠ΄Π°Ρ ΠΌΠΎΠ΄Π΅Π»Ρ ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΡΠ΅Ρ ΠΎΠ΄Π½ΠΎΠΉ ΡΠ°Π±Π»ΠΈΡΠ΅ ΠΠ, Π° ΠΊΠ°ΠΆΠ΄ΡΠΉ Π°ΡΡΠΈΠ±ΡΡ ΠΌΠΎΠ΄Π΅Π»ΠΈ
ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΡΠ΅Ρ ΠΎΠ΄Π½ΠΎΠΌΡ ΡΡΠΎΠ»Π±ΡΡ ΡΠ°Π±Π»ΠΈΡΡ. ΠΠ· ΡΡΠΎΠ³ΠΎ ΠΏΡΠ°Π²ΠΈΠ»Π° Π΅ΡΡΡ ΠΈΡΠΊΠ»ΡΡΠ΅Π½ΠΈΠ΅,
ΠΊΠ°ΡΠ°ΡΡΠ΅Π΅ΡΡ ΠΎΡΠ½ΠΎΡΠ΅Π½ΠΈΡ ΠΌΠ½ΠΎΠ³ΠΈΠ΅-ΠΊΠΎ-ΠΌΠ½ΠΎΠ³ΠΈΠΌ <a class="footnote-reference" href="#holovaty" id="footnote-reference-3">[2]</a>.</p>
<div class="section" id="section-2">
<h3>Π‘ΠΎΠ³Π»Π°ΡΠ΅Π½ΠΈΡ ΠΏΠΎ ΠΎΡΠΎΡΠΌΠ»Π΅Π½ΠΈΡ ΠΊΠΎΠ΄Π°</h3>
<p>ΠΡΠΈ ΠΊΠΎΠ½ΡΡΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠΈ ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ ΠΏΡΠΈΠ½ΡΡΡ ΡΠ»Π΅Π΄ΡΡΡΠΈΠ΅ ΡΠΎΠ³Π»Π°ΡΠ΅Π½ΠΈΡ ΠΏΠΎ ΠΎΡΠΎΡΠΌΠ»Π΅Π½ΠΈΡ ΠΊΠΎΠ΄Π°.
ΠΠΎΡΠ»Π΅Π΄ΠΎΠ²Π°ΡΠ΅Π»ΡΠ½ΠΎΡΡΡ ΡΠ°Π·ΠΌΠ΅ΡΠ΅Π½ΠΈΡ Π°ΡΡΠΈΠ±ΡΡΠΎΠ² ΠΊΠ»Π°ΡΡΠ°-ΠΌΠΎΠ΄Π΅Π»ΠΈ <a class="footnote-reference" href="#django" id="footnote-reference-4">[3]</a>:</p>
<ul class="simple">
<li>Π½Π°Π·Π²Π°Π½ΠΈΡ ΠΏΠΎΠ»Π΅ΠΉ;</li>
<li>Π°ΡΡΠΈΠ±ΡΡΡ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΠ° ΠΌΠΎΠ΄Π΅Π»ΠΈ;</li>
<li>ΠΊΠ»Π°ΡΡ <tt class="docutils literal">Meta</tt>;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ <tt class="docutils literal">__unicode__</tt>;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ <tt class="docutils literal">__str__</tt>;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ <tt class="docutils literal">save</tt>;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ <tt class="docutils literal">get_absolute_url</tt>;</li>
<li>ΠΎΡΡΠ°Π»ΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ.</li>
</ul>
<p>ΠΡΡΠ΅ΠΏΠ΅ΡΠ΅ΡΠΈΡΠ»Π΅Π½Π½ΡΠ΅ Π³ΡΡΠΏΠΏΡ Π°ΡΡΠΈΠ±ΡΡΠΎΠ² Π΄ΠΎΠ»ΠΆΠ½Ρ Π±ΡΡΡ ΠΎΡΠ΄Π΅Π»Π΅Π½Ρ Π΄ΡΡΠ³ ΠΎΡ Π΄ΡΡΠ³Π° ΠΏΡΡΡΠΎΠΉ
ΡΡΡΠΎΠΊΠΎΠΉ.</p>
<p>ΠΡΠ»ΠΈ Π² ΠΌΠΎΠ΄Π΅Π»ΠΈ Π΅ΡΡΡ ΠΏΠΎΠ»Π΅ Π²ΡΠ±ΠΎΡΠ° (<em>choices</em>), ΠΎΠ½ΠΎ Π΄ΠΎΠ»ΠΆΠ½ΠΎ Π±ΡΡΡ ΠΎΡΠΎΡΠΌΠ»Π΅Π½ΠΎ Π² Π²ΠΈΠ΄Π΅
ΠΊΠΎΡΡΠ΅ΠΆΠ° ΠΈΠ· ΠΊΠΎΡΡΠ΅ΠΆΠ΅ΠΉ, Π·Π°Π³Π»Π°Π²Π½ΡΠΌΠΈ Π±ΡΠΊΠ²Π°ΠΌΠΈ. ΠΠΎΠ»ΠΆΠ½Ρ Π±ΡΡΡ ΡΠ°ΡΠΏΠΎΠ»ΠΎΠΆΠ΅Π½Ρ Π² Π½Π°ΡΠ°Π»Π΅
ΠΌΠΎΠ΄Π΅Π»ΠΈ <a class="footnote-reference" href="#django" id="footnote-reference-5">[3]</a> ΠΈ Π½Π°Π·Π²Π°Π½Ρ Π±Π΅Π· ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΡ "ΠΌΠ°Π³ΠΈΡΠ΅ΡΠΊΠΈΡ
ΡΠΈΡΠ΅Π»"
<a class="footnote-reference" href="#mcconnell" id="footnote-reference-6">[1]</a>. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Task</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">UNCERTAIN_STATUS</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">DONE_STATUS</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">STOP_STATUS</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">WAIT_STATUS</span> <span class="o">=</span> <span class="mi">3</span>
<span class="n">WORK_STATUS</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">STATUS_CHOICES</span> <span class="o">=</span> <span class="p">(</span>
<span class="p">(</span><span class="n">DONE_STATUS</span><span class="p">,</span> <span class="s1">'done'</span><span class="p">),</span>
<span class="p">(</span><span class="n">STOP_STATUS</span><span class="p">,</span> <span class="s1">'stop'</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">status</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">IntegerField</span><span class="p">(</span><span class="n">choices</span><span class="o">=</span><span class="n">STATUS_CHOICES</span><span class="p">,</span>
<span class="n">default</span><span class="o">=</span><span class="n">UNCERTAIN_STATUS</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="section-3">
<h3>ΠΠ΅ΡΠΎΠ΄Ρ ΠΌΠΎΠ΄Π΅Π»ΠΈ</h3>
<p>Π Π°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊ ΠΌΠΎΠΆΠ΅Ρ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΠΈΡΡ Π² ΠΌΠΎΠ΄Π΅Π»ΠΈ ΡΠ²ΠΎΠΈ ΡΠΎΠ±ΡΡΠ²Π΅Π½Π½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΈ ΡΠ΅ΠΌ ΡΠ°ΠΌΡΠΌ
Π½Π°Π΄Π΅Π»ΡΡΡ ΡΠ²ΠΎΠΈ ΠΎΠ±ΡΠ΅ΠΊΡΡ Π΄ΠΎΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»ΡΠ½ΠΎΠΉ ΡΡΠ½ΠΊΡΠΈΠΎΠ½Π°Π»ΡΠ½ΠΎΡΡΡΡ Π½Π° ΡΡΠΎΠ²Π½Π΅ ΡΡΡΠΎΠΊ. ΠΠ΅ΡΠΎΠ΄Ρ
ΠΌΠΎΠ΄Π΅Π»ΠΈ ΠΏΠΎΠ΄Ρ
ΠΎΠ΄ΡΡ Π΄Π»Ρ ΠΈΠ½ΠΊΠ°ΠΏΡΡΠ»ΡΡΠΈΠΈ Π²ΡΠ΅ΠΉ Π±ΠΈΠ·Π½Π΅Ρ-Π»ΠΎΠ³ΠΈΠΊΠΈ Π² ΠΎΠ΄Π½ΠΎΠΌ ΠΌΠ΅ΡΡΠ΅. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ,
ΡΡΠ°Π³ΠΌΠ΅Π½Ρ ΠΌΠΎΠ΄Π΅Π»ΠΈ <em>Π¦Π΅ΠΏΠΎΡΠΊΠ°</em>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Chain</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="sd">"""Π¦Π΅ΠΏΠΎΡΠΊΠ° Π·Π°Π΄Π°Ρ."""</span>
<span class="c1"># ΠΡΠΎΠΏΡΡΠ΅Π½Ρ Π°ΡΡΠΈΠ±ΡΡΡ ΠΌΠΎΠ΄Π΅Π»ΠΈ Π΄Π»Ρ ΡΠΊΠΎΠ½ΠΎΠΌΠΈΠΈ ΠΌΠ΅ΡΡΠ°.</span>
<span class="c1"># Default manager.</span>
<span class="n">objects</span> <span class="o">=</span> <span class="n">PassThroughManager</span><span class="o">.</span><span class="n">for_queryset_class</span><span class="p">(</span><span class="n">ChainQuerySet</span><span class="p">)()</span>
<span class="k">def</span> <span class="nf">actual_status</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""ΠΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΡΠ°ΡΡΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ."""</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_date</span> <span class="o">></span> <span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">():</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">WAIT_STATUS</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_set</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">status</span><span class="o">=</span><span class="n">Task</span><span class="o">.</span><span class="n">STOP_STATUS</span><span class="p">)</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">STOP_STATUS</span>
<span class="n">last_task</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_task</span><span class="p">()</span>
<span class="k">if</span> <span class="n">last_task</span><span class="o">.</span><span class="n">actual_status</span><span class="p">()</span> <span class="o">==</span> <span class="n">Task</span><span class="o">.</span><span class="n">DONE_STATUS</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">DONE_STATUS</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">WORK_STATUS</span>
</pre></div>
<p>Π‘ΠΎΠ³Π»Π°ΡΠ½ΠΎ ΡΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½Π½ΠΎΠΉ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ΅ ΡΠΈΡΡΠ΅ΠΌΡ Π±ΡΠ»ΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠ°Π½Ρ ΡΠ»Π΅Π΄ΡΡΡΠΈΠ΅
ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΌΠΎΠ΄Π΅Π»ΠΈ Π΄Π°Π½Π½ΡΡ
<em>ΠΠ°Π΄Π°ΡΠ°</em>:</p>
<ul class="simple">
<li><em>be_in_time</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ, ΡΡΠΏΠ΅Π²Π°Π΅Ρ Π»ΠΈ Π·Π°Π΄Π°ΡΠ° ΠΊ Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ;</li>
<li><em>actual_status</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΡΠ°ΡΡΡ Π·Π°Π΄Π°ΡΠΈ, ΠΎΡΠ½ΠΎΠ²ΡΠ²Π°ΡΡΡ Π½Π° ΡΠ°ΠΊΠΈΡ
Π΄Π°Π½Π½ΡΡ
, ΠΊΠ°ΠΊ Β«ΡΡΠ°ΡΠΈΡΠ΅ΡΠΊΠΈΠΉΒ» ΡΡΠ°ΡΡΡ Π·Π°Π΄Π°ΡΠΈ (DONE, STOP), ΡΡΠ°ΡΡΡ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ
Π·Π°Π΄Π°ΡΠΈ, ΠΏΠΎΡΡΠ΄ΠΊΠΎΠ²ΡΠΉ Π½ΠΎΠΌΠ΅Ρ Π·Π°Π΄Π°ΡΠΈ ΠΈ ΡΠ΅ΠΊΡΡΠ°Ρ Π΄Π°ΡΠ°;</li>
<li><em>start_date</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ, Π΅ΡΠ»ΠΈ ΡΡΠΎ Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΠΎ;</li>
<li><em>days_to_start</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄
Π·Π°Π΄Π°ΡΠ΅ΠΉ;</li>
<li><em>remaining_days</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ ΠΏΠΎΠ»Π½ΡΡ
Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°.
ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, Π·Π°Π΄Π°ΡΠ° ΠΎΠ³ΡΠ°Π½ΠΈΡΠ΅Π½Π° ΡΡΠΎΠΊΠΎΠΌ <tt class="docutils literal">[26; 29)</tt> ΠΈ ΡΠ΅ΠΉΡΠ°Ρ <em>27 ΡΠΈΡΠ»ΠΎ</em>. ΠΠΎ
Π΄Π΅Π΄Π»Π°ΠΉΠ½Π° ΠΎΡΡΠ°Π»ΡΡ ΠΎΠ΄ΠΈΠ½ ΠΏΠΎΠ»Π½ΡΠΉ Π΄Π΅Π½Ρ <em>(28 ΡΠΈΡΠ»ΠΎ)</em>, ΡΠ°ΠΊ ΠΊΠ°ΠΊ ΡΠ΅ΠΊΡΡΠΈΠΉ Π΄Π΅Π½Ρ Π½Π΅
ΡΡΠΈΡΡΠ²Π°Π΅ΡΡΡ;</li>
<li><em>days_quantity_after_deadline</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π½Π° ΠΊΠΎΡΠΎΡΡΠ΅
ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π° Π·Π°Π΄Π°ΡΠ°;</li>
<li><em>expended_days</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π·Π°ΡΡΠ°ΡΠ΅Π½Π½ΡΡ
Π½Π° Π·Π°Π΄Π°ΡΡ;</li>
<li><em>duration</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π²ΡΠ΄Π΅Π»Π΅Π½Π½ΡΡ
Π½Π° Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΠ΅ Π·Π°Π΄Π°ΡΠΈ.</li>
</ul>
<p>ΠΠ»Ρ ΠΌΠΎΠ΄Π΅Π»ΠΈ <em>Π¦Π΅ΠΏΠΎΡΠΊΠ°</em> Π±ΡΠ»ΠΈ ΡΠ΅Π°Π»ΠΈΠ·ΠΎΠ²Π°Π½Ρ ΠΌΠ΅ΡΠΎΠ΄Ρ:</p>
<ul class="simple">
<li><em>actual_status</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΡΠ°ΡΡΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ, ΠΎΡΠ½ΠΎΠ²ΡΠ²Π°ΡΡΡ Π½Π° ΡΠ°ΠΊΠΈΡ
Π΄Π°Π½Π½ΡΡ
, ΠΊΠ°ΠΊ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ, ΡΡΠ°ΡΡΡ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠ΅ΠΏΠΎΡΠΊΠΈ;</li>
<li><em>deadline</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΡΠ΅ΠΏΠΎΡΠΊΠΈ. ΠΠ΅Π΄Π»Π°ΠΉΠ½ ΡΠ΅ΠΏΠΎΡΠΊΠΈ ΡΠ°Π²Π΅Π½ Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ
ΠΏΠΎΡΠ»Π΅Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅;</li>
<li><em>finish_date</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΄Π°ΡΡ Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ. ΠΠ°ΡΠ° Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ
ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΠΏΠΎΡΠ»Π΅Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅;</li>
<li><em>be_in_time</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ, ΡΡΠΏΠ΅Π²Π°Π΅Ρ Π»ΠΈ ΡΠ΅ΠΏΠΎΡΠΊΠ° Π·Π°Π΄Π°Ρ ΠΊ Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ;</li>
<li><em>days_to_start</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ
ΡΠ΅ΠΏΠΎΡΠΊΠΈ;</li>
<li><em>remaining_days</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π° ΡΠ΅ΠΏΠΎΡΠΊΠΈ.
Π‘ΠΎΠ²ΠΏΠ°Π΄Π°Π΅Ρ Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎΠΌ Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π° ΠΏΠΎΡΠ»Π΅Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ Π²
ΡΠ΅ΠΏΠΎΡΠΊΠ΅;</li>
<li><em>days_quantity_after_deadline</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π½Π° ΠΊΠΎΡΠΎΡΡΠ΅
ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π° ΡΠ΅ΠΏΠΎΡΠΊΠ°;</li>
<li><em>expended_days</em> ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π·Π°ΡΡΠ°ΡΠ΅Π½Π½ΡΡ
Π½Π° ΡΠ΅ΠΏΠΎΡΠΊΡ;</li>
<li><em>last_task</em> Π²ΠΎΠ·Π²ΡΠ°ΡΠ°Π΅Ρ ΠΏΠΎΡΠ»Π΅Π΄Π½ΡΡ Π·Π°Π΄Π°ΡΡ ΠΈΠ· ΡΠ΅ΠΏΠΎΡΠΊΠΈ.</li>
</ul>
</div>
<div class="section" id="section-4">
<h3>ΠΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ ΠΌΠΎΠ΄Π΅Π»ΠΈ</h3>
<p>ΠΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ ΠΌΠΎΠ΄Π΅Π»ΠΈ -- ΡΡΠΎ ΠΎΠ±ΡΠ΅ΠΊΡ, Ρ ΠΏΠΎΠΌΠΎΡΡΡ ΠΊΠΎΡΠΎΡΠΎΠ³ΠΎ Django Π²ΡΠΏΠΎΠ»Π½ΡΠ΅Ρ Π·Π°ΠΏΡΠΎΡΡ ΠΊ
ΠΠ. ΠΠ°ΠΆΠ΄Π°Ρ ΠΌΠΎΠ΄Π΅Π»Ρ Django ΠΈΠΌΠ΅Π΅Ρ ΠΏΠΎ ΠΌΠ΅Π½ΡΡΠ΅ΠΉ ΠΌΠ΅ΡΠ΅ ΠΎΠ΄ΠΈΠ½ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ, ΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊ
ΠΌΠΎΠΆΠ΅Ρ ΡΠΎΠ·Π΄Π°Π²Π°ΡΡ ΡΠ²ΠΎΠΈ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΡ Π΄Π»Ρ ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΠΈ ΡΠΏΠ΅ΡΠΈΠ°Π»ΠΈΠ·ΠΈΡΠΎΠ²Π°Π½Π½ΡΡ
Π²ΠΈΠ΄ΠΎΠ²
Π΄ΠΎΡΡΡΠΏΠ°.</p>
<p>ΠΠΎΡΡΠ΅Π±Π½ΠΎΡΡΡ ΡΠΎΠ·Π΄Π°Π½ΠΈΡ ΡΠΎΠ±ΡΡΠ²Π΅Π½Π½ΠΎΠ³ΠΎ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΠ° ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ Π²ΡΠ·Π²Π°Π½Π° Π΄Π²ΡΠΌΡ
ΠΏΡΠΈΡΠΈΠ½Π°ΠΌΠΈ: Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎΡΡΡΡ Π΄ΠΎΠ±Π°Π²ΠΈΡΡ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΡ Π΄ΠΎΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»ΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΈΠ»ΠΈ
Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎΡΡΡΡ ΠΌΠΎΠ΄ΠΈΡΠΈΡΠΈΡΠΎΠ²Π°ΡΡ ΠΈΡΡ
ΠΎΠ΄Π½ΡΠΉ ΠΎΠ±ΡΠ΅ΠΊΡ QuerySet, Π²ΠΎΠ·Π²ΡΠ°ΡΠ°Π΅ΠΌΡΠΉ
ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΠΎΠΌ <a class="footnote-reference" href="#holovaty" id="footnote-reference-7">[2]</a>.</p>
<p>ΠΡΡΡΠΎΠ΅Π½Π½ΡΠΉ Π² Django ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ ΠΌΠΎΠ΄Π΅Π»Π΅ΠΉ Π½Π΅ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΡΡΡΠΎΠΈΡΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ ΠΌΠ΅ΡΠΎΠ΄ΠΎΠ²,
ΡΠ°ΠΊΠΈΡ
ΠΊΠ°ΠΊ <tt class="docutils literal">actual_tasks = <span class="pre">Task.objects.by_worker(user).actual()</span></tt>. ΠΠ»Ρ ΠΎΠ±Ρ
ΠΎΠ΄Π°
ΡΡΠΎΠ³ΠΎ ΠΎΠ³ΡΠ°Π½ΠΈΡΠ΅Π½ΠΈΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΡΡΠΎΡΠΎΠ½Π½ΡΡ Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠ° <em>django-model-utils</em>,
ΠΊΠΎΡΠΎΡΠ°Ρ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΠΏΠΈΡΠ°ΡΡ QuerySet Π²ΠΌΠ΅ΡΡΠΎ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅ΡΠ° <a class="footnote-reference" href="#korobov" id="footnote-reference-8">[4]</a>. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">from</span> <span class="nn">django.db.models.query</span> <span class="kn">import</span> <span class="n">QuerySet</span>
<span class="k">class</span> <span class="nc">ChainQuerySet</span><span class="p">(</span><span class="n">QuerySet</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">by_owner</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">owner</span><span class="p">):</span>
<span class="sd">"""ΠΠΎΠ·Π²ΡΠ°ΡΠ°Π΅Ρ ΡΠ΅ΠΏΠΎΡΠΊΠΈ Π²Π»Π°Π΄Π΅Π»ΡΡΠ°."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">owner</span><span class="o">=</span><span class="n">owner</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">actual</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""ΠΠΎΠ·Π²ΡΠ°ΡΠ°Π΅Ρ Π°ΠΊΡΡΠ°Π»ΡΡΡΠ½ΡΠ΅ ΡΠ΅ΠΏΠΎΡΠΊΠΈ Π·Π°Π΄Π°Ρ."""</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">archive</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="s1">'start_date'</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="section-5">
<h3>ΠΠΈΠ³ΡΠ°ΡΠΈΡ ΡΡ
Π΅ΠΌΡ ΠΌΠΎΠ΄Π΅Π»ΠΈ</h3>
<p>ΠΠΎ Π²ΡΠ΅ΠΌΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π°ΡΡΡΠΏΠ°Π΅Ρ ΠΌΠΎΠΌΠ΅Π½Ρ, ΠΊΠΎΠ³Π΄Π° Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ
ΠΈΠ·ΠΌΠ΅Π½ΠΈΡΡ ΡΡ
Π΅ΠΌΡ ΠΌΠΎΠ΄Π΅Π»ΠΈ Π΄Π°Π½Π½ΡΡ
, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, Π΄ΠΎΠ±Π°Π²ΠΈΡΡ Π½ΠΎΠ²ΠΎΠ΅ ΠΏΠΎΠ»Π΅. ΠΡΠ»ΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ°
Π²Π΅Π΄Π΅ΡΡΡ Π² Π³ΡΡΠΏΠΏΠ΅, ΡΠΎ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ° ΡΡΡΠ³ΡΠ±Π»ΡΠ΅ΡΡΡ ΡΠ΅ΠΌ, ΡΡΠΎ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ
ΡΠΈΠ½Ρ
ΡΠΎΠ½ΠΈΠ·ΠΈΡΠΎΠ²Π°ΡΡ ΠΌΠΎΠ΄Π΅Π»Ρ. ΠΡΡ ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ ΠΏΡΠΈΠ·Π²Π°Π½Π° ΡΠ΅ΡΠΈΡΡ Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠ° South. ΠΠ΅
ΠΎΡΠ½ΠΎΠ²Π½ΡΠΌΠΈ Π·Π°Π΄Π°ΡΠ°ΠΌΠΈ ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΎΠ±Π΅ΡΠΏΠ΅ΡΠ΅Π½ΠΈΠ΅ ΠΏΡΠΎΡΡΠΎΠ³ΠΎ, ΡΡΠ°Π±ΠΈΠ»ΡΠ½ΠΎΠ³ΠΎ ΠΈ Π½Π΅Π·Π°Π²ΠΈΡΠΈΠΌΠΎΠ³ΠΎ
ΠΎΡ ΠΠ ΡΠ»ΠΎΡ ΠΌΠΈΠ³ΡΠ°ΡΠΈΠΈ, ΡΡΠΎΠ±Ρ ΠΈΠ·Π±Π°Π²ΠΈΡΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ° ΠΎΡ ΠΏΡΠΎΠ±Π»Π΅ΠΌ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΡ ΡΡ
Π΅ΠΌΡ.</p>
<p>Π Π°ΡΡΠΌΠΎΡΡΠΈΠΌ ΡΠΈΠΏΠΎΠ²ΡΠ΅ ΠΏΡΠΈΠΌΠ΅ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Ρ South <a class="footnote-reference" href="#south" id="footnote-reference-9">[5]</a>. Π‘Π»ΡΡΠ°ΠΉ, ΠΊΠΎΠ³Π΄Π° Π²
Π±Π°Π·Π΅ Π΄Π°Π½Π½ΡΡ
Π½Π΅Ρ ΡΠ°Π±Π»ΠΈΡ ΠΈ Π½Π΅Ρ ΡΠ°ΠΉΠ»ΠΎΠ² ΠΌΠΈΠ³ΡΠ°ΡΠΈΠΉ -- ΡΠΎΡΡΠΎΡΠ½ΠΈΠ΅ ΠΏΡΠΎΠ΅ΠΊΡΠ° ΡΡΠ°Π·Ρ ΠΏΠΎΡΠ»Π΅
Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ ΠΊΠΎΠΌΠ°Π½Π΄Π° <tt class="docutils literal">manage.py startapp myapp</tt>. ΠΠ°Π»Π΅Π΅, Π²ΠΌΠ΅ΡΡΠΎ ΠΊΠΎΠΌΠ°Π½Π΄Ρ
<tt class="docutils literal">manage.py syncdb</tt> Π½ΡΠΆΠ½ΠΎ ΡΠΎΠ·Π΄Π°ΡΡ Π½Π°ΡΠ°Π»ΡΠ½ΡΡ ΠΌΠΈΠ³ΡΠ°ΡΠΈΡ ΠΊΠΎΠΌΠ°Π½Π΄ΠΎΠΉ <tt class="docutils literal">manage.py
schemamigration myapp <span class="pre">--initial</span></tt> ΠΈ ΠΏΡΠΈΠΌΠ΅Π½ΠΈΡΡ ΠΌΠΈΠ³ΡΠ°ΡΠΈΡ ΠΊΠΎΠΌΠ°Π½Π΄ΠΎΠΉ
<tt class="docutils literal">manage.py migrate myapp</tt>.</p>
<p>Π‘Π»Π΅Π΄ΡΡΡΠΈΠΉ Π²Π°ΡΠΈΠ°Π½Ρ, ΠΊΠΎΠ³Π΄Π° ΡΠ°Π±Π»ΠΈΡΡ ΡΠΆΠ΅ ΡΠΎΠ·Π΄Π°Π½Ρ, Π½ΠΎ Π½Π΅Ρ ΡΠ°ΠΉΠ»ΠΎΠ² ΠΌΠΈΠ³ΡΠ°ΡΠΈΠΉ. Π Π΄Π°Π½Π½ΠΎΠΉ
ΡΠΈΡΡΠ°ΡΠΈΠΈ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ Π²ΡΠΏΠΎΠ»Π½ΠΈΡΡ ΠΊΠΎΠΌΠ°Π½Π΄Ρ <tt class="docutils literal">manage.py convert_to_south myapp</tt>.</p>
<p>ΠΠΎΠ·ΠΌΠΎΠΆΠ½Π° ΡΠΈΡΡΠ°ΡΠΈΡ, ΠΊΠΎΠ³Π΄Π° ΡΠ°Π±Π»ΠΈΡΡ ΡΠΆΠ΅ ΡΠΎΠ·Π΄Π°Π½Ρ, Π½ΠΎ ΠΌΠΈΠ³ΡΠ°ΡΠΈΠΈ Π΅ΡΠ΅ Π½Π΅ ΠΏΡΠΈΠΌΠ΅Π½Π΅Π½Ρ.
Π’ΠΎΠ³Π΄Π° Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ Π²ΡΠΏΠΎΠ»Π½ΠΈΡΡ <tt class="docutils literal">manage.py migrate myapp 0001 <span class="pre">--fake</span></tt>.</p>
</div>
</div>
<div class="section" id="section-6">
<h2>Π¨Π°Π±Π»ΠΎΠ½</h2>
<p>Π¨Π°Π±Π»ΠΎΠ½ -- ΡΡΠΎ ΡΠ΅ΠΊΡΡΠΎΠ²ΡΠΉ Π΄ΠΎΠΊΡΠΌΠ΅Π½Ρ ΠΈΠ»ΠΈ ΡΡΡΠΎΠΊΠ° Python, ΠΊΠΎΡΠΎΡΡΠΉ ΡΠ°Π·ΠΌΠ΅ΡΠ΅Π½ Ρ
ΠΏΡΠΈΠΌΠ΅Π½Π΅Π½ΠΈΠ΅ΠΌ ΡΠ·ΡΠΊΠ° ΡΠ°Π±Π»ΠΎΠ½ΠΎΠ² Django. Π¨Π°Π±Π»ΠΎΠ½ ΠΌΠΎΠΆΠ΅Ρ ΡΠΎΠ΄Π΅ΡΠΆΠ°ΡΡ ΡΠ°Π±Π»ΠΎΠ½Π½ΡΠ΅ ΡΠ΅Π³ΠΈ ΠΈ
ΡΠ°Π±Π»ΠΎΠ½Π½ΡΠ΅ ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΡΠ΅.</p>
<p>ΠΡΠΈ Π²ΡΠ±ΠΎΡΠ΅ ΠΌΠ΅ΡΡΠ° Ρ
ΡΠ°Π½Π΅Π½ΠΈΡ ΡΠ°Π±Π»ΠΎΠ½ΠΎΠ² Π² ΠΌΠ½ΠΎΠ³ΠΎΡΠ°Π·ΠΎΠ²ΡΡ
Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡΡ
ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅ΡΡΡ <a class="footnote-reference" href="#lincolnloop" id="footnote-reference-10">[6]</a> ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ ΡΠ»Π΅Π΄ΡΡΡΠΈΠΉ ΠΏΡΡΡ:
<tt class="docutils literal"><span class="pre">ΠΊΠΎΡΠ΅Π½Ρ-ΡΠ΅ΠΏΠΎΠ·ΠΈΡΠΎΡΠΈΡ/Π½Π°Π·Π²Π°Π½ΠΈΠ΅_ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ/templates/Π½Π°Π·Π²Π°Π½ΠΈΠ΅_ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ/</span>
Π½Π°Π·Π²Π°Π½ΠΈΠ΅_ΡΠ°Π±Π»ΠΎΠ½Π°</tt>. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal"><span class="pre">django-todo/todo/templates/todo/base.html</span></tt>.</p>
<p>ΠΠ°Π·Π²Π°Π½ΠΈΠ΅ ΡΠ°Π±Π»ΠΎΠ½ΠΎΠ² ΡΠ»Π΅Π΄ΡΠ΅Ρ Π²ΡΠ±ΠΈΡΠ°ΡΡ ΠΏΡΠΈΠ΄Π΅ΡΠΆΠΈΠ²Π°ΡΡΡ ΡΠ»Π΅Π΄ΡΡΡΠ΅ΠΉ ΠΊΠΎΠ½Π²Π΅Π½ΡΠΈΠΈ
<tt class="docutils literal"><span class="pre">[model]_[function].html</span></tt>, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">task_list.html</tt>. ΠΡΠ½ΡΠ΄Ρ Π½Π΅ ΠΊΠ°ΠΆΠ΄ΠΎΠ΅
Π½Π°Π·Π²Π°Π½ΠΈΠ΅ ΡΠ°Π±Π»ΠΎΠ½Π°, ΠΏΠΎΠ»ΡΡΠ΅Π½Π½ΠΎΠ΅ Π² ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΠΈΠΈ Ρ ΠΊΠΎΠ½Π²Π΅Π½ΡΠΈΠ΅ΠΉ, ΠΏΠΎΠ»ΡΡΠ°Π΅ΡΡΡ
ΠΏΠΎΠ΄Ρ
ΠΎΠ΄ΡΡΠΈΠΌ. Π ΡΠ°ΠΊΠΈΡ
ΡΠ»ΡΡΠ°ΡΡ
ΡΠ»Π΅Π΄ΡΠ΅Ρ Π²ΡΠ±ΠΈΡΠ°ΡΡ Π½Π°Π·Π²Π°Π½ΠΈΠ΅ ΠΏΠΎ ΡΠ²ΠΎΠ΅ΠΌΡ ΡΡΠΌΠΎΡΡΠ΅Π½ΠΈΡ.</p>
<p>Π¨Π°Π±Π»ΠΎΠ½Π½ΡΠΉ ΡΠ΅Π³ -- ΡΡΠΎ Π½Π΅ΠΊΠΎΡΠΎΡΠΎΠ΅ ΠΎΠ±ΠΎΠ·Π½Π°ΡΠ΅Π½ΠΈΠ΅ Π² ΡΠ°Π±Π»ΠΎΠ½Π΅, Ρ ΠΊΠΎΡΠΎΡΡΠΌ Π°ΡΡΠΎΡΠΈΠΈΡΠΎΠ²Π°Π½Π°
ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠ½Π°Ρ Π»ΠΎΠ³ΠΈΠΊΠ°. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, ΡΠ°Π±Π»ΠΎΠ½Π½ΡΠΉ ΡΠ΅Π³ ΠΌΠΎΠΆΠ΅Ρ ΠΏΠΎΡΠΎΠΆΠ΄Π°ΡΡ ΡΠΎΠ΄Π΅ΡΠΆΠΈΠΌΠΎΠ΅,
Π²ΡΡΡΡΠΏΠ°ΡΡ Π² ΡΠΎΠ»ΠΈ ΡΠΏΡΠ°Π²Π»ΡΡΡΠ΅ΠΉ ΠΊΠΎΠ½ΡΡΡΡΠΊΡΠΈΠΈ, ΠΏΠΎΠ»ΡΡΠ°ΡΡ ΡΠΎΠ΄Π΅ΡΠΆΠΈΠΌΠΎΠ΅ ΠΈΠ· Π±Π°Π·Ρ Π΄Π°Π½Π½ΡΡ
ΠΈΠ»ΠΈ ΡΠ°Π·ΡΠ΅ΡΠ°ΡΡ Π΄ΠΎΡΡΡΠΏ ΠΊ Π΄ΡΡΠ³ΠΈΠΌ ΡΠ°Π±Π»ΠΎΠ½Π½ΡΠΌ ΡΠ΅Π³Π°ΠΌ.</p>
<p>Π¨Π°Π±Π»ΠΎΠ½Ρ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»ΡΡΠΊΠΈΡ
ΡΠ΅Π³ΠΎΠ² ΠΈ ΡΠ°ΡΡΠΈΡΠ½ΡΠ΅ ΡΠ°Π±Π»ΠΎΠ½Ρ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅ΡΡΡ Ρ
ΡΠ°Π½ΠΈΡΡ Π²
Π΄ΠΈΡΠ΅ΠΊΡΠΎΡΠΈΠΈ <tt class="docutils literal">includes</tt>. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, ΠΎΡΠΎΠ±ΡΠ°ΠΆΠ΅Π½ΠΈΠ΅ ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠΈ ΠΎ ΡΠ΅ΠΏΠΎΡΠΊΠ΅ Π·Π°Π΄Π°Ρ
Π²ΡΠ½Π΅ΡΠ΅Π½ΠΎ Π² ΡΠ°Π±Π»ΠΎΠ½ <tt class="docutils literal">includes/chain.html</tt>.</p>
<p>ΠΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»ΡΡΠΊΠΈΠ΅ ΡΠ΅Π³ΠΈ ΡΠ°Π±Π»ΠΎΠ½Π° ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅ΡΡΡ Ρ
ΡΠ°Π½ΠΈΡΡ Π² ΠΊΠ°ΡΠ°Π»ΠΎΠ³Π΅:
<tt class="docutils literal"><span class="pre">ΠΊΠΎΡΠ΅Π½Ρ-ΡΠ΅ΠΏΠΎΠ·ΠΈΡΠΎΡΠΈΡ/Π½Π°Π·Π²Π°Π½ΠΈΠ΅_ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ/templatetags/</span>
[Π½Π°Π·Π²Π°Π½ΠΈΠ΅_ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ]_tags.py</tt>. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal"><span class="pre">django-todo/todo/templatetags/</span>
todo_tags.py</tt>.</p>
</div>
<div class="section" id="url">
<h2>ΠΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΡ ΠΈ ΠΊΠΎΠ½ΡΠΈΠ³ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ URL</h2>
<p>ΠΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΠ΅ -- ΡΡΠ½ΠΊΡΠΈΡ Π½Π° ΡΠ·ΡΠΊΠ΅ Python, ΠΊΠΎΡΠΎΡΠ°Ρ ΠΏΡΠΈΠ½ΠΈΠΌΠ°Π΅Ρ ΡΠΊΠ·Π΅ΠΌΠΏΠ»ΡΡ ΠΊΠ»Π°ΡΡΠ°
<em>HttpRequest</em> Π² ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅ ΠΏΠ΅ΡΠ²ΠΎΠ³ΠΎ ΠΏΠ°ΡΠ°ΠΌΠ΅ΡΡΠ° ΠΈ Π²ΠΎΠ·Π²ΡΠ°ΡΠ°Π΅Ρ ΡΠΊΠ·Π΅ΠΌΠΏΠ»ΡΡ ΠΊΠ»Π°ΡΡΠ°
<em>HttpResponse</em>. ΠΠΈΠΆΠ΅ ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½ ΠΊΠΎΠ΄ ΡΡΠ½ΠΊΡΠΈΠΈ Π²ΠΌΠ΅ΡΡΠ΅ Ρ ΠΊΠΎΠΌΠ°Π½Π΄Π°ΠΌΠΈ ΠΈΠΌΠΏΠΎΡΡΠ° ΠΈΠ· ΡΠ°ΠΉΠ»Π°
<tt class="docutils literal">views.py</tt>:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">render</span><span class="p">,</span> <span class="n">get_object_or_404</span>
<span class="kn">from</span> <span class="nn">django.contrib.auth.decorators</span> <span class="kn">import</span> <span class="n">login_required</span>
<span class="kn">from</span> <span class="nn">todo.models</span> <span class="kn">import</span> <span class="n">Chain</span><span class="p">,</span> <span class="n">Task</span>
<span class="nd">@login_required</span>
<span class="k">def</span> <span class="nf">actual_tasks</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="sd">"""ΠΡΠΎΠ±ΡΠ°ΠΆΠ°Π΅Ρ ΡΠΏΠΈΡΠΎΠΊ Π°ΠΊΡΡΠ°Π»ΡΠ½ΡΡ
Π·Π°Π΄Π°Ρ Π΄Π»Ρ ΠΈΡΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»Ρ."""</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">user</span>
<span class="n">actual_tasks</span> <span class="o">=</span> <span class="n">Task</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">by_worker</span><span class="p">(</span><span class="n">user</span><span class="p">)</span><span class="o">.</span><span class="n">actual</span><span class="p">()</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s1">'todo/task_list.html'</span><span class="p">,</span> <span class="p">{</span>
<span class="s1">'place'</span><span class="p">:</span> <span class="s1">'tasks'</span><span class="p">,</span>
<span class="s1">'actual_tasks'</span><span class="p">:</span> <span class="n">actual_tasks</span><span class="p">,</span>
<span class="p">})</span>
</pre></div>
<p>Π§ΡΠΎΠ±Ρ ΡΠ²ΡΠ·Π°ΡΡ ΡΡΠ½ΠΊΡΠΈΡ ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΡ Ρ URL, Π² Django ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΠΌΠ΅Ρ
Π°Π½ΠΈΠ·ΠΌ
ΠΊΠΎΠ½ΡΠΈΠ³ΡΡΠ°ΡΠΈΠΈ URL. Django ΠΎΠΆΠΈΠ΄Π°Π΅Ρ Π½Π°ΠΉΡΠΈ ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΡΡ <em>urlpatterns</em> Π² ΠΊΠΎΠ½ΡΠΈΠ³ΡΡΠ°ΡΠΈΠΈ
URL. ΠΠ½Π° ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΠΈΠ΅ ΠΌΠ΅ΠΆΠ΄Ρ URL-Π°Π΄ΡΠ΅ΡΠ°ΠΌΠΈ ΠΈ ΠΎΠ±ΡΠ°Π±Π°ΡΡΠ²Π°ΡΡΠΈΠΌ ΠΈΡ
ΠΊΠΎΠ΄ΠΎΠΌ.</p>
<p>ΠΠΎΡ ΠΊΠ°ΠΊ ΠΏΠΎΠ΄ΠΊΠ»ΡΡΠ°Π΅ΡΡΡ ΠΏΡΠ΅Π΄ΡΡΠ°Π²Π»Π΅Π½ΠΈΠ΅ <em>actual_tasks</em> ΠΈ <em>task_archive</em> Π² ΡΠ°ΠΉΠ»Π΅
<tt class="docutils literal">urls.py</tt>:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.conf.urls.defaults</span> <span class="kn">import</span> <span class="n">patterns</span><span class="p">,</span> <span class="n">url</span>
<span class="n">urlpatterns</span> <span class="o">=</span> <span class="n">patterns</span><span class="p">(</span><span class="s1">'todo.views'</span><span class="p">,</span>
<span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^$'</span><span class="p">,</span> <span class="s1">'actual_tasks'</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s1">'todo_actual_tasks'</span><span class="p">),</span>
<span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^task/archive/$'</span><span class="p">,</span> <span class="s1">'task_archive'</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s1">'todo_task_archive'</span><span class="p">),</span>
<span class="p">)</span>
</pre></div>
<p>ΠΡΠ±ΠΎΠΉ Π·Π°ΠΏΡΠΎΡ ΠΊ URL <tt class="docutils literal">/task/archive/</tt> Π΄ΠΎΠ»ΠΆΠ΅Π½ ΠΎΠ±ΡΠ°Π±Π°ΡΡΠ²Π°ΡΡΡΡ ΡΡΠ½ΠΊΡΠΈΠ΅ΠΉ
<em>task_archive</em>, Π° Π·Π°ΠΏΡΠΎΡ <tt class="docutils literal">/</tt> Π±ΡΠ΄Π΅Ρ ΠΎΠ±ΡΠ°Π±Π°ΡΡΠ²Π°ΡΡΡΡ <em>actual_tasks</em>.</p>
<p>ΠΠ°Π·Π²Π°Π½ΠΈΡ ΡΠ°Π±Π»ΠΎΠ½Π°ΠΌ URL ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅ΡΡΡ Π΄Π°Π²Π°ΡΡ Π² ΡΠΎΡΠΌΠ΅ <tt class="docutils literal">APP_MODEL_VIEW</tt>,
Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">blog_post_detail</tt> ΠΈΠ»ΠΈ <tt class="docutils literal">blog_post_list</tt>.</p>
<table class="docutils footnote" frame="void" id="mcconnell" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#footnote-reference-1">1</a>, <a class="fn-backref" href="#footnote-reference-2">2</a>, <a class="fn-backref" href="#footnote-reference-6">3</a>)</em> ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π» Π‘. Π‘ΠΎΠ²Π΅ΡΡΠ΅Π½Π½ΡΠΉ ΠΊΠΎΠ΄. ΠΠ°ΡΡΠ΅Ρ-ΠΊΠ»Π°ΡΡ /
ΠΠ΅Ρ. Ρ Π°Π½Π³Π». β Π. : ΠΠ·Π΄Π°ΡΠ΅Π»ΡΡΡΠ²ΠΎ "Π ΡΡΡΠΊΠ°Ρ ΡΠ΅Π΄Π°ΠΊΡΠΈΡ", 2012. β 896 ΡΡΡ. : ΠΈΠ».</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="holovaty" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[2]</td><td><em>(<a class="fn-backref" href="#footnote-reference-3">1</a>, <a class="fn-backref" href="#footnote-reference-7">2</a>)</em> ΠΠΎΠ»ΠΎΠ²Π°ΡΡΠΉ Π., ΠΠ°ΠΏΠ»Π°Π½-ΠΠΎΡΡ ΠΠΆ. Django. ΠΠΎΠ΄ΡΠΎΠ±Π½ΠΎΠ΅ ΡΡΠΊΠΎΠ²ΠΎΠ΄ΡΡΠ²ΠΎ,
2-Π΅ ΠΈΠ·Π΄Π°Π½ΠΈΠ΅. β ΠΠ΅Ρ. Ρ Π°Π½Π³Π». β Π‘ΠΠ±.: Π‘ΠΈΠΌΠ²ΠΎΠ»-ΠΠ»ΡΡ, 2010. β 560 Ρ., ΠΈΠ».</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="django" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[3]</td><td><em>(<a class="fn-backref" href="#footnote-reference-4">1</a>, <a class="fn-backref" href="#footnote-reference-5">2</a>)</em> Django community. <a class="reference external" href="https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/">Django Coding Style</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="korobov" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-8">[4]</a></td><td>ΠΠΎΡΠΎΠ±ΠΎΠ² Π. <a class="reference external" href="http://habrahabr.ru/post/142703/">Π Π΅ΡΠ΅ΠΏΡΡ ΠΎΡ ΠΠ°Π½ΠΡΡΠΌΠ°Π½Π°</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="south" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-9">[5]</a></td><td>Godwin A. <a class="reference external" href="http://south.readthedocs.org/en/latest/tutorial/index.html">South documentation</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="lincolnloop" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-10">[6]</a></td><td>Lincoln Loop company. <a class="reference external" href="http://lincolnloop.com/django-best-practices/">Django Best Practices</a>.</td></tr>
</tbody>
</table>
</div>
Django TODO: ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΡ ΡΠΈΡΡΠ΅ΠΌΡ2012-06-29T11:00:00+07:002012-06-29T11:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-29:/django-todo-design-architecture.html<p>Π‘Π»Π΅Π΄ΡΡΡΠΈΠΌ ΡΡΠ°ΠΏΠΎΠΌ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ ΡΠΈΡΡΠ΅ΠΌΡ ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΡ.</p>
<p>ΠΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ° Π΄ΠΎΠ»ΠΆΠ½Π° Π±ΡΡΡ ΠΏΡΠΎΠ΄ΡΠΌΠ°Π½Π½ΡΠΌ ΠΊΠΎΠ½ΡΠ΅ΠΏΡΡΠ°Π»ΡΠ½ΡΠΌ ΡΠ΅Π»ΡΠΌ. ΠΠ»Π°Π²Π½ΡΠΉ ΡΠ΅Π·ΠΈΡ ΡΠ°ΠΌΠΎΠΉ
ΠΏΠΎΠΏΡΠ»ΡΡΠ½ΠΎΠΉ ΠΊΠ½ΠΈΠ³ΠΈ ΠΏΠΎ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΠ "ΠΠΈΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΠ΅Π»ΠΎΠ²Π΅ΠΊΠΎ-ΠΌΠ΅ΡΡΡ" Π³Π»Π°ΡΠΈΡ, ΡΡΠΎ
ΠΎΡΠ½ΠΎΠ²Π½ΠΎΠΉ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠΎΠΉ, Ρ
Π°ΡΠ°ΠΊΡΠ΅ΡΠ½ΠΎΠΉ Π΄Π»Ρ ΠΊΡΡΠΏΠ½ΡΡ
ΡΠΈΡΡΠ΅ΠΌ, ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠ°Π½ΠΈΠ΅ ΠΈΡ
ΠΊΠΎΠ½ΡΠ΅ΠΏΡΡΠ°Π»ΡΠ½ΠΎΠΉ ΡΠ΅Π»ΠΎΡΡΠ½ΠΎΡΡΠΈ. Π₯ΠΎΡΠΎΡΠ°Ρ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ° Π΄ΠΎΠ»ΠΆΠ½Π° ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΠΎΠ²Π°ΡΡ
ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ΅ <a class="footnote-reference" href="#mcconnell" id="footnote-reference-1">[1]</a>.</p>
<p>Π Π°Π·Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΡΠΈΡΡΠ΅ΠΌΡ Π½Π° ΠΏΠΎΠ΄ΡΠΈΡΡΠ΅ΠΌΡ Π½Π° ΡΡΠΎΠ²Π½Π΅ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΡ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ β¦</p><p>Π‘Π»Π΅Π΄ΡΡΡΠΈΠΌ ΡΡΠ°ΠΏΠΎΠΌ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ ΡΠΈΡΡΠ΅ΠΌΡ ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΏΡΠΎΠ΅ΠΊΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΡ.</p>
<p>ΠΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ° Π΄ΠΎΠ»ΠΆΠ½Π° Π±ΡΡΡ ΠΏΡΠΎΠ΄ΡΠΌΠ°Π½Π½ΡΠΌ ΠΊΠΎΠ½ΡΠ΅ΠΏΡΡΠ°Π»ΡΠ½ΡΠΌ ΡΠ΅Π»ΡΠΌ. ΠΠ»Π°Π²Π½ΡΠΉ ΡΠ΅Π·ΠΈΡ ΡΠ°ΠΌΠΎΠΉ
ΠΏΠΎΠΏΡΠ»ΡΡΠ½ΠΎΠΉ ΠΊΠ½ΠΈΠ³ΠΈ ΠΏΠΎ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΠ "ΠΠΈΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΠ΅Π»ΠΎΠ²Π΅ΠΊΠΎ-ΠΌΠ΅ΡΡΡ" Π³Π»Π°ΡΠΈΡ, ΡΡΠΎ
ΠΎΡΠ½ΠΎΠ²Π½ΠΎΠΉ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠΎΠΉ, Ρ
Π°ΡΠ°ΠΊΡΠ΅ΡΠ½ΠΎΠΉ Π΄Π»Ρ ΠΊΡΡΠΏΠ½ΡΡ
ΡΠΈΡΡΠ΅ΠΌ, ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠ°Π½ΠΈΠ΅ ΠΈΡ
ΠΊΠΎΠ½ΡΠ΅ΠΏΡΡΠ°Π»ΡΠ½ΠΎΠΉ ΡΠ΅Π»ΠΎΡΡΠ½ΠΎΡΡΠΈ. Π₯ΠΎΡΠΎΡΠ°Ρ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΠ° Π΄ΠΎΠ»ΠΆΠ½Π° ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²ΠΎΠ²Π°ΡΡ
ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ΅ <a class="footnote-reference" href="#mcconnell" id="footnote-reference-1">[1]</a>.</p>
<p>Π Π°Π·Π΄Π΅Π»Π΅Π½ΠΈΠ΅ ΡΠΈΡΡΠ΅ΠΌΡ Π½Π° ΠΏΠΎΠ΄ΡΠΈΡΡΠ΅ΠΌΡ Π½Π° ΡΡΠΎΠ²Π½Π΅ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΡ, ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ
ΠΊΠΎΠ½ΡΠ΅Π½ΡΡΠΈΡΠΎΠ²Π°ΡΡΡΡ Π² ΠΊΠ°ΠΆΠ΄ΡΠΉ ΠΊΠΎΠ½ΠΊΡΠ΅ΡΠ½ΡΠΉ ΠΌΠΎΠΌΠ΅Π½Ρ Π²ΡΠ΅ΠΌΠ΅Π½ΠΈ Π½Π° ΠΌΠ΅Π½ΡΡΠ΅ΠΉ ΡΠ°ΡΡΠΈ ΡΠΈΡΡΠ΅ΠΌΡ.</p>
<p>ΠΠ±ΠΎΠ±ΡΠ΅Π½Π½Π°Ρ ΡΡ
Π΅ΠΌΠ° ΡΠ°Π±ΠΎΡΡ ΡΠΈΡΡΠ΅ΠΌΡ ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½Π° Π½Π° ΡΠΈΡΡΠ½ΠΊΠ΅.</p>
<img alt="" src="http://dl.dropbox.com/u/15875449/django_todo_block_schema.png" />
<p>Π Π°Π±ΠΎΡΠ° ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ Ρ ΡΠΈΡΡΠ΅ΠΌΠΎΠΉ Π½Π°ΡΠΈΠ½Π°Π΅ΡΡΡ Ρ Π°Π²ΡΠΎΡΠΈΠ·Π°ΡΠΈΠΈ. ΠΠΎΡΠ»Π΅ ΡΡΠΏΠ΅ΡΠ½ΠΎΠΉ
Π°ΡΡΠ΅Π½ΡΠΈΡΠΈΠΊΠ°ΡΠΈΠΈ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ ΠΏΠΎΠΏΠ°Π΄Π°Π΅Ρ Π½Π° ΡΡΡΠ°Π½ΠΈΡΡ Ρ Π·Π°Π΄Π°ΡΠ°ΠΌΠΈ, ΠΏΠΎΡΡΠ°Π²Π»Π΅Π½Π½ΡΠΌΠΈ Π΅ΠΌΡ
Π΄ΡΡΠ³ΠΈΠΌΠΈ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»ΡΠΌΠΈ. ΠΠ°Π»Π΅Π΅ ΠΎΠ½ ΠΌΠΎΠΆΠ΅Ρ ΠΏΠ΅ΡΠ΅ΠΉΡΠΈ Π² ΡΠ°Π·Π΄Π΅Π» ΠΏΡΠΎΠ΅ΠΊΡΠΎΠ², Π»ΠΈΠ±ΠΎ Π²
ΡΠ°Π·Π΄Π΅Π» "ΡΠΎΡΡΡΠ΄Π½ΠΈΠΊΠΈ".</p>
<div class="section" id="section-1">
<h2>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
</h2>
<p>Π£ΡΠΎΠ²Π΅Π½Ρ Π΄ΠΎΡΡΡΠΏΠ° ΠΊ Π΄Π°Π½Π½ΡΠΌ. ΠΠ° ΡΡΠΎΠΌΡ ΡΡΠΎΠ²Π½Π΅ ΡΠΎΡΡΠ΅Π΄ΠΎΡΠΎΡΠ΅Π½Π° Π²ΡΡ ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΡ ΠΎ
Π΄Π°Π½Π½ΡΡ
: ΠΊΠ°ΠΊ ΠΏΠΎΠ»ΡΡΠΈΡΡ ΠΊ Π½ΠΈΠΌ Π΄ΠΎΡΡΡΠΏ, ΠΊΠ°ΠΊ ΠΎΡΡΡΠ΅ΡΡΠ²Π»ΡΡΡ ΠΊΠΎΠ½ΡΡΠΎΠ»Ρ, ΠΊΠ°ΠΊΠΎΠ²ΠΎ ΠΈΡ
ΠΏΠΎΠ²Π΅Π΄Π΅Π½ΠΈΠ΅, ΠΊΠ°ΠΊΠΎΠ²Ρ ΠΎΡΠ½ΠΎΡΠ΅Π½ΠΈΡ ΠΌΠ΅ΠΆΠ΄Ρ Π΄Π°Π½Π½ΡΠΌΠΈ.</p>
<div class="section" id="section-2">
<h3>ΠΠΎΠ΄Π΅Π»Ρ "ΠΠ°Π΄Π°ΡΠ°"</h3>
<p>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
<em>ΠΠ°Π΄Π°ΡΠ°</em> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΡΠ»Π΅Π΄ΡΡΡΠΈΠ΅ Π°ΡΡΠΈΠ±ΡΡΡ:</p>
<ul>
<li><p class="first"><em>Π·Π°Π΄Π°ΡΠ°</em> -- ΠΎΠΏΠΈΡΠ°Π½ΠΈΠ΅ ΡΡΡΠΈ Π·Π°Π΄Π°ΡΠΈ. Π’Π΅ΠΊΡΡΠΎΠ²ΠΎΠ΅ ΠΏΠΎΠ»Π΅, ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ΅ Π΄Π»Ρ
Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</p>
</li>
<li><p class="first"><em>ΠΈΡΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»Ρ</em> -- ΠΈΠ΄Π΅Π½ΡΠΈΡΠΈΠΊΠ°ΡΠΎΡ ΡΠΎΡΡΡΠ΄Π½ΠΈΠΊΠ°. Π¦Π΅Π»ΠΎΡΠΈΡΠ»Π΅Π½Π½ΠΎΠ΅ ΠΏΠΎΠ»Π΅, ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ΅
Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</p>
</li>
<li><p class="first"><em>ΡΡΠ°ΡΡΡ</em> -- ΡΠ΅Π»ΠΎΡΠΈΡΠ»Π΅Π½Π½ΠΎΠ΅ ΠΏΠΎΠ»Π΅, ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ΅ Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ. ΠΠΎΠΆΠ΅Ρ ΠΏΡΠΈΠ½ΠΈΠΌΠ°ΡΡ
Π·Π½Π°ΡΠ΅Π½ΠΈΡ: UNCERTAIN (Π½Π΅ ΠΎΠΏΡΠ΅Π΄Π΅Π»Π΅Π½), DONE (Π·Π°Π²Π΅ΡΡΠ΅Π½), STOP (ΠΏΡΠ΅ΡΠ²Π°Π½), WAIT
(Π² ΠΎΠΆΠΈΠ΄Π°Π½ΠΈΠΈ), WORK (Π² ΡΠ°Π±ΠΎΡΠ΅). Π‘ΡΠ°ΡΡΡΡ WAIT ΠΈ WORK Π²ΡΡΠΈΡΠ»ΡΡΡΡΡ ΠΏΡΠΈ Π²ΡΠ±ΠΎΡΠΊΠ΅
ΠΈΠ· Π±Π°Π·Ρ Π΄Π°Π½Π½ΡΡ
Π½Π° ΠΎΡΠ½ΠΎΠ²Π΅ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΠΈ ΡΡΠ°ΡΡΡΠ° ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΠ°Π΄Π°ΡΠ°
ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ Π·Π°Π²Π΅ΡΡΠ΅Π½Π° (DONE) ΠΈΠ»ΠΈ ΠΏΡΠ΅ΡΠ²Π°Π½Π° (STOP) ΡΠΎΠ»ΡΠΊΠΎ ΠΈΠ· ΡΡΠ°ΡΡΡΠ° WORK.
Π‘ΡΠ°ΡΡΡ STOP Π½ΡΠΆΠ΅Π½ Π² ΡΠ»ΡΡΠ°Π΅, ΠΊΠΎΠ³Π΄Π° Π΄Π°Π»ΡΠ½Π΅ΠΉΡΠ΅Π΅ Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΠ΅ ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ
Π½Π΅Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΠΎ. Π‘ΠΈΠ³Π½Π°Π»ΠΈΠ·ΠΈΡΡΠ΅Ρ, ΡΡΠΎ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ Π²ΠΌΠ΅ΡΠ°ΡΠ΅Π»ΡΡΡΠ²ΠΎ ΡΡΠΊΠΎΠ²ΠΎΠ΄ΠΈΡΠ΅Π»Ρ
ΠΏΡΠΎΠ΅ΠΊΡΠ°. Π’ΠΎΠ»ΡΠΊΠΎ ΡΡΠΊΠΎΠ²ΠΎΠ΄ΠΈΡΠ΅Π»Ρ ΠΌΠΎΠΆΠ΅Ρ ΠΎΡΡΠ΅Π΄Π°ΠΊΡΠΈΡΠΎΠ²Π°ΡΡ ΠΏΡΠ΅ΡΠ²Π°Π½Π½ΡΡ Π·Π°Π΄Π°ΡΡ
(Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, ΠΈΠ·ΠΌΠ΅Π½ΠΈΡΡ Π΄Π΅Π΄Π»Π°ΠΉΠ½ Π·Π°Π΄Π°ΡΠΈ) ΠΈ Π²Π½ΠΎΠ²Ρ Π·Π°ΠΏΡΡΡΠΈΡΡ Π΅Π΅ Π½Π° Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΠ΅
(ΡΠΌΠ΅Π½ΠΈΡΡ ΡΡΠ°ΡΡΡ Π½Π° WORK). Π‘ΡΠ°ΡΡΡ Π·Π°Π΄Π°ΡΠΈ WAIT ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ Π² ΡΠ»ΡΡΠ°Π΅:</p>
<ul class="simple">
<li>Π΅ΡΠ»ΠΈ ΠΎΠ½Π° ΠΏΠ΅ΡΠ²Π°Ρ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅ ΠΈ Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ Π½Π΅ Π½Π°ΡΡΡΠΏΠΈΠ»Π°;</li>
<li>Π΅ΡΠ»ΠΈ ΡΡΠ°ΡΡΡ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ Π½Π΅ DONE.</li>
</ul>
<p>Π‘ΡΠ°ΡΡΡ Π·Π°Π΄Π°ΡΠΈ WORK ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ Π² ΡΠ»ΡΡΠ°Π΅:</p>
<ul class="simple">
<li>Π΅ΡΠ»ΠΈ ΠΎΠ½Π° ΠΏΠ΅ΡΠ²Π°Ρ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅ ΠΈ Π½Π°ΡΡΡΠΏΠΈΠ»Π° Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ;</li>
<li>Π΅ΡΠ»ΠΈ ΡΡΠ°ΡΡΡ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ DONE.</li>
</ul>
</li>
<li><p class="first"><em>Π΄Π΅Π΄Π»Π°ΠΉΠ½</em> -- ΠΊΡΠ°ΠΉΠ½ΡΡ Π΄Π°ΡΠ° Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ. ΠΠ΅Π΄Π»Π°ΠΉΠ½ Π·Π°Π΄Π°ΡΠΈ
Π΄ΠΎΠ»ΠΆΠ΅Π½ Π±ΡΡΡ Π±ΠΎΠ»ΡΡΠ΅, ΡΠ΅ΠΌ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</p>
</li>
<li><p class="first"><em>Π΄Π°ΡΠ° Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ</em> -- Π΄Π°ΡΠ° Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ. ΠΠΎΠ»Π΅ Π·Π°ΠΏΠΎΠ»Π½ΡΠ΅ΡΡΡ ΠΏΠΎ
Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ. ΠΠΎΠΆΠ΅Ρ Π½Π΅ ΡΠΎΠ²ΠΏΠ°Π΄Π°ΡΡ Ρ Π΄Π΅Π΄Π»Π°ΠΉΠ½ΠΎΠΌ;</p>
</li>
<li><p class="first"><em>ΡΠ΅ΠΏΠΎΡΠΊΠ°</em> -- ΠΈΠ΄Π΅Π½ΡΠΈΡΠΈΠΊΠ°ΡΠΎΡ ΡΠ΅ΠΏΠΈ, ΠΊ ΠΊΠΎΡΠΎΡΠΎΠΉ ΠΎΡΠ½ΠΎΡΠΈΡΡΡ Π·Π°Π΄Π°ΡΠ°. ΠΠΎΠ»Π΅
ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎ Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</p>
</li>
<li><p class="first"><em>ΠΏΠΎΡΡΠ΄ΠΊΠΎΠ²ΡΠΉ Π½ΠΎΠΌΠ΅Ρ</em> -- ΡΠ΅Π»ΠΎΡΠΈΡΠ»Π΅Π½Π½ΠΎΠ΅ ΠΏΠΎΠ»Π΅, Π±ΠΎΠ»ΡΡΠ΅ ΠΈΠ»ΠΈ ΡΠ°Π²Π½ΠΎ Π΅Π΄ΠΈΠ½ΠΈΡΠ΅,
ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ΅ Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ. ΠΠΎΠ»Π΅ Π²Π²Π΅Π΄Π΅Π½ΠΎ Π΄Π»Ρ ΡΠΎΡΡΠΈΡΠΎΠ²ΠΊΠΈ Π·Π°Π΄Π°Ρ Π² ΡΠ΅ΠΏΠΎΡΠΊΠ΅ ΠΈ
ΠΏΡΠΎΡΡΠΎΠ³ΠΎ ΡΠΏΠΎΡΠΎΠ±Π° ΠΏΠΎΠ»ΡΡΠ΅Π½ΠΈΡ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</p>
</li>
<li><p class="first"><em>Π°ΡΡ
ΠΈΠ²</em> -- ΡΠ»Π°Π³, ΠΏΠΎ ΠΊΠΎΡΠΎΡΠΎΠΌΡ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅ΡΡΡ Π°ΠΊΡΡΠ°Π»ΡΠ½ΠΎΡΡΡ Π·Π°Π΄Π°ΡΠΈ Π΄Π»Ρ Π΅Π΅
ΠΈΡΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»Ρ.</p>
</li>
</ul>
<p>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
<em>ΠΠ°Π΄Π°ΡΠ°</em> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Π²ΡΡΠΈΡΠ»ΡΠ΅ΠΌΡΠ΅ Π°ΡΡΠΈΠ±ΡΡΡ:</p>
<ul class="simple">
<li><em>Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π°</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΄Π°ΡΡ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ. ΠΠ»Ρ ΠΏΠ΅ΡΠ²ΠΎΠΉ
Π·Π°Π΄Π°ΡΠΈ ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ. ΠΠ»Ρ Π·Π°Π΄Π°ΡΠΈ ΡΠΎ ΡΡΠ°ΡΡΡΠΎΠΌ WAIT
ΡΠ°Π²Π½Π° Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΡΠ»ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½ Π½Π°ΡΡΡΠ΅Π½, Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° Π·Π°Π΄Π°ΡΠΈ
Π½Π΅ ΠΏΡΠΎΠ³Π½ΠΎΠ·ΠΈΡΡΠ΅ΠΌΠ°. ΠΠ»Ρ ΠΎΡΡΠ°Π»ΡΠ½ΡΡ
ΡΡΠ°ΡΡΡΠΎΠ² ΡΠ°Π²Π½Π° Π΄Π°ΡΠ΅ ΠΎΠΊΠΎΠ½ΡΠ°Π½ΠΈΡ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ
Π·Π°Π΄Π°ΡΠΈ;</li>
<li><em>ΡΡΠΎΠΊ Π΄ΠΎ Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π½Π°ΡΠ°Π»Π°
ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ Π·Π°Π΄Π°ΡΠ΅ΠΉ (ΡΡΠ°ΡΡΡ WAIT);</li>
<li><em>ΡΡΠΎΠΊ Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°
Π·Π°Π΄Π°ΡΠΈ;</li>
<li><em>ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΠ΅ Π΄Π½ΠΈ</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π½Π° ΠΊΠΎΡΠΎΡΡΠ΅ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π°
Π·Π°Π΄Π°ΡΠ°;</li>
<li><em>ΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΡΠΎΠΊ</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π·Π°ΡΡΠ°ΡΠ΅Π½Π½ΡΡ
Π½Π° Π·Π°Π΄Π°ΡΡ.</li>
</ul>
</div>
<div class="section" id="section-3">
<h3>ΠΠΎΠ΄Π΅Π»Ρ "Π¦Π΅ΠΏΠΎΡΠΊΠ°"</h3>
<p>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
<em>Π¦Π΅ΠΏΠΎΡΠΊΠ°</em> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΡΠ»Π΅Π΄ΡΡΡΠΈΠ΅ Π°ΡΡΠΈΠ±ΡΡΡ:</p>
<ul class="simple">
<li><em>Π½Π°Π·Π²Π°Π½ΠΈΠ΅</em> -- ΡΠ΅ΠΊΡΡΠΎΠ²ΠΎΠ΅ ΠΏΠΎΠ»Π΅, ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ΅ Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</li>
<li><em>Π²Π»Π°Π΄Π΅Π»Π΅Ρ</em> -- ΠΈΠ΄Π΅Π½ΡΠΈΡΠΈΠΊΠ°ΡΠΎΡ ΡΠΎΡΡΡΠ΄Π½ΠΈΠΊΠ°. Π¦Π΅Π»ΠΎΡΠΈΡΠ»Π΅Π½Π½ΠΎΠ΅ ΠΏΠΎΠ»Π΅, ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎΠ΅ Π΄Π»Ρ
Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</li>
<li><em>Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π°</em> -- Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π°Π΄ ΡΠ΅ΠΏΠΎΡΠΊΠΎΠΉ Π·Π°Π΄Π°Ρ. ΠΠΎΠ»Π΅ ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΠΎ Π΄Π»Ρ
Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</li>
<li><em>ΠΏΡΠΈΠΎΡΠΈΡΠ΅Ρ</em> -- ΠΏΡΠΈΠΎΡΠΈΡΠ΅Ρ ΡΠ΅ΠΏΠΎΡΠΊΠΈ (ΡΡΠΎΡΠ½ΠΎ, Π½Π΅ ΡΡΠΎΡΠ½ΠΎ);</li>
<li><em>Π°ΡΡ
ΠΈΠ²</em> -- ΡΠ»Π°Π³, ΠΏΠΎ ΠΊΠΎΡΠΎΡΠΎΠΌΡ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅ΡΡΡ Π°ΠΊΡΡΠ°Π»ΡΠ½ΠΎΡΡΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ Π΄Π»Ρ Π΅Π΅
Π²Π»Π°Π΄Π΅Π»ΡΡΠ°;</li>
</ul>
<p>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
<em>Π¦Π΅ΠΏΠΎΡΠΊΠ°</em> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Π²ΡΡΠΈΡΠ»ΡΠ΅ΠΌΡΠ΅ Π°ΡΡΠΈΠ±ΡΡΡ:</p>
<ul class="simple">
<li><em>ΡΡΠ°ΡΡΡ</em> -- ΡΡΠ°ΡΡΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ ΡΠΎΠ²ΠΏΠ°Π΄Π°Π΅Ρ ΡΠΎ ΡΡΠ°ΡΡΡΠΎΠΌ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠ΅ΠΏΠΎΡΠΊΠΈ;</li>
<li><em>Π΄Π΅Π΄Π»Π°ΠΉΠ½</em> -- Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΡΠ΅ΠΏΠΎΡΠΊΠΈ ΡΠΎΠ²ΠΏΠ°Π΄Π°Π΅Ρ Ρ Π΄Π΅Π΄Π»Π°ΠΉΠ½ΠΎΠΌ ΠΏΠΎΡΠ»Π΅Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li><em>ΡΡΠΎΠΊ Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π°</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ ΠΏΠΎΠ»Π½ΡΡ
Π΄Π½Π΅ΠΉ, ΠΎΡΡΠ°Π²ΡΠΈΡ
ΡΡ Π΄ΠΎ
Π΄Π΅Π΄Π»Π°ΠΉΠ½Π° ΠΏΠΎΡΠ»Π΅Π΄Π½Π΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li><em>ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π½ΡΠ΅ Π΄Π½ΠΈ</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π½Π° ΠΊΠΎΡΠΎΡΡΠ΅ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½Π°
ΠΏΠΎΡΠ»Π΅Π΄Π½ΡΡ Π·Π°Π΄Π°ΡΠ°;</li>
<li><em>ΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΡΠΎΠΊ</em> -- ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ Π΄Π½Π΅ΠΉ, Π·Π°ΡΡΠ°ΡΠ΅Π½Π½ΡΡ
Π½Π° ΡΠ΅ΠΏΠΎΡΠΊΡ.</li>
</ul>
<p>ΠΠΎΠ΄Π΅Π»Ρ Π΄Π°Π½Π½ΡΡ
<em>Π‘ΠΎΡΡΡΠ΄Π½ΠΈΠΊ</em> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΡΠ΅ΠΊΡΡΠΎΠ²ΡΠ΅ Π°ΡΡΠΈΠ±ΡΡΡ <em>Π€ΠΠ</em> ΠΈ <em>Π΄ΠΎΠ»ΠΆΠ½ΠΎΡΡΡ</em>,
ΠΎΠ±ΡΠ·Π°ΡΠ΅Π»ΡΠ½ΡΠ΅ Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ.</p>
<table class="docutils footnote" frame="void" id="mcconnell" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π» Π‘. Π‘ΠΎΠ²Π΅ΡΡΠ΅Π½Π½ΡΠΉ ΠΊΠΎΠ΄. ΠΠ°ΡΡΠ΅Ρ-ΠΊΠ»Π°ΡΡ /
ΠΠ΅Ρ. Ρ Π°Π½Π³Π». β Π. : ΠΠ·Π΄Π°ΡΠ΅Π»ΡΡΡΠ²ΠΎ "Π ΡΡΡΠΊΠ°Ρ ΡΠ΅Π΄Π°ΠΊΡΠΈΡ", 2012. β 896 ΡΡΡ. : ΠΈΠ».</td></tr>
</tbody>
</table>
</div>
</div>
Django TODO: Π²ΡΡΠ°Π±ΠΎΡΠΊΠ° ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΠΉ ΠΊ ΡΠΈΡΡΠ΅ΠΌΠ΅2012-06-29T11:00:00+07:002012-06-29T11:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-29:/django-todo-development-of-system-requirements.html<p>ΠΠΎΡΠ»Π΅ ΠΏΡΠΎΡΡΠ΅Π½ΠΈΡ ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π»Π° Π·Π°Ρ
ΠΎΡΠ΅Π»ΠΎΡΡ ΡΠΏΡΠΎΠ΅ΡΠΈΡΠΎΠ²Π°ΡΡ Π΅Π³ΠΎ ΡΠΎΠ²Π΅ΡΡ Π½Π° Django.
ΠΠ»Ρ ΡΡΠΎΠ³ΠΎ Ρ Π²Π·ΡΠ» Π·Π° ΠΎΡΠ½ΠΎΠ²Ρ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΡ ΡΠΈΡΡΠ΅ΠΌΡ <a class="reference external" href="https://github.com/marselester/django-todo">Django TODO</a>. ΠΡΠ°ΠΊ, ΠΏΠ΅ΡΠ²ΡΠΉ ΡΡΠ°ΠΏ -- Π²ΡΡΠ°Π±ΠΎΡΠΊΠ° ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΠΉ ΠΊ
ΡΠΈΡΡΠ΅ΠΌΠ΅.</p>
<p>Π’ΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎΠ΄ΡΠΎΠ±Π½ΠΎ ΠΎΠΏΠΈΡΡΠ²Π°ΡΡ, ΡΡΠΎ Π΄ΠΎΠ»ΠΆΠ½Π° Π΄Π΅Π»Π°ΡΡ ΡΠΈΡΡΠ΅ΠΌΠ°. ΠΠ½ΠΈΠΌΠ°Π½ΠΈΠ΅ ΠΊ
ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡΠΌ ΠΏΠΎΠΌΠΎΠ³Π°Π΅Ρ ΡΠ²Π΅ΡΡΠΈ ΠΊ ΠΌΠΈΠ½ΠΈΠΌΡΠΌΡ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΡ ΡΠΈΡΡΠ΅ΠΌΡ ΠΏΠΎΡΠ»Π΅ Π½Π°ΡΠ°Π»Π°
ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ. Π―Π²Π½ΡΠ΅ ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎΠΌΠΎΠ³Π°ΡΡ Π³Π°ΡΠ°Π½ΡΠΈΡΠΎΠ²Π°ΡΡ, ΡΡΠΎ β¦</p><p>ΠΠΎΡΠ»Π΅ ΠΏΡΠΎΡΡΠ΅Π½ΠΈΡ ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π»Π° Π·Π°Ρ
ΠΎΡΠ΅Π»ΠΎΡΡ ΡΠΏΡΠΎΠ΅ΡΠΈΡΠΎΠ²Π°ΡΡ Π΅Π³ΠΎ ΡΠΎΠ²Π΅ΡΡ Π½Π° Django.
ΠΠ»Ρ ΡΡΠΎΠ³ΠΎ Ρ Π²Π·ΡΠ» Π·Π° ΠΎΡΠ½ΠΎΠ²Ρ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΡ ΡΠΈΡΡΠ΅ΠΌΡ <a class="reference external" href="https://github.com/marselester/django-todo">Django TODO</a>. ΠΡΠ°ΠΊ, ΠΏΠ΅ΡΠ²ΡΠΉ ΡΡΠ°ΠΏ -- Π²ΡΡΠ°Π±ΠΎΡΠΊΠ° ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΠΉ ΠΊ
ΡΠΈΡΡΠ΅ΠΌΠ΅.</p>
<p>Π’ΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎΠ΄ΡΠΎΠ±Π½ΠΎ ΠΎΠΏΠΈΡΡΠ²Π°ΡΡ, ΡΡΠΎ Π΄ΠΎΠ»ΠΆΠ½Π° Π΄Π΅Π»Π°ΡΡ ΡΠΈΡΡΠ΅ΠΌΠ°. ΠΠ½ΠΈΠΌΠ°Π½ΠΈΠ΅ ΠΊ
ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡΠΌ ΠΏΠΎΠΌΠΎΠ³Π°Π΅Ρ ΡΠ²Π΅ΡΡΠΈ ΠΊ ΠΌΠΈΠ½ΠΈΠΌΡΠΌΡ ΠΈΠ·ΠΌΠ΅Π½Π΅Π½ΠΈΡ ΡΠΈΡΡΠ΅ΠΌΡ ΠΏΠΎΡΠ»Π΅ Π½Π°ΡΠ°Π»Π°
ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ. Π―Π²Π½ΡΠ΅ ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎΠΌΠΎΠ³Π°ΡΡ Π³Π°ΡΠ°Π½ΡΠΈΡΠΎΠ²Π°ΡΡ, ΡΡΠΎ ΡΡΠ½ΠΊΡΠΈΠΎΠ½Π°Π»ΡΠ½ΠΎΡΡΡ
ΡΠΈΡΡΠ΅ΠΌΡ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅ΡΡΡ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Π΅ΠΌ, Π° Π½Π΅ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΡΠΎΠΌ. ΠΡΠ»ΠΈ ΡΠ²Π½ΡΡ
ΡΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΠΉ
Π½Π΅Ρ, ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΡΡ ΠΎΠ±ΡΡΠ½ΠΎ ΡΠ°ΠΌΠΎΠΌΡ ΠΏΡΠΈΡ
ΠΎΠ΄ΠΈΡΡΡ Π²ΡΡΠ°Π±Π°ΡΡΠ²Π°ΡΡ ΠΈΡ
Π²ΠΎ Π²ΡΠ΅ΠΌΡ
ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΠΎΠ²Π°Π½ΠΈΡ <a class="footnote-reference" href="#mcconnell" id="footnote-reference-1">[1]</a>.</p>
<p>Π Π΄Π°Π½Π½ΠΎΠΌ ΡΠ»ΡΡΠ°Π΅ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠ΅ Π΄ΠΎΠ»ΠΆΠ½ΠΎ ΠΏΡΠ΅Π΄ΠΎΡΡΠ°Π²Π»ΡΡΡ ΠΏΡΠΎΡΡΠΎΠΉ ΠΌΠ΅Ρ
Π°Π½ΠΈΠ·ΠΌ ΡΠΎΠ·Π΄Π°Π½ΠΈΡ
ΡΠ΅ΠΏΠΎΡΠ΅ΠΊ Π·Π°Π΄Π°Ρ, ΠΌΠΎΠ½ΠΈΡΠΎΡΠΈΠ½Π³ ΡΡΠ°ΡΡΡΠ° ΠΈΡ
Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ, Π° ΡΠ°ΠΊΠΆΠ΅ Π΄ΠΎΠ»ΠΆΠ½ΠΎ ΠΎΡΠΎΠ±ΡΠ°ΠΆΠ°ΡΡ
Π°ΠΊΡΡΠ°Π»ΡΠ½ΡΠ΅ Π·Π°Π΄Π°ΡΠΈ Π΄Π»Ρ ΠΈΡΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»Ρ.</p>
<p>ΠΠ»Ρ ΠΊΠ°ΠΆΠ΄ΠΎΠΉ Π·Π°Π΄Π°ΡΠΈ ΡΠ΅ΠΏΠΎΡΠΊΠΈ ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°Π΅ΡΡΡ Π΄Π΅Π΄Π»Π°ΠΉΠ½, ΠΊΠΎΡΠΎΡΡΠΉ Π΄ΠΎΠ»ΠΆΠ΅Π½ Π±ΡΡΡ Π±ΠΎΠ»ΡΡΠ΅,
ΡΠ΅ΠΌ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ. ΠΠΎΠ΄ Π΄Π΅Π΄Π»Π°ΠΉΠ½ΠΎΠΌ ΠΏΠΎΠ΄ΡΠ°Π·ΡΠΌΠ΅Π²Π°Π΅ΡΡΡ Π΄Π°ΡΠ° Π² ΡΠΎΡΠΌΠ°ΡΠ΅
<em>Π΄Π΄.ΠΌΠΌ.Π³Π³Π³Π³</em>, Π΄ΠΎ ΠΊΠΎΡΠΎΡΠΎΠΉ Π·Π°Π΄Π°ΡΠ° Π΄ΠΎΠ»ΠΆΠ½Π° Π±ΡΡΡ Π·Π°Π²Π΅ΡΡΠ΅Π½Π°.</p>
<p>Π ΡΠΏΠΈΡΠΊΠ΅ Π·Π°Π΄Π°Ρ ΠΎΡΠΎΠ±ΡΠ°ΠΆΠ°ΡΡΡΡ Π°ΠΊΡΡΠ°Π»ΡΠ½ΡΠ΅ Π·Π°Π΄Π°ΡΠΈ, Π² ΠΊΠΎΡΠΎΡΡΡ
ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ
ΡΠΈΠ³ΡΡΠΈΡΡΠ΅Ρ ΠΊΠ°ΠΊ ΠΈΡΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»Ρ. ΠΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ ΡΠ°ΠΌ ΠΎΡΠΏΡΠ°Π²Π»ΡΠ΅Ρ Π·Π°Π΄Π°ΡΠΈ Π² Π°ΡΡ
ΠΈΠ², ΡΠ΅ΠΌ
ΡΠ°ΠΌΡΠΌ ΠΎΠ½ ΡΠ΅ΡΠ°Π΅Ρ, ΠΊΠ°ΠΊΠΈΠ΅ Π·Π°Π΄Π°ΡΠΈ Π΄Π»Ρ Π½Π΅Π³ΠΎ Π°ΠΊΡΡΠ°Π»ΡΠ½Ρ.</p>
<p>ΠΡΠΎΠ±ΡΠ°ΠΆΠ΅Π½ΠΈΠ΅ Π΄ΠΎΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»ΡΠ½ΠΎΠΉ ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠΈ ΠΎ ΡΡΠΎΠΊΠ΅ Π·Π°Π΄Π°ΡΠΈ Π·Π°Π²ΠΈΡΠΈΡ ΠΎΡ Π΅Π΅ ΡΡΠ°ΡΡΡΠ°.
ΠΡΠΈΠΌΠ΅Ρ Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΡ
Π²Π°ΡΠΈΠ°Π½ΡΠΎΠ² Π΄Π»Ρ ΡΡΠ°ΡΡΡΠ° <em>WAIT</em>:</p>
<ul class="simple">
<li>Π·Π°Π΄Π°ΡΠ° Π½Π°ΡΠ½Π΅Ρ ΡΠ°Π±ΠΎΡΠ°ΡΡ ΡΠ΅ΡΠ΅Π· Π΄Π΅ΡΡΡΡ Π΄Π½Π΅ΠΉ;</li>
<li>Π΄Π°ΡΠ° Π½Π°ΡΠ°Π»Π° ΡΠ°Π±ΠΎΡΡ Π½Π΅ ΠΏΡΠΎΠ³Π½ΠΎΠ·ΠΈΡΡΠ΅ΠΌΠ°, ΡΠ°ΠΊ ΠΊΠ°ΠΊ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΏΡΠ΅Π²ΡΡΠΈΠ»Π°
ΡΠ²ΠΎΠΉ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΈ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅Ρ ΡΠ°Π±ΠΎΡΠΈΠ΅ Π΄Π½ΠΈ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ;</li>
<li>Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½ Π½Π° ΠΎΠ΄ΠΈΠ½ Π΄Π΅Π½Ρ, ΡΠ°ΠΊ ΠΊΠ°ΠΊ ΠΏΡΠ΅Π΄ΡΠ΄ΡΡΠ°Ρ Π·Π°Π΄Π°ΡΠ° ΠΏΡΠ΅Π²ΡΡΠΈΠ»Π° ΡΠ²ΠΎΠΉ
Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΈ Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΡΠ΅ΠΊΡΡΠ΅ΠΉ Π·Π°Π΄Π°ΡΠΈ.</li>
</ul>
<p>ΠΡΠΈΠΌΠ΅Ρ Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΡ
Π²Π°ΡΠΈΠ°Π½ΡΠΎΠ² Π΄Π»Ρ ΡΡΠ°ΡΡΡΠΎΠ² <em>WORK</em> ΠΈ <em>STOP</em>: Π΄ΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Π° ΠΎΡΡΠ°Π»ΠΎΡΡ
Π΄Π²Π° Π΄Π½Ρ, Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½ Π½Π° ΠΎΠ΄ΠΈΠ½ Π΄Π΅Π½Ρ. ΠΡΠΈΠΌΠ΅Ρ Π²ΠΎΠ·ΠΌΠΎΠΆΠ½ΡΡ
Π²Π°ΡΠΈΠ°Π½ΡΠΎΠ² Π΄Π»Ρ
ΡΡΠ°ΡΡΡΠ° <em>DONE</em>: Π²ΡΠΏΠΎΠ»Π½Π΅Π½ΠΎ Π² ΡΡΠΎΠΊ Π·Π° ΠΏΡΡΡ Π΄Π½Π΅ΠΉ, Π΄Π΅Π΄Π»Π°ΠΉΠ½ ΠΏΡΠΎΡΡΠΎΡΠ΅Π½ Π½Π° ΠΎΠ΄ΠΈΠ½ Π΄Π΅Π½Ρ.</p>
<p>Π ΡΠΏΠΈΡΠΊΠ΅ ΡΠ΅ΠΏΠΎΡΠ΅ΠΊ Π·Π°Π΄Π°Ρ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ ΠΎΡΠΎΠ±ΡΠ°ΠΆΠ°ΡΡ Π°ΠΊΡΡΠ°Π»ΡΠ½ΡΠ΅ ΡΠ΅ΠΏΠΎΡΠΊΠΈ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Ρ,
ΠΎΡΡΠΎΡΡΠΈΡΠΎΠ²Π°Π½Π½ΡΠ΅ ΠΏΠΎ Π΄Π΅Π΄Π»Π°ΠΉΠ½Ρ (ΡΠ²Π΅ΡΡ
Ρ ΡΡΠ°ΡΡΠ΅, Π²Π½ΠΈΠ·Ρ Π½ΠΎΠ²ΡΠ΅). ΠΠΊΡΡΠ°Π»ΡΠ½ΠΎΡΡΡ ΡΠ΅ΠΏΠΎΡΠΊΠΈ
ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ Π΅Π΅ Π²Π»Π°Π΄Π΅Π»Π΅Ρ. ΠΠ°ΠΆΠ΄Π°Ρ ΡΠ΅ΠΏΠΎΡΠΊΠ° Π·Π°Π΄Π°Ρ Π΄ΠΎΠ»ΠΆΠ½Π° ΡΠΎΠ΄Π΅ΡΠΆΠ°ΡΡ:</p>
<ul class="simple">
<li>Π½Π°Π·Π²Π°Π½ΠΈΠ΅;</li>
<li>Π΄Π΅Π΄Π»Π°ΠΉΠ½;</li>
<li>ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΡ ΠΎΠ± ΠΈΡΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»Π΅ Π·Π°Π΄Π°ΡΠΈ, Π΅Π΅ ΡΡΠ°ΡΡΡΠ΅, Π²ΡΠ΄Π΅Π»Π΅Π½Π½ΡΠΉ ΠΈ ΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΡΠΎΠΊ
ΠΈΡΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ;</li>
<li>ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΡ ΠΎ ΡΡΠΎΠΊΠ΅ ΡΠ΅ΠΏΠΎΡΠΊΠΈ, ΠΊΠΎΡΠΎΡΠ°Ρ Π·Π°Π²ΠΈΡΠΈΡ ΠΎΡ ΡΡΠ°ΡΡΡΠ° ΡΠ΅ΠΏΠΎΡΠΊΠΈ.</li>
</ul>
<table class="docutils footnote" frame="void" id="mcconnell" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π» Π‘. Π‘ΠΎΠ²Π΅ΡΡΠ΅Π½Π½ΡΠΉ ΠΊΠΎΠ΄. ΠΠ°ΡΡΠ΅Ρ-ΠΊΠ»Π°ΡΡ /
ΠΠ΅Ρ. Ρ Π°Π½Π³Π». β Π. : ΠΠ·Π΄Π°ΡΠ΅Π»ΡΡΡΠ²ΠΎ "Π ΡΡΡΠΊΠ°Ρ ΡΠ΅Π΄Π°ΠΊΡΠΈΡ", 2012. β 896 ΡΡΡ. : ΠΈΠ».</td></tr>
</tbody>
</table>
Π‘ΠΎΠ³Π»Π°ΡΠ΅Π½ΠΈΡ ΠΏΠΎ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ Π½Π° Python/Django2012-06-29T00:00:00+07:002012-06-29T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-29:/links-to-best-practices-of-python-django.html<p>ΠΠΎ Π²ΡΠ΅ΠΌΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ Ρ ΡΠ°ΡΡΠΎ ΡΠ²Π΅ΡΡΡΡΡ Ρ ΠΈΠ·Π²Π΅ΡΡΠ½ΡΠΌΠΈ ΠΌΠ½Π΅ ΡΠΎΠ³Π»Π°ΡΠ΅Π½ΠΈΡΠΌΠΈ,
ΡΡΠ°ΡΠ°ΡΡΡ ΡΠ»Π΅Π΄ΠΎΠ²Π°ΡΡ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΡΠΌ. Π¦ΠΈΡΠΈΡΠΎΠ²Π°ΡΡ ΠΈΡ
Π½Π΅ ΠΈΠΌΠ΅Π΅Ρ ΡΠΌΡΡΠ»Π° -- Π»ΡΡΡΠ΅
ΠΏΡΠΈΠ²Π΅Π΄Ρ ΡΡΡΠ»ΠΊΠΈ.</p>
<p><a class="reference external" href="http://www.python.org/dev/peps/pep-0008/">PEP 8 -- Style Guide for Python Code</a>.</p>
<p><a class="reference external" href="http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#long-lines-continuations">Code Like a Pythonista: Idiomatic Python</a>.
Π Π½Π΅ΠΌ Ρ Π½Π°ΡΠ΅Π» ΠΎΡΠ²Π΅ΡΡ Π½Π° Π²ΠΎΠΏΡΠΎΡΡ ΡΠΎΡΠΌΠ°ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π΄Π»ΠΈΠ½Π½ΡΡ
ΡΡΡΠΎΠΊ:</p>
<pre class="literal-block">
expended_time = (self.finish_date() - self.start_date
+ datetime β¦</pre><p>ΠΠΎ Π²ΡΠ΅ΠΌΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ Ρ ΡΠ°ΡΡΠΎ ΡΠ²Π΅ΡΡΡΡΡ Ρ ΠΈΠ·Π²Π΅ΡΡΠ½ΡΠΌΠΈ ΠΌΠ½Π΅ ΡΠΎΠ³Π»Π°ΡΠ΅Π½ΠΈΡΠΌΠΈ,
ΡΡΠ°ΡΠ°ΡΡΡ ΡΠ»Π΅Π΄ΠΎΠ²Π°ΡΡ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΡΠΌ. Π¦ΠΈΡΠΈΡΠΎΠ²Π°ΡΡ ΠΈΡ
Π½Π΅ ΠΈΠΌΠ΅Π΅Ρ ΡΠΌΡΡΠ»Π° -- Π»ΡΡΡΠ΅
ΠΏΡΠΈΠ²Π΅Π΄Ρ ΡΡΡΠ»ΠΊΠΈ.</p>
<p><a class="reference external" href="http://www.python.org/dev/peps/pep-0008/">PEP 8 -- Style Guide for Python Code</a>.</p>
<p><a class="reference external" href="http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#long-lines-continuations">Code Like a Pythonista: Idiomatic Python</a>.
Π Π½Π΅ΠΌ Ρ Π½Π°ΡΠ΅Π» ΠΎΡΠ²Π΅ΡΡ Π½Π° Π²ΠΎΠΏΡΠΎΡΡ ΡΠΎΡΠΌΠ°ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π΄Π»ΠΈΠ½Π½ΡΡ
ΡΡΡΠΎΠΊ:</p>
<pre class="literal-block">
expended_time = (self.finish_date() - self.start_date
+ datetime.timedelta(days=1))
</pre>
<p><a class="reference external" href="http://google-styleguide.googlecode.com/svn/trunk/pyguide.html">Google Python Style Guide</a>.</p>
<p><a class="reference external" href="https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/">Coding style</a> ΠΈΠ· Π΄ΠΎΠΊΡΠΌΠ΅Π½ΡΠ°ΡΠΈΠΈ Django.</p>
<p><a class="reference external" href="http://lincolnloop.com/django-best-practices/">Django Best Practices</a> ΠΎΡ
ΠΊΠΎΠΌΠΏΠ°Π½ΠΈΠΈ Lincoln Loop.</p>
<p><a class="reference external" href="http://docs.python-guide.org/en/latest/index.html">The Hitchhikerβs Guide to Python!</a> ΠΎΡ Kenneth Reitz.</p>
<p><a class="reference external" href="http://guide.python-distribute.org/">The Hitchhikerβs Guide to Packaging</a>
ΠΎΡ Tarek ZiadΓ©.</p>
<p><a class="reference external" href="http://www.s7labs.com/learn/ewa/">Elements of Web Architecture</a> ΠΎΡ s7labs.</p>
<p><a class="reference external" href="http://twitter.com/getpy">Π’Π²ΠΈΡΡ</a> Ρ ΡΡΡΠ»ΠΊΠ°ΠΌΠΈ Π½Π° ΠΈΠ½ΡΠ΅ΡΠ΅ΡΠ½ΡΠ΅ ΡΡΠ°ΡΡΠΈ ΠΏΠΎ Python
ΠΎΡ <a class="reference external" href="https://twitter.com/czheng">@czheng</a>.</p>
<div class="section" id="section-1">
<h2>Π Π΅ΠΊΠΎΠΌΠ΅Π½Π΄Π°ΡΠΈΠΈ ΠΏΠΎ ΠΈΠΌΠ΅Π½ΠΎΠ²Π°Π½ΠΈΡ ΠΎΡ ΠΠ°ΠΊΠΊΠΎΠ½Π΅Π»Π»Π°</h2>
<p>Π§ΠΈΡΠ°ΡΡ ΠΊΠ°ΠΊ ΠΌΠ°Π½ΡΡΡ.</p>
<div class="section" id="section-2">
<h3>ΠΠΌΠ΅Π½ΠΎΠ²Π°Π½ΠΈΠ΅ ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΡΡ
</h3>
<p>ΠΠΌΡ Π΄ΠΎΠ»ΠΆΠ½ΠΎ ΠΏΠΎΠ»Π½ΠΎ ΠΈ ΡΠΎΡΠ½ΠΎ ΠΎΠΏΠΈΡΡΠ²Π°ΡΡ ΡΡΡΠ½ΠΎΡΡΡ. Π€ΠΎΡΠΌΡΠ»ΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΡΡΡΠΈ ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΠΎΠΉ Π²
ΡΠ»ΠΎΠ²Π°Ρ
.</p>
<p>ΠΠΌΡ ΡΠ°ΡΠ΅ Π²ΡΠ΅Π³ΠΎ ΠΎΠΏΠΈΡΡΠ²Π°Π΅Ρ ΠΏΡΠΎΠ±Π»Π΅ΠΌΡ ΡΠ΅Π°Π»ΡΠ½ΠΎΠ³ΠΎ ΠΌΠΈΡΠ°, Π° Π½Π΅ Π΅Π΅ ΡΠ΅ΡΠ΅Π½ΠΈΠ΅ Π½Π° ΡΠ·ΡΠΊΠ΅
ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΠΎΠ²Π°Π½ΠΈΡ.</p>
<p>ΠΠ½Π°ΡΠΈΠΌΠ°Ρ ΡΠ°ΡΡΡ ΠΈΠΌΠ΅Π½ΠΈ ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΠΎΠΉ ΡΠ°ΡΠΏΠΎΠ»Π°Π³Π°ΡΡΡΡ Π² Π½Π°ΡΠ°Π»Π΅ ΠΈ ΡΠΈΡΠ°Π΅ΡΡΡ ΠΏΠ΅ΡΠ²ΠΎΠΉ.
Π‘ΠΏΠ΅ΡΠΈΡΠΈΠΊΠ°ΡΠΎΡΡ Π²ΡΡΠΈΡΠ»ΡΠ΅ΠΌΡΡ
Π·Π½Π°ΡΠ΅Π½ΠΈΠΉ Π½Π°Ρ
ΠΎΠ΄ΡΡΡΡ Π² ΠΊΠΎΠ½ΡΠ΅:</p>
<pre class="literal-block">
revenue_total
expense_average
</pre>
<p>ΠΠ΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠΉΡΠ΅ "Number" ΠΈΠ·-Π·Π° Π΄Π²ΠΎΠΉΠ½ΠΎΠ³ΠΎ ΡΠΌΡΡΠ»Π° (ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²ΠΎ ΠΈ Π½ΠΎΠΌΠ΅Ρ). ΠΡΡΡΠ΅:</p>
<pre class="literal-block">
customer_count
customer_index
</pre>
<p>ΠΠΌΠ΅Π½ΠΎΠ²Π°Π½ΠΈΠ΅ Π±ΡΠ»Π΅Π²ΡΡ
ΠΏΠ΅ΡΠ΅ΠΌΠ΅Π½Π½ΡΡ
Π² ΡΡΠ²Π΅ΡΠ΄ΠΈΡΠ΅Π»ΡΠ½ΠΎΠΉ ΡΠΎΡΠΌΠ΅:</p>
<pre class="literal-block">
done
success ΠΈΠ»ΠΈ ok
found
error
</pre>
<p>ΠΠΌΠ΅Π½Π° Π΄ΠΎΠ»ΠΆΠ½Ρ Π»Π΅Π³ΠΊΠΎ ΡΠΈΡΠ°ΡΡΡΡ.</p>
<p>ΠΠ·Π±Π΅Π³Π°ΠΉΡΠ΅ ΠΈΠΌΠ΅Π½, ΠΈΠΌΠ΅ΡΡΠΈΡ
:</p>
<ul class="simple">
<li>ΠΏΠΎΡ
ΠΎΠΆΠΈΠ΅ Π·Π½Π°ΡΠ΅Π½ΠΈΡ: <tt class="docutils literal">file_number, file_index</tt>;</li>
<li>ΠΏΠΎΡ
ΠΎΠΆΠΈΠ΅ Π·Π²ΡΡΠ°Π½ΠΈΡ: <tt class="docutils literal">wrap, rap</tt>;</li>
<li>ΠΏΠΎΡ
ΠΎΠΆΠΈΠ΅ ΠΈΠΌΠ΅Π½Π°: ΠΏΠ»ΠΎΡ
ΠΎ -- <tt class="docutils literal">client_recs, client_reps</tt>, Ρ
ΠΎΡΠΎΡΠΎ --
<tt class="docutils literal">client_records, clients_reports</tt>;</li>
<li>ΡΠΈΡΡΡ: <tt class="docutils literal">file1, file2</tt>.</li>
</ul>
<p>ΠΠ΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠΉΡΠ΅ "ΠΌΠ°Π³ΠΈΡΠ΅ΡΠΊΠΈΠ΅ ΡΠΈΡΠ»Π°", Π·Π°ΠΌΠ΅Π½ΡΠΉΡΠ΅ ΠΈΡ
Π°Π±ΡΡΡΠ°ΠΊΡΠ½ΡΠΌΠΈ ΡΡΡΠ½ΠΎΡΡΡΠΌΠΈ:</p>
<pre class="literal-block">
CYCLES_NEEDED = 5
</pre>
</div>
<div class="section" id="section-3">
<h3>ΠΠΌΠ΅Π½ΠΎΠ²Π°Π½ΠΈΠ΅ ΡΡΠ½ΠΊΡΠΈΠΉ</h3>
<p>ΠΡΠ»ΠΈ ΡΡΠ½ΠΊΡΠΈΡ Π²ΠΎΠ·Π²ΡΠ°ΡΠ°Π΅Ρ Π·Π½Π°ΡΠ΅Π½ΠΈΠ΅, Π² ΠΈΠΌΠ΅Π½ΠΈ Π΄ΠΎΠ»ΠΆΠ½ΠΎ Π±ΡΡΡ ΡΠΊΠ°Π·Π°Π½ΠΎ ΠΎΠΏΠΈΡΠ°Π½ΠΈΠ΅
Π²ΠΎΠ·Π²ΡΠ°ΡΠ°Π΅ΠΌΠΎΠ³ΠΎ Π·Π½Π°ΡΠ΅Π½ΠΈΡ.</p>
<p>ΠΠ»Π°Π³ΠΎΠ» <em>get</em> Π·Π΄Π΅ΡΡ ΠΈΠ·Π»ΠΈΡΠ΅Π½:</p>
<pre class="literal-block">
cos()
pen.current_color()
user.last_id()
</pre>
<p>ΠΠ»Ρ ΠΈΠΌΠ΅Π½ΠΎΠ²Π°Π½ΠΈΡ ΠΏΡΠΎΡΠ΅Π΄ΡΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠΉΡΠ΅ <strong>Π³Π»Π°Π³ΠΎΠ» + ΠΎΠ±ΡΠ΅ΠΊΡ</strong>, Π½Π°Π΄ ΠΊΠΎΡΠΎΡΡΠΌ
Π²ΡΠΏΠΎΠ»Π½ΡΠ΅ΡΡΡ Π΄Π΅ΠΉΡΡΠ²ΠΈΠ΅:</p>
<pre class="literal-block">
print_document()
repaginate_document()
document.print()
order_info.check()
monthly_revenue.calc()
</pre>
</div>
</div>
Π Π°Π·Π΄Π΅Π»Π΅Π½ΠΈΠ΅ Π½Π°ΡΡΡΠΎΠ΅ΠΊ Π² Django2012-06-29T00:00:00+07:002012-06-29T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-29:/splitting-settings-in-django.html<p>Π <a class="reference external" href="https://code.djangoproject.com/wiki/SplitSettings">Django wiki</a> ΡΠΎΠ±ΡΠ°Π½Ρ
ΡΠ°Π·Π»ΠΈΡΠ½ΡΠ΅ ΡΠΏΠΎΡΠΎΠ±Ρ ΡΠ°Π·Π΄Π΅Π»Π΅Π½ΠΈΡ Π½Π°ΡΡΡΠΎΠ΅ΠΊ. ΠΠ½Π΅ Π½ΡΠ°Π²ΠΈΡΡΡ <a class="reference external" href="http://senko.net/en/django-quickstart-skeleton-project/">Π²Π°ΡΠΈΠ°Π½Ρ</a>, ΠΎΠΏΠΈΡΠ°Π½Π½ΡΠΉ Π² Π±Π»ΠΎΠ³Π΅
Senko RaΕ‘iΔ:</p>
<pre class="literal-block">
settings/
βββ __init__.py
βββ base.py
βββ development.py
βββ local.py
βββ production.py
</pre>
<p><tt class="docutils literal">base.py</tt> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΠΎΠ±ΡΠΈΠ΅ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ Π΄Π»Ρ <tt class="docutils literal">development.py</tt> ΠΈ
<tt class="docutils literal">production.py</tt>, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ:</p>
<div class="highlight"><pre><span></span><span class="n">ADMINS</span> <span class="o">=</span> <span class="p">()</span>
<span class="n">MANAGERS</span> <span class="o">=</span> <span class="n">ADMINS</span>
<span class="n">TIME_ZONE</span> <span class="o">=</span> <span class="s1">'Asia/Yekaterinburg'</span>
<span class="c1"># ...</span>
</pre></div>
<p><tt class="docutils literal">production.py</tt> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ Π΄Π»Ρ β¦</p><p>Π <a class="reference external" href="https://code.djangoproject.com/wiki/SplitSettings">Django wiki</a> ΡΠΎΠ±ΡΠ°Π½Ρ
ΡΠ°Π·Π»ΠΈΡΠ½ΡΠ΅ ΡΠΏΠΎΡΠΎΠ±Ρ ΡΠ°Π·Π΄Π΅Π»Π΅Π½ΠΈΡ Π½Π°ΡΡΡΠΎΠ΅ΠΊ. ΠΠ½Π΅ Π½ΡΠ°Π²ΠΈΡΡΡ <a class="reference external" href="http://senko.net/en/django-quickstart-skeleton-project/">Π²Π°ΡΠΈΠ°Π½Ρ</a>, ΠΎΠΏΠΈΡΠ°Π½Π½ΡΠΉ Π² Π±Π»ΠΎΠ³Π΅
Senko RaΕ‘iΔ:</p>
<pre class="literal-block">
settings/
βββ __init__.py
βββ base.py
βββ development.py
βββ local.py
βββ production.py
</pre>
<p><tt class="docutils literal">base.py</tt> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΠΎΠ±ΡΠΈΠ΅ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ Π΄Π»Ρ <tt class="docutils literal">development.py</tt> ΠΈ
<tt class="docutils literal">production.py</tt>, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ:</p>
<div class="highlight"><pre><span></span><span class="n">ADMINS</span> <span class="o">=</span> <span class="p">()</span>
<span class="n">MANAGERS</span> <span class="o">=</span> <span class="n">ADMINS</span>
<span class="n">TIME_ZONE</span> <span class="o">=</span> <span class="s1">'Asia/Yekaterinburg'</span>
<span class="c1"># ...</span>
</pre></div>
<p><tt class="docutils literal">production.py</tt> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ Π΄Π»Ρ ΡΠΊΡΠΏΠ»ΡΠ°ΡΠ°ΡΠΈΠΈ. ΠΠ°ΠΊ ΠΌΠΈΠ½ΠΈΠΌΡΠΌ, Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ
Π²ΡΠΊΠ»ΡΡΠΈΡΡ ΡΠ΅ΠΆΠΈΠΌ ΠΎΡΠ»Π°Π΄ΠΊΠΈ:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">.base</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">DEBUG</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">TEMPLATE_DEBUG</span> <span class="o">=</span> <span class="n">DEBUG</span>
<span class="c1"># ...</span>
</pre></div>
<p><tt class="docutils literal">development.py</tt> ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ, Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΡΠ΅ Π΄Π»Ρ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ. ΠΠ°ΡΡΡΠΎΠΉΠΊΠΈ
Π΄ΠΎΠ»ΠΆΠ½Ρ Π±ΡΡΡ <strong>ΠΎΠ΄ΠΈΠ½Π°ΠΊΠΎΠ²Ρ</strong> ΠΈ <strong>ΠΏΠΎΠ»Π΅Π·Π½Ρ Π΄Π»Ρ Π²ΡΠ΅Ρ
ΡΡΠ°ΡΡΠ½ΠΈΠΊΠΎΠ²</strong> ΠΏΡΠΎΡΠ΅ΡΡΠ°
ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">.base</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">DEBUG</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">TEMPLATE_DEBUG</span> <span class="o">=</span> <span class="n">DEBUG</span>
<span class="c1"># ...</span>
</pre></div>
<p>ΠΡΠ΅ ΠΈΠ½Π΄ΠΈΠ²ΠΈΠ΄ΡΠ°Π»ΡΠ½ΡΠ΅ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ° Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ Π²ΡΠ½Π΅ΡΡΠΈ Π² <tt class="docutils literal">local.py</tt>.
ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, ΡΡΠΎ ΠΌΠΎΠ³ΡΡ Π±ΡΡΡ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ ΠΏΠΎΠ΄ΠΊΠ»ΡΡΠ΅Π½ΠΈΡ ΠΊ ΠΠ, Π»ΡΠ±ΠΈΠΌΡΠ΅ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½ΡΡ
ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ ΠΈ ΡΠ°ΠΊ Π΄Π°Π»Π΅Π΅:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">.development</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'default'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'ENGINE'</span><span class="p">:</span> <span class="s1">'django.db.backends.mysql'</span><span class="p">,</span>
<span class="s1">'NAME'</span><span class="p">:</span> <span class="s2">"developer's db name"</span><span class="p">,</span>
<span class="s1">'USER'</span><span class="p">:</span> <span class="s2">"developer's db user"</span><span class="p">,</span>
<span class="s1">'PASSWORD'</span><span class="p">:</span> <span class="s2">"developer's db password"</span><span class="p">,</span>
<span class="s1">'HOST'</span><span class="p">:</span> <span class="s1">''</span>
<span class="s1">'PORT'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>ΠΠ° production ΡΠ΅ΡΠ²Π΅ΡΠ΅ <tt class="docutils literal">local.py</tt> ΠΎΠ±ΡΡΠ½ΠΎ ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Π½Π°ΡΡΡΠΎΠΉΠΊΠΈ ΠΏΠΎΠ΄ΠΊΠ»ΡΡΠ΅Π½ΠΈΡ ΠΊ
ΠΠ:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">.production</span> <span class="kn">import</span> <span class="o">*</span>
</pre></div>
<p><tt class="docutils literal">local.py</tt> Π½Π΅ Π΄ΠΎΠ»ΠΆΠ΅Π½ ΠΎΡΡΠ»Π΅ΠΆΠΈΠ²Π°ΡΡΡΡ VCS.</p>
<p>ΠΠ»Ρ Django 1.4 Π² ΡΠ°ΠΉΠ»Π°Ρ
<tt class="docutils literal"><span class="pre">repo-name/project_name/wsgi.py</span></tt> ΠΈ
<tt class="docutils literal"><span class="pre">repo-name/manage.py</span></tt> Π½ΡΠΆΠ½ΠΎ ΡΠΊΠ°Π·Π°ΡΡ ΠΏΡΡΡ Π΄ΠΎ <tt class="docutils literal">local.py</tt>:</p>
<div class="highlight"><pre><span></span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="s2">"DJANGO_SETTINGS_MODULE"</span><span class="p">,</span> <span class="s2">"project_name.settings.local"</span><span class="p">)</span>
</pre></div>
<p>Π£ ΠΌΠ΅Π½Ρ Π½Π΅ ΠΏΠΎΠ»ΡΡΠΈΠ»ΠΎΡΡ ΡΠ°Π·ΠΌΠ΅ΡΡΠΈΡΡ <tt class="docutils literal">local.py</tt> Π½Π° <a class="reference external" href="http://www.heroku.com/">heroku</a>, ΡΡΠΎΠ±Ρ ΡΠ°ΠΉΠ» Π½Π΅ Π½Π°Ρ
ΠΎΠ΄ΠΈΠ»ΡΡ ΠΏΠΎΠ΄ ΠΊΠΎΠ½ΡΡΠΎΠ»Π΅ΠΌ git (ΠΏΠ»ΠΎΡ
ΠΎ
ΠΈΡΠΊΠ°Π»?). Π‘Π°ΠΌΠΎΠ΅ ΠΏΡΠΎΡΡΠΎΠ΅ ΡΠ΅ΡΠ΅Π½ΠΈΠ΅ -- ΠΈΠ·Π±Π°Π²ΠΈΡΡΡΡ ΠΎΡ <tt class="docutils literal">local.py</tt>, Π² <tt class="docutils literal">manage.py</tt>
ΡΠΊΠ°Π·Π°ΡΡ <tt class="docutils literal">project_name.settings.development</tt>, Π° Π² <tt class="docutils literal">wsgi.py</tt> --
<tt class="docutils literal">project_name.settings.production</tt>. ΠΡΠ΅ ΠΌΠΎΠΆΠ½ΠΎ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ
<tt class="docutils literal">heroku config</tt>:</p>
<pre class="literal-block">
$ heroku config:add SECRET_KEY=my_unique_secret_key
</pre>
ΠΡΠ°ΡΠΊΠΈΠΉ ΠΎΠ±Π·ΠΎΡ ΠΈΠ½ΡΡΠ°ΡΡΡΡΠΊΡΡΡΡ Π΄Π»Ρ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ reusable Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ2012-06-13T00:00:00+07:002012-06-13T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2012-06-13:/short-overview-of-infrastructure-for-developing-reusable-django-apps.html<p>ΠΠ°ΡΠΈΠ½Π°Ρ Π²ΠΏΠ΅ΡΠ²ΡΠ΅ ΡΠ°Π·ΡΠ°Π±Π°ΡΡΠ²Π°ΡΡ Π²Π΅Π±-ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ Π½Π° Π½ΠΎΠ²ΠΎΠΌ ΡΡΠ΅ΠΉΠΌΠ²ΠΎΡΠΊΠ΅ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΡ
Π·Π°ΡΠ°ΡΡΡΡ ΡΡΠ°Π»ΠΊΠΈΠ²Π°Π΅ΡΡΡ Ρ Π½Π΅ΠΊΠΎΡΠΎΡΡΠΌΠΈ ΡΡΡΠ΄Π½ΠΎΡΡΡΠΌΠΈ. ΠΡΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΎΡΡΡΠΆΠ΄Π°Π΅ΠΌΡΡ
Π²Π΅Π±-ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π° Django ΠΊ ΡΡΠΈΠΌ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ°ΠΌ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ ΠΎΡΠ½Π΅ΡΡΠΈ ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΡ
ΡΠ°ΠΉΠ»ΠΎΠ² Π² ΠΏΡΠΎΠ΅ΠΊΡΠ΅, ΠΎΠ±Π½Π°ΡΡΠΆΠ΅Π½ΠΈΠ΅ ΡΠ΅ΡΡΠΎΠ², Π²ΠΎΠΏΡΠΎΡΡ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ ΠΈ
ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΠΈ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·ΠΈΡΠΎΠ²Π°Π½Π½ΠΎΠ³ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ. Π Π΄Π°Π½Π½ΠΎΠΉ ΡΡΠ°ΡΡΠ΅ ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½Ρ ΠΏΡΡΠΈ
ΡΠ΅ΡΠ΅Π½ΠΈΡ ΡΡΠΈΡ
ΠΏΡΠΎΠ±Π»Π΅ΠΌ.</p>
<p>ΠΠ°ΠΆΠ½ΠΎ Π·Π½Π°ΡΡ ΡΠ°Π·Π»ΠΈΡΠΈΡ ΠΌΠ΅ΠΆΠ΄Ρ Π΄Π²ΡΠΌΡ β¦</p><p>ΠΠ°ΡΠΈΠ½Π°Ρ Π²ΠΏΠ΅ΡΠ²ΡΠ΅ ΡΠ°Π·ΡΠ°Π±Π°ΡΡΠ²Π°ΡΡ Π²Π΅Π±-ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ Π½Π° Π½ΠΎΠ²ΠΎΠΌ ΡΡΠ΅ΠΉΠΌΠ²ΠΎΡΠΊΠ΅ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΈΡΡ
Π·Π°ΡΠ°ΡΡΡΡ ΡΡΠ°Π»ΠΊΠΈΠ²Π°Π΅ΡΡΡ Ρ Π½Π΅ΠΊΠΎΡΠΎΡΡΠΌΠΈ ΡΡΡΠ΄Π½ΠΎΡΡΡΠΌΠΈ. ΠΡΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΎΡΡΡΠΆΠ΄Π°Π΅ΠΌΡΡ
Π²Π΅Π±-ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π½Π° Django ΠΊ ΡΡΠΈΠΌ ΠΏΡΠΎΠ±Π»Π΅ΠΌΠ°ΠΌ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎ ΠΎΡΠ½Π΅ΡΡΠΈ ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΡ
ΡΠ°ΠΉΠ»ΠΎΠ² Π² ΠΏΡΠΎΠ΅ΠΊΡΠ΅, ΠΎΠ±Π½Π°ΡΡΠΆΠ΅Π½ΠΈΠ΅ ΡΠ΅ΡΡΠΎΠ², Π²ΠΎΠΏΡΠΎΡΡ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ ΠΈ
ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΠΈ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·ΠΈΡΠΎΠ²Π°Π½Π½ΠΎΠ³ΠΎ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ. Π Π΄Π°Π½Π½ΠΎΠΉ ΡΡΠ°ΡΡΠ΅ ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½Ρ ΠΏΡΡΠΈ
ΡΠ΅ΡΠ΅Π½ΠΈΡ ΡΡΠΈΡ
ΠΏΡΠΎΠ±Π»Π΅ΠΌ.</p>
<p>ΠΠ°ΠΆΠ½ΠΎ Π·Π½Π°ΡΡ ΡΠ°Π·Π»ΠΈΡΠΈΡ ΠΌΠ΅ΠΆΠ΄Ρ Π΄Π²ΡΠΌΡ ΡΠΏΠΎΡΠΎΠ±Π°ΠΌΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ Django ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ.
ΠΠ΅ΡΠ²ΡΠΉ ΡΠΏΠΎΡΠΎΠ± Π·Π°ΠΊΠ»ΡΡΠ°Π΅ΡΡΡ Π² ΡΠΎΠ·Π΄Π°Π½ΠΈΠΈ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ Π²Π½ΡΡΡΠΈ ΠΊΠ°ΡΠ°Π»ΠΎΠ³Π° ΠΏΡΠΎΠ΅ΠΊΡΠ° Ρ
ΠΏΠΎΠΌΠΎΡΡΡ ΠΊΠΎΠΌΠ°Π½Π΄Ρ <tt class="docutils literal">manage.py startapp</tt>. ΠΡΠΎΡΠΎΠΉ ΡΠΏΠΎΡΠΎΠ± β ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ°
Π½Π΅Π·Π°Π²ΠΈΡΠΈΠΌΠΎΠ³ΠΎ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ. ΠΠ²ΡΠΎΠ½ΠΎΠΌΠ½ΠΎΠ΅ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠ΅ ΠΏΡΠΎΡΠ΅ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ ΠΈ
ΡΠ°ΡΠΏΡΠΎΡΡΡΠ°Π½ΡΡΡ <a class="footnote-reference" href="#bennett" id="footnote-reference-1">[1]</a>.</p>
<div class="section" id="section-1">
<h2>Π‘ΠΈΡΡΠ΅ΠΌΡ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ</h2>
<p>ΠΠ»Ρ ΡΠΎΠ³ΠΎ ΡΡΠΎΠ±Ρ Django ΡΠΌΠΎΠ³ Π½Π°ΠΉΡΠΈ ΠΈΠ·ΠΎΠ»ΠΈΡΠΎΠ²Π°Π½Π½ΠΎΠ΅ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠ΅, Π½ΡΠΆΠ½ΠΎ ΡΡΠΎΠ±Ρ ΠΎΠ½ΠΎ
Π½Π°Ρ
ΠΎΠ΄ΠΈΠ»ΠΎΡΡ Π² Python ΠΏΡΡΠΈ. Python ΠΏΡΡΡ β ΡΡΠΎ ΡΠΏΠΈΡΠΎΠΊ ΠΊΠ°ΡΠ°Π»ΠΎΠ³ΠΎΠ², Π² ΠΊΠΎΡΠΎΡΡΡ
Python
ΠΈΡΠ΅Ρ ΠΌΠΎΠ΄ΡΠ»ΠΈ, Π²ΡΡΠΊΠΈΠΉ ΡΠ°Π· ΠΈΡΠΏΠΎΠ»Π½ΡΡ ΠΎΠΏΠ΅ΡΠ°ΡΠΎΡ <tt class="docutils literal">import</tt>. ΠΡΠΈ ΡΡΡΠ°Π½ΠΎΠ²ΠΊΠ΅
ΠΈΠ½ΡΠ΅ΡΠΏΡΠ΅ΡΠ°ΡΠΎΡΠ° Python ΡΠΎΠ·Π΄Π°Π΅ΡΡΡ Π΄ΠΈΡΠ΅ΠΊΡΠΎΡΠΈΡ <tt class="docutils literal"><span class="pre">site-packages</span></tt> ΠΈ Π΄ΠΎΠ±Π°Π²Π»ΡΠ΅ΡΡΡ Π²
Python ΠΏΡΡΡ. ΠΡΠ΅ ΡΡΠΎΡΠΎΠ½Π½ΠΈΠ΅ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡ ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°ΡΡΡΡ Π² <tt class="docutils literal"><span class="pre">site-packages</span></tt>.</p>
<p>Python ΠΈΠΌΠ΅Π΅Ρ ΡΠΈΡΡΠ΅ΠΌΡ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ², ΠΊΠΎΡΠΎΡΠ°Ρ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΡΠ°ΡΠΏΡΠΎΡΡΡΠ°Π½ΡΡΡ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ ΠΈ
Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠΈ Π² ΡΡΠ°Π½Π΄Π°ΡΡΠ½ΠΎΠΌ ΡΠΎΡΠΌΠ°ΡΠ΅, ΡΡΠΎ ΠΏΠΎΠΌΠΎΠ³Π°Π΅Ρ Π»Π΅Π³ΠΊΠΎ ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°ΡΡ ΠΈ
ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ ΠΈΡ
. ΠΠΎΠΌΠΈΠΌΠΎ ΡΠ°ΡΠΏΡΠΎΡΡΡΠ°Π½Π΅Π½ΠΈΡ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ² Python ΡΠ°ΠΊΠΆΠ΅ ΠΎΠ±Π΅ΡΠΏΠ΅ΡΠΈΠ²Π°Π΅Ρ
ΡΠ΅Π½ΡΡΠ°Π»ΠΈΠ·ΠΎΠ²Π°Π½Π½ΡΠΉ ΡΠ΅ΡΠ²ΠΈΡ Π΄Π»Ρ ΠΊΠΎΠ½ΡΡΠΈΠ±ΡΡΠΈΠΈ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ². ΠΡΠΎΡ ΡΠ΅ΡΠ²ΠΈΡ Π½Π°Π·ΡΠ²Π°Π΅ΡΡΡ ΠΠ½Π΄Π΅ΠΊΡ
Python ΠΠ°ΠΊΠ΅ΡΠΎΠ² (PyPI). ΠΠ½ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ°ΠΌ ΡΠ°ΡΠΏΡΠΎΡΡΡΠ°Π½ΡΡΡ ΠΏΠ°ΠΊΠ΅Ρ
Π±ΠΎΠ»ΡΡΠΎΠΌΡ ΡΠΎΠΎΠ±ΡΠ΅ΡΡΠ²Ρ Ρ Π½Π΅Π±ΠΎΠ»ΡΡΠΈΠΌΠΈ Π·Π°ΡΡΠ°ΡΠ°ΠΌΠΈ ΡΡΠΈΠ»ΠΈΠΉ. ΠΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½ΠΈΠ΅ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ² Π΄Π°Π΅Ρ
ΡΠ°ΠΊΠΈΠ΅ ΠΏΡΠ΅ΠΈΠΌΡΡΠ΅ΡΡΠ²Π°, ΠΊΠ°ΠΊ:</p>
<ul class="simple">
<li>ΡΠΏΡΠ°Π²Π»Π΅Π½ΠΈΠ΅ Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΡΠΌΠΈ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ²;</li>
<li>ΠΏΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠΈ ΠΎΠ± ΡΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½Π½ΡΡ
ΠΏΠ°ΠΊΠ΅ΡΠ°Ρ
(Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, Π²Π΅ΡΡΠΈΡ ΠΏΠ°ΠΊΠ΅ΡΠ°);</li>
<li>ΡΠ΄Π°Π»Π΅Π½ΠΈΠ΅ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ²;</li>
<li>ΠΏΠΎΠΈΡΠΊ ΠΏΠ°ΠΊΠ΅ΡΠΎΠ² ΠΏΠΎ PyPI <a class="footnote-reference" href="#ziade" id="footnote-reference-2">[2]</a>.</li>
</ul>
<p>Π’Π΅ΠΊΡΡΠ΅Π΅ ΡΠΎΡΡΠΎΡΠ½ΠΈΠ΅ ΡΠΈΡΡΠ΅ΠΌ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎΠΊΠ°Π·Π°Π½ΠΎ Π½Π° ΡΠΈΡΡΠ½ΠΊΠ΅. ΠΡΠ»ΠΈ
ΡΠ°Π·ΡΠ°Π±Π°ΡΡΠ²Π°Π΅ΠΌΠΎΠΌΡ ΠΏΠ°ΠΊΠ΅ΡΡ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌ ΠΌΠΎΠ΄ΡΠ»Ρ <em>Setuptools</em>, ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΠ΅ΡΡΡ
ΡΡΡΠ°Π½ΠΎΠ²ΠΈΡΡ <em>Distribute</em>, ΠΊΠΎΡΠΎΡΡΠΉ ΡΠ²Π»ΡΠ΅ΡΡΡ ΠΎΠ±Π½ΠΎΠ²Π»Π΅Π½Π½ΠΎΠΉ Π²Π΅ΡΡΠΈΠ΅ΠΉ ΠΎΡΠΈΠ³ΠΈΠ½Π°Π»ΡΠ½ΠΎΠ³ΠΎ
<em>Setuptools</em>. <em>Distribute</em> Π±ΡΠ» ΡΠΎΠ·Π΄Π°Π½ ΡΠ°ΠΊ ΠΊΠ°ΠΊ <em>Setuptools</em> Π±ΠΎΠ»ΡΡΠ΅ Π½Π΅
ΠΏΠΎΠ΄Π΄Π΅ΡΠΆΠΈΠ²Π°Π΅ΡΡΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ°ΠΌΠΈ. ΠΠΎΠ΄ΡΠ»Ρ <em>distutils</em> ΡΠ²Π»ΡΠ΅ΡΡΡ ΡΠ°ΡΡΡΡ ΡΡΠ°Π½Π΄Π°ΡΡΠ½ΠΎΠΉ
Π±ΠΈΠ±Π»ΠΈΠΎΡΠ΅ΠΊΠΈ Python ΠΈ ΠΎΠ±Π΅ΡΠΏΠ΅ΡΠΈΠ²Π°Π΅Ρ ΠΎΡΠ½ΠΎΠ²Ρ Π΄Π»Ρ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ. Π Π±ΡΠ΄ΡΡΠ΅ΠΌ
<em>distutils2</em> Π·Π°ΠΌΠ΅Π½ΠΈΡ <em>Setuptools</em> ΠΈ <em>distutils</em>, Π° ΡΠ°ΠΊΠΆΠ΅ ΡΡΡΡΠ°Π½ΠΈΡ
Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΠΎΡΡΡ Π² <em>Distribute</em>. ΠΠΎ Π·Π°Π²Π΅ΡΡΠ΅Π½ΠΈΡ ΡΠ°Π±ΠΎΡ Π½Π°Π΄ <em>distutils2</em>
ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠ°ΠΌ ΡΠ΅ΠΊΠΎΠΌΠ΅Π½Π΄ΡΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΡ <em>distutils</em> <a class="footnote-reference" href="#ziade" id="footnote-reference-3">[2]</a>.</p>
<a class="reference external image-reference" href="http://ziade.org/2012/11/17/chronology-of-packaging/"><img alt="" src="http://blog.ziade.org/history-part1.png" /></a>
<a class="reference external image-reference" href="http://ziade.org/2012/11/17/chronology-of-packaging/"><img alt="" src="http://blog.ziade.org/history-part2.png" /></a>
</div>
<div class="section" id="section-2">
<h2>Π£ΠΏΡΠ°Π²Π»Π΅Π½ΠΈΠ΅ ΡΠΊΠΎΡΠΈΡΡΠ΅ΠΌΠΎΠΉ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ</h2>
<p>ΠΠ»Ρ ΠΌΠ°Π½ΠΈΠΏΡΠ»ΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΡΠΊΠΎΡΠΈΡΡΠ΅ΠΌΠΎΠΉ ΠΏΠ°ΠΊΠ΅ΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Π±ΡΠ»ΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠ°Π½Ρ ΡΠ°ΠΊΠΈΠ΅
ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½ΡΡ, ΠΊΠ°ΠΊ Pip ΠΈ Virtualenv. Pip β ΡΡΠΎ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ Π΄Π»Ρ ΡΡΡΠ°Π½ΠΎΠ²ΠΊΠΈ ΠΈ
ΡΠΏΡΠ°Π²Π»Π΅Π½ΠΈΡ Python ΠΏΠ°ΠΊΠ΅ΡΠ°ΠΌΠΈ. Π‘ Π΅Π³ΠΎ ΠΏΠΎΠΌΠΎΡΡΡ ΠΌΠΎΠΆΠ½ΠΎ:</p>
<ul class="simple">
<li>ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°ΡΡ ΠΏΠ°ΠΊΠ΅ΡΡ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">pip install Django</tt>;</li>
<li>ΠΏΠΎΠΊΠ°Π·ΡΠ²Π°ΡΡ ΡΠΏΠΈΡΠΎΠΊ ΡΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½Π½ΡΡ
ΠΏΠ°ΠΊΠ΅ΡΠΎΠ² ΠΈ ΠΈΡ
Π²Π΅ΡΡΠΈΠΈ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ,
<tt class="docutils literal">pip freeze</tt> ΠΎΡΠΎΠ±ΡΠ°Π·ΠΈΡ <tt class="docutils literal"><span class="pre">Django==1.3.1,</span> <span class="pre">pep8==0.6.1,</span>
<span class="pre">virtualenv==1.7.1.2</span></tt>;</li>
<li>ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°ΡΡ ΠΏΠ°ΠΊΠ΅ΡΡ ΠΎΠΏΡΠ΅Π΄Π΅Π»Π΅Π½Π½ΡΡ
Π²Π΅ΡΡΠΈΠΉ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ,
<tt class="docutils literal">pip install <span class="pre">'Mark-down>2.0,<2.0.3'</span></tt>;</li>
<li>ΠΎΠ±Π½ΠΎΠ²Π»ΡΡΡ ΠΏΠ°ΠΊΠ΅ΡΡ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">pip install <span class="pre">--upgrade</span> Django</tt>;</li>
<li>ΡΠ΄Π°Π»ΡΡΡ ΠΏΠ°ΠΊΠ΅ΡΡ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">pip uninstall Markdown</tt>.</li>
</ul>
<p>ΠΠ»Ρ ΡΠΎΠ³ΠΎ ΡΡΠΎΠ±Ρ Π½Π΅ ΡΠΌΠ΅ΡΠΈΠ²Π°ΡΡ ΡΠΊΡΠΏΠ΅ΡΠΈΠΌΠ΅Π½ΡΠ°Π»ΡΠ½ΡΠ΅ ΠΏΠ°ΠΊΠ΅ΡΡ ΡΠΎ ΡΡΠ°Π±ΠΈΠ»ΡΠ½ΡΠΌΠΈ ΠΏΠ°ΠΊΠ΅ΡΠ°ΠΌΠΈ,
ΡΠ°Π·ΡΠ°Π±ΠΎΡΡΠΈΠΊΠΈ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΡΡ Virtualenv. ΠΠ°Π½Π½ΡΠΉ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ ΠΏΠΎΠ·Π²ΠΎΠ»ΡΠ΅Ρ ΡΠΎΠ·Π΄Π°Π²Π°ΡΡ
ΠΈΠ·ΠΎΠ»ΠΈΡΠΎΠ²Π°Π½Π½ΡΠ΅ Python ΠΎΠΊΡΡΠΆΠ΅Π½ΠΈΡ ΠΈ ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°ΡΡ Π² Π½ΠΈΡ
ΠΏΠ°ΠΊΠ΅ΡΡ, Π½Π΅ ΠΌΠΎΠ΄ΠΈΡΠΈΡΠΈΡΡΡ
ΡΠΈΡΡΠ΅ΠΌΠ½ΠΎΠ΅ ΠΎΠΊΡΡΠΆΠ΅Π½ΠΈΠ΅ Python. ΠΠ°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">virtualenv <span class="pre">--no-site-packages</span> my_env</tt>
ΡΠΎΠ·Π΄Π°ΡΡ ΠΎΠΊΡΡΠΆΠ΅Π½ΠΈΠ΅ <tt class="docutils literal">my_env</tt> Π±Π΅Π· ΠΏΠ°ΠΊΠ΅ΡΠΎΠ², <tt class="docutils literal">source my_env/bin/activate</tt>
Π°ΠΊΡΠΈΠ²ΠΈΡΡΠ΅Ρ ΠΎΠΊΡΡΠΆΠ΅Π½ΠΈΠ΅ <tt class="docutils literal">my_env</tt>, Π° <tt class="docutils literal">deactivate</tt> ΠΎΡΠΊΠ»ΡΡΠΈΡ Π΅Π³ΠΎ.</p>
</div>
<div class="section" id="section-3">
<h2>ΠΠ΅ΡΠ°Π΄Π°Π½Π½ΡΠ΅ ΠΏΡΠΎΠ΅ΠΊΡΠ°</h2>
<p>ΠΠ°ΠΈΠΌΠ΅Π½ΡΡΠΈΠΉ ΠΏΡΠΎΠ΅ΠΊΡ Π½Π° Python ΡΠΎΡΡΠΎΠΈΡ ΠΈΠ· Π΄Π²ΡΡ
ΡΠ°ΠΉΠ»ΠΎΠ². Π€Π°ΠΉΠ» <tt class="docutils literal">setup.py</tt>, ΠΊΠΎΡΠΎΡΡΠΉ
ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ ΠΌΠ΅ΡΠ°Π΄Π°Π½Π½ΡΠ΅ ΠΎ ΠΏΡΠΎΠ΅ΠΊΡΠ΅ ΠΈ ΡΠ°ΠΉΠ», ΠΊΠΎΡΠΎΡΡΠΉ ΡΠΎΠ΄Π΅ΡΠΆΠΈΡ Python ΠΊΠΎΠ΄ Π΄Π»Ρ
ΡΠ΅Π°Π»ΠΈΠ·Π°ΡΠΈΠΈ ΡΡΠ½ΠΊΡΠΈΠΎΠ½Π°Π»ΡΠ½ΠΎΡΡΠΈ ΠΏΡΠΎΠ΅ΠΊΡΠ°. Π <tt class="docutils literal">setup.py</tt> Π΅ΡΡΡ ΡΠΎΠ»ΡΠΊΠΎ ΡΡΠΈ ΠΏΠΎΠ»Ρ,
Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΡΠ΅ Π΄Π»Ρ Π·Π°ΠΏΠΎΠ»Π½Π΅Π½ΠΈΡ: <em>name</em>, <em>version</em> ΠΈ <em>packages</em>. ΠΠΎΠ»Π΅ <em>name</em> Π΄ΠΎΠ»ΠΆΠ½ΠΎ
Π±ΡΡΡ ΡΠ½ΠΈΠΊΠ°Π»ΡΠ½ΡΠΌ Π΄Π»Ρ ΠΏΡΠ±Π»ΠΈΠΊΠ°ΡΠΈΠΈ Π² PyPI. ΠΠΎΠ»Π΅ <em>version</em> ΡΠ»Π΅Π΄ΠΈΡ Π·Π° ΡΠ°Π·Π»ΠΈΡΠ½ΡΠΌΠΈ
Π²Π΅ΡΡΠΈΡΠΌΠΈ ΠΏΡΠΎΠ΅ΠΊΡΠ°. ΠΠΎΠ»Π΅ <em>packages</em> ΠΎΠΏΠΈΡΡΠ²Π°Π΅Ρ, Π³Π΄Π΅ ΡΠ°Π·ΠΌΠ΅ΡΠ΅Π½ Python ΠΊΠΎΠ΄ ΠΏΡΠΎΠ΅ΠΊΡΠ°
<a class="footnote-reference" href="#ziade" id="footnote-reference-4">[2]</a>. ΠΠΈΠΆΠ΅ ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½ ΠΏΡΠΈΠΌΠ΅Ρ ΡΠ°ΠΉΠ»Π° <tt class="docutils literal">setup.py</tt>, ΠΊΠΎΡΠΎΡΡΠΉ ΡΠ°ΠΊΠΆΠ΅ Π²ΠΊΠ»ΡΡΠ°Π΅Ρ
ΠΈΠ½ΡΠΎΡΠΌΠ°ΡΠΈΡ ΠΎ Π»ΠΈΡΠ΅Π½Π·ΠΈΠΈ ΠΈ ΠΎΠΏΠΈΡΠ°Π½ΠΈΠ΅ ΠΏΡΠΎΠ΅ΠΊΡΠ°:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="kn">from</span> <span class="nn">distutils.core</span> <span class="kn">import</span> <span class="n">setup</span>
<span class="n">setup</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s1">'django-todo'</span><span class="p">,</span>
<span class="n">version</span><span class="o">=</span><span class="s1">'0.1dev'</span><span class="p">,</span>
<span class="n">long_description</span><span class="o">=</span><span class="nb">open</span><span class="p">(</span><span class="s1">'README.rst'</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span>
<span class="n">licence</span><span class="o">=</span><span class="s1">'MIT license'</span><span class="p">,</span>
<span class="n">packages</span><span class="o">=</span><span class="p">[</span>
<span class="s1">'todo'</span><span class="p">,</span>
<span class="s1">'todo.templatetags'</span><span class="p">,</span>
<span class="p">]</span>
<span class="p">)</span>
</pre></div>
</div>
<div class="section" id="section-4">
<h2>Π‘ΡΡΡΠΊΡΡΡΠ° ΡΠ΅ΠΏΠΎΠ·ΠΈΡΠΎΡΠΈΡ</h2>
<p>ΠΠ»Ρ ΡΠ΄ΠΎΠ±ΡΡΠ²Π° ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ Π² Django ΡΡΡΠ΅ΡΡΠ²ΡΠ΅Ρ ΡΡΠ°Π½Π΄Π°ΡΡ ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΠΈ ΠΏΡΠΎΠ΅ΠΊΡΠ° ΠΈ
ΡΡΠΈΠ»Ρ ΠΊΠΎΠ΄ΠΈΡΠΎΠ²Π°Π½ΠΈΡ <a class="footnote-reference" href="#django" id="footnote-reference-5">[3]</a>. ΠΡΠΈ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π°Π²ΡΠΎΡ Π² Π±ΠΎΠ»ΡΡΠ΅ΠΉ
ΡΡΠ΅ΠΏΠ΅Π½ΠΈ ΠΎΠΏΠΈΡΠ°Π΅ΡΡΡ Π½Π° Π½ΠΈΡ
, Π° ΡΠ°ΠΊΠΆΠ΅ Π½Π° ΠΏΡΠ°ΠΊΡΠΈΡΠ΅ΡΠΊΠΈΠΉ ΠΎΠΏΡΡ ΠΊΠΎΠΌΠΏΠ°Π½ΠΈΠΈ Lincoln Loop
<a class="footnote-reference" href="#lincolnloop" id="footnote-reference-6">[4]</a>. Π ΡΠΊΠΎΠ²ΠΎΠ΄ΡΡΠ²ΠΎ ΠΏΠΎ ΡΡΠΈΠ»Ρ ΠΊΠΎΠ΄ΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΈ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·Π°ΡΠΈΡ ΠΈΠΌΠ΅Π΅Ρ Π²Π°ΠΆΠ½ΠΎΠ΅
Π·Π½Π°ΡΠ΅Π½ΠΈΠ΅ Π΄Π»Ρ ΡΠΈΠΊΠ»Π° ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠΈ. Π‘ΡΡΡΠΊΡΡΡΠ° ΡΠ΅ΠΏΠΎΠ·ΠΈΡΠΎΡΠΈΡ Π² ΡΠΎΠΉ ΠΆΠ΅ ΡΡΠ΅ΠΏΠ΅Π½ΠΈ ΡΠ²Π»ΡΠ΅ΡΡΡ
Π²Π°ΠΆΠ½ΠΎΠΉ ΡΠ°ΡΡΡΡ Π°ΡΡ
ΠΈΡΠ΅ΠΊΡΡΡΡ ΠΏΡΠΎΠ΅ΠΊΡΠ°. Π Π°ΡΡΠΌΠΎΡΡΠΈΠΌ Π½Π΅ΠΊΠΎΡΠΎΡΡΠ΅ ΠΎΡΠΎΠ±Π΅Π½Π½ΠΎΡΡΠΈ:</p>
<ul class="simple">
<li><tt class="docutils literal">README.rst</tt> Π΄ΠΎΠ»ΠΆΠ΅Π½ ΡΠΎΠ΄Π΅ΡΠΆΠ°ΡΡ ΠΎΠΏΠΈΡΠ°Π½ΠΈΠ΅ ΠΏΡΠΎΠ΅ΠΊΡΠ° Π² ΡΠΎΡΠΌΠ°ΡΠ΅ reST;</li>
<li><tt class="docutils literal">LICENSE</tt> Π΄ΠΎΠ»ΠΆΠ΅Π½ ΡΠΎΠ΄Π΅ΡΠΆΠ°ΡΡ ΠΏΠΎΠ»Π½ΡΠΉ ΡΠ΅ΠΊΡΡ Π»ΠΈΡΠ΅Π½Π·ΠΈΠΈ;</li>
<li>Π°ΡΠ³ΡΠΌΠ΅Π½Ρ <em>install_requires</em> Π² <tt class="docutils literal">setup.py</tt> Π½ΡΠΆΠ΅Π½ Π΄Π»Ρ ΠΏΠ΅ΡΠ΅ΡΠΈΡΠ»Π΅Π½ΠΈΡ
Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠ΅ΠΉ ΠΏΡΠΎΠ΅ΠΊΡΠ°;</li>
<li><tt class="docutils literal">MANIFEST.in</tt> β ΡΡΠΎ ΡΠ°Π±Π»ΠΎΠ½, ΠΊΠΎΡΠΎΡΡΠΉ ΠΎΠΏΡΠ΅Π΄Π΅Π»ΡΠ΅Ρ, ΠΊΠ°ΠΊΠΈΠ΅ ΡΠ°ΠΉΠ»Ρ Π΄ΠΎΠ»ΠΆΠ½Ρ Π±ΡΡΡ
Π²ΠΊΠ»ΡΡΠ΅Π½Ρ Π² ΠΏΠ°ΠΊΠ΅Ρ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, <tt class="docutils literal">include README.rst</tt>;</li>
<li>Π² <tt class="docutils literal">requirements.txt</tt> ΡΠ»Π΅Π΄ΡΠ΅Ρ ΡΠΊΠ°Π·ΡΠ²Π°ΡΡ Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠΈ, Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΡΠ΅ Π΄Π»Ρ
ΡΡΠ°ΡΡΠΈΡ Π² ΡΠ°Π·ΡΠ°Π±ΠΎΡΠΊΠ΅ ΠΏΡΠΎΠ΅ΠΊΡΠ° (ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅, Π³Π΅Π½Π΅ΡΠ°ΡΠΈΡ Π΄ΠΎΠΊΡΠΌΠ΅Π½ΡΠ°ΡΠΈΠΈ)
<a class="footnote-reference" href="#reitz" id="footnote-reference-7">[5]</a>;</li>
<li><tt class="docutils literal">todo/__init__.py</tt>;</li>
<li><tt class="docutils literal">todo/models.py</tt>;</li>
<li><tt class="docutils literal">docs/conf.py</tt>;</li>
<li><tt class="docutils literal">docs/index.rst</tt>;</li>
<li><tt class="docutils literal">tests/__init__.py</tt>;</li>
<li><tt class="docutils literal">tests/models.py</tt>.</li>
</ul>
</div>
<div class="section" id="section-5">
<h2>ΠΠ±Π½Π°ΡΡΠΆΠ΅Π½ΠΈΠ΅ ΠΈ Π·Π°ΠΏΡΡΠΊ ΡΠ΅ΡΡΠΎΠ²</h2>
<p>Π’Π΅ΡΡΡ Π½Π΅ ΡΠ»Π΅Π΄ΡΠ΅Ρ ΡΠ°ΡΠΏΡΠΎΡΡΡΠ°Π½ΡΡΡ Π²ΠΌΠ΅ΡΡΠ΅ Ρ ΠΌΠΎΠ΄ΡΠ»Π΅ΠΌ, ΡΠ°ΠΊ ΠΊΠ°ΠΊ ΡΡΠΎ ΠΏΡΠΈΠ²ΠΎΠ΄ΠΈΡ ΠΊ
ΡΠ²Π΅Π»ΠΈΡΠ΅Π½ΠΈΡ ΡΠ»ΠΎΠΆΠ½ΠΎΡΡΠΈ Π΄Π»Ρ ΠΊΠΎΠ½Π΅ΡΠ½ΡΡ
ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Π΅ΠΉ β Π½Π°Π±ΠΎΡΡ ΡΠ΅ΡΡΠΎΠ² ΡΡΠ΅Π±ΡΡΡ
Π΄ΠΎΠΏΠΎΠ»Π½ΠΈΡΠ΅Π»ΡΠ½ΡΡ
Π·Π°Π²ΠΈΡΠΈΠΌΠΎΡΡΠ΅ΠΉ. ΠΠ° ΠΊΠΎΠ½ΡΠ΅ΡΠ΅Π½ΡΠΈΠΈ <em>PyCon US 2012</em> ΠΠ°ΡΠ» ΠΠ°ΠΉΠ΅Ρ
ΠΏΡΠ΅Π΄Π»ΠΎΠΆΠΈΠ» ΡΠ΅ΡΠ΅Π½ΠΈΠ΅ <a class="footnote-reference" href="#meyer" id="footnote-reference-8">[6]</a>, ΠΊΠΎΡΠΎΡΠΎΠ΅ ΠΏΠΎΠ·Π²ΠΎΠ»ΠΈΠ»ΠΎ ΠΎΡΠ΄Π΅Π»ΠΈΡΡ ΡΠ΅ΡΡΡ ΠΎΡ ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΠΉ Π²
ΠΏΡΠΎΠ΅ΠΊΡΠ΅ ΠΈ ΡΠ΅Π°Π»ΠΈΠ·ΠΎΠ²Π°ΡΡ ΠΎΠ±Π½Π°ΡΡΠΆΠ΅Π½ΠΈΠ΅ ΠΈ Π·Π°ΠΏΡΡΠΊ Π²ΡΠ΅Ρ
ΡΠ΅ΡΡΠΎΠ² ΠΈΠ· ΠΊΠ°ΡΠ°Π»ΠΎΠ³Π° <tt class="docutils literal">tests</tt>.
ΠΠ²ΡΠΎΡ ΠΏΡΠΈΠΌΠ΅Π½ΠΈΠ» Π΄Π°Π½Π½ΠΎΠ΅ ΡΠ΅ΡΠ΅Π½ΠΈΠ΅ Π΄Π»Ρ ΠΎΡΠ³Π°Π½ΠΈΠ·Π°ΡΠΈΠΈ ΡΠ΅ΡΡΠΎΠ² Π² ΠΌΠ½ΠΎΠ³ΠΎΡΠ°Π·ΠΎΠ²ΡΡ
Django
ΠΏΡΠΈΠ»ΠΎΠΆΠ΅Π½ΠΈΡΡ
<a class="footnote-reference" href="#app-skeleton" id="footnote-reference-9">[7]</a>. Π ΠΊΠΎΡΠ½Π΅ ΡΠ΅ΠΏΠΎΠ·ΠΈΡΠΎΡΠΈΡ ΡΠ°ΡΠΏΠΎΠ»Π°Π³Π°Π΅ΡΡΡ ΡΠΊΡΠΈΠΏΡ
<tt class="docutils literal">runtests.py</tt>, ΠΊΠΎΡΠΎΡΡΠΉ Π·Π°ΠΏΡΡΠΊΠ°Π΅Ρ ΡΠ΅ΡΡΡ:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'DJANGO_SETTINGS_MODULE'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'tests.settings'</span>
<span class="kn">from</span> <span class="nn">django.test.utils</span> <span class="kn">import</span> <span class="n">get_runner</span>
<span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="k">def</span> <span class="nf">runtests</span><span class="p">():</span>
<span class="n">TestRunner</span> <span class="o">=</span> <span class="n">get_runner</span><span class="p">(</span><span class="n">settings</span><span class="p">)</span>
<span class="n">test_runner</span> <span class="o">=</span> <span class="n">TestRunner</span><span class="p">(</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">interactive</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">failfast</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">failures</span> <span class="o">=</span> <span class="n">test_runner</span><span class="o">.</span><span class="n">run_tests</span><span class="p">([])</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="n">failures</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">runtests</span><span class="p">()</span>
</pre></div>
<p>ΠΠ°ΡΡΡΠΎΠΉΠΊΠΈ Π΄Π»Ρ ΠΈΡ
Π·Π°ΠΏΡΡΠΊΠ° ΡΠΊΠ°Π·Π°Π½Ρ Π² ΡΠ°ΠΉΠ»Π΅ <tt class="docutils literal">tests/settings.py</tt>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'default'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'ENGINE'</span><span class="p">:</span> <span class="s1">'django.db.backends.sqlite3'</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">INSTALLED_APPS</span> <span class="o">=</span> <span class="p">(</span>
<span class="s1">'app_name'</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">BASE_PATH</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="vm">__file__</span><span class="p">))</span>
<span class="n">TEST_DISCOVERY_ROOT</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BASE_PATH</span><span class="p">,</span> <span class="s1">'tests'</span><span class="p">)</span>
<span class="n">TEST_RUNNER</span> <span class="o">=</span> <span class="s1">'tests.runner.DiscoveryDjangoTestSuiteRunner'</span>
<span class="n">FIXTURE_DIRS</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">TEST_DISCOVERY_ROOT</span><span class="p">,</span> <span class="s1">'fixtures'</span><span class="p">),</span>
<span class="p">)</span>
</pre></div>
<p>ΠΠ±Π½Π°ΡΡΠΆΠ΅Π½ΠΈΠ΅ ΡΠ΅ΡΡΠΎΠ² ΠΎΡΡΡΠ΅ΡΡΠ²Π»ΡΠ΅ΡΡΡ Π²ΠΎ Π²ΡΠ΅Ρ
ΡΠ°ΠΉΠ»Π°Ρ
, ΠΊΠΎΡΠΎΡΡΠ΅ Π½Π°Ρ
ΠΎΠ΄ΡΡΡΡ Π² ΠΊΠ°ΡΠ°Π»ΠΎΠ³Π΅
<tt class="docutils literal">tests</tt> ΠΈ Π½Π°Π·Π²Π°Π½ΠΈΠ΅ ΠΊΠΎΡΠΎΡΡΡ
ΡΠΎΠ²ΠΏΠ°Π΄Π°Π΅Ρ Ρ <em>models.py</em>, <em>tests.py</em> ΠΈΠ»ΠΈ
<em>test*.py</em>.</p>
</div>
<div class="section" id="section-6">
<h2>ΠΠ²ΡΠΎΠΌΠ°ΡΠΈΠ·Π°ΡΠΈΡ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ</h2>
<p>ΠΠ»Ρ Π°Π²ΡΠΎΠΌΠ°ΡΠΈΠ·Π°ΡΠΈΠΈ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ Python ΠΏΡΠΎΠ΅ΠΊΡΠΎΠ² Π°Π²ΡΠΎΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅Ρ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ
tox. ΠΠ½ ΠΌΠΎΠΆΠ΅Ρ Π±ΡΡΡ ΠΈΡΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°Π½:</p>
<ul class="simple">
<li>Π΄Π»Ρ ΠΏΡΠΎΠ²Π΅ΡΠΊΠΈ, ΡΡΠΎ ΠΏΠ°ΠΊΠ΅ΡΡ ΡΡΡΠ°Π½Π°Π²Π»ΠΈΠ²Π°ΡΡΡΡ ΠΏΡΠ°Π²ΠΈΠ»ΡΠ½ΠΎ Π² ΡΠ°Π·Π½ΡΡ
Π²Π΅ΡΡΠΈΡΡ
Python;</li>
<li>Π΄Π»Ρ Π·Π°ΠΏΡΡΠΊΠ° ΡΠ΅ΡΡΠΎΠ² Π² ΠΊΠ°ΠΆΠ΄ΠΎΠΉ ΠΈΠ· ΡΡΠ΅Π΄;</li>
<li>Π² ΠΊΠ°ΡΠ΅ΡΡΠ²Π΅ ΠΈΠ½ΡΠ΅ΡΡΠ΅ΠΉΡΠ° Π΄Π»Ρ ΡΠ΅ΡΠ²Π΅ΡΠ° Π½Π΅ΠΏΡΠ΅ΡΡΠ²Π½ΠΎΠΉ ΠΈΠ½ΡΠ΅Π³ΡΠ°ΡΠΈΠΈ, Π½Π°ΠΏΡΠΈΠΌΠ΅Ρ, Jenkins.</li>
</ul>
<p>ΠΠΈΠΆΠ΅ ΠΏΡΠΈΠ²Π΅Π΄Π΅Π½ ΠΏΡΠΈΠΌΠ΅Ρ ΠΊΠΎΠ½ΡΠΈΠ³ΡΡΠ°ΡΠΈΠΈ <tt class="docutils literal">tox.ini</tt> ΡΠΎ ΡΡΠ΅Π΄Π°ΠΌΠΈ <em>Python 2.6</em>,
<em>Python 2.7</em> ΠΈ <em>Django 1.3</em> <a class="footnote-reference" href="#django-todo" id="footnote-reference-10">[8]</a>:</p>
<pre class="literal-block">
[tox]
envlist=py26,py27,dj13
[testenv]
deps=
django==1.4.0
git+https://github.com/rbarrois/factory_boy.git
webtest
django-webtest
commands=python runtests.py
[testenv:dj13]
deps=
django==1.3.1
git+https://github.com/rbarrois/factory_boy.git
webtest
django-webtest
</pre>
<p>ΠΠΊΡΡΠΆΠ΅Π½ΠΈΠ΅ <em>testenv</em> ΡΠ²Π»ΡΠ΅ΡΡΡ ΡΡΠ΅Π΄ΠΎΠΉ ΠΏΠΎ ΡΠΌΠΎΠ»ΡΠ°Π½ΠΈΡ. Π Π½Π΅ΠΉ ΠΎΠΏΠΈΡΠ°Π½Ρ ΠΏΠ°ΠΊΠ΅ΡΡ Ρ
ΡΠΊΠ°Π·Π°Π½ΠΈΡΠΌΠΈ Π²Π΅ΡΡΠΈΠΉ, ΠΊΠΎΡΠΎΡΡΠ΅ Π½Π΅ΠΎΠ±Ρ
ΠΎΠ΄ΠΈΠΌΡ Π΄Π»Ρ ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ ΠΏΡΠΎΠ΅ΠΊΡΠ° (Π² Π΄Π°Π½Π½ΠΎΠΌ
ΡΠ»ΡΡΠ°Π΅ ΡΡΠΎ ΡΡΠ΅ΠΉΠΌΠ²ΠΎΡΠΊ Django Π²Π΅ΡΡΠΈΠΈ 1.4.1, ΠΏΠΎΡΠ»Π΅Π΄Π½ΠΈΠ΅ Π²Π΅ΡΡΠΈΠΈ ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½ΡΠΎΠ² Π΄Π»Ρ
ΡΠ΅ΡΡΠΈΡΠΎΠ²Π°Π½ΠΈΡ β factory_boy, webtest, django-webtest).</p>
<table class="docutils footnote" frame="void" id="bennett" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Bennett B. Practical Django Projects.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="ziade" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[2]</td><td><em>(<a class="fn-backref" href="#footnote-reference-2">1</a>, <a class="fn-backref" href="#footnote-reference-3">2</a>, <a class="fn-backref" href="#footnote-reference-4">3</a>)</em> ZiadΓ© T. <a class="reference external" href="http://guide.python-distribute.org/">The Hitchhiker's Guide to Packaging</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="django" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[3]</a></td><td>Django community. <a class="reference external" href="https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/">Django Coding Style</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="lincolnloop" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-6">[4]</a></td><td>Lincoln Loop company. <a class="reference external" href="http://lincolnloop.com/django-best-practices/">Django Best Practices</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="reitz" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-7">[5]</a></td><td>Reitz K. <a class="reference external" href="http://kennethreitz.com/repository-structure-and-python.html">Repository Structure and Python</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="meyer" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-8">[6]</a></td><td>Meyer C. <a class="reference external" href="http://carljm.github.com/django-testing-slides/">Testing and Django</a> at PyCon US 2012.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="app-skeleton" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-9">[7]</a></td><td>ΠΠ°Π²Π»Π΅ΡΠΊΡΠ»ΠΎΠ² Π. <a class="reference external" href="https://github.com/marselester/reusable-django-app-skeleton">Reusable Django app skeleton</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="django-todo" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-10">[8]</a></td><td>ΠΠ°Π²Π»Π΅ΡΠΊΡΠ»ΠΎΠ² Π. <a class="reference external" href="https://github.com/marselester/django-todo">Π‘ΠΈΡΡΠ΅ΠΌΠ° ΡΠΏΡΠ°Π²Π»Π΅Π½ΠΈΡ ΡΠ΅ΠΏΠΎΡΠΊΠ°ΠΌΠΈ Π·Π°Π΄Π°Ρ</a>.</td></tr>
</tbody>
</table>
</div>
ΠΡΡΠΈΡΠ»ΠΈΡΠ΅Π»ΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΎΠ΄Π½ΠΎΠΌΠ΅ΡΠ½ΠΎΠΉ ΠΎΠΏΡΠΈΠΌΠΈΠ·Π°ΡΠΈΠΈ2010-10-06T00:00:00+07:002010-10-06T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2010-10-06:/computing-methods-of-one-dimensional-optimization.html<p>ΠΠ° ΡΡΠ΅ΡΡΠ΅ΠΌ ΠΊΡΡΡΠ΅ ΠΏΠΎ ΠΏΡΠ΅Π΄ΠΌΠ΅ΡΡ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΎΠΏΡΠΈΠΌΠΈΠ·Π°ΡΠΈΠΈ Π΄Π΅Π»Π°Π»ΠΈ Π»Π°Π±ΠΎΡΠ°ΡΠΎΡΠ½ΡΡ ΡΠ°Π±ΠΎΡΡ Π½Π°
ΡΠ΅ΠΌΡ Β«ΠΡΡΠΈΡΠ»ΠΈΡΠ΅Π»ΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΎΠ΄Π½ΠΎΠΌΠ΅ΡΠ½ΠΎΠΉ ΠΎΠΏΡΠΈΠΌΠΈΠ·Π°ΡΠΈΠΈΒ».
ΠΠ°Π΄Π°ΡΠ° Π·Π°ΠΊΠ»ΡΡΠ°Π»Π°ΡΡ Π² ΠΏΠΎΠΈΡΠΊΠ΅ Π±Π΅Π·ΡΡΠ»ΠΎΠ²Π½ΠΎΠ³ΠΎ ΠΌΠΈΠ½ΠΈΠΌΡΠΌΠ° ΡΡΠ½ΠΊΡΠΈΠΈ
<tt class="docutils literal">f(x) = pow(x, 3) β x + pow(e, <span class="pre">-x)</span></tt> Π½Π° Π½Π°ΡΠ°Π»ΡΠ½ΠΎΠΌ ΠΈΠ½ΡΠ΅ΡΠ²Π°Π»Π΅ <tt class="docutils literal">[0, 1]</tt>
Ρ ΡΠΎΡΠ½ΠΎΡΡΡΡ <tt class="docutils literal">0.00001</tt>.</p>
<p>ΠΡΡΠΈΡΠ»Π΅Π½ΠΈΡ ΠΏΡΠΎΠΈΠ·Π²ΠΎΠ΄ΠΈΠ»ΠΈΡΡ ΡΠ΅ΡΠ΅Π·:</p>
<ul class="simple">
<li>ΠΏΠ°ΡΡΠΈΠ²Π½ΡΠΉ ΠΌΠ΅ΡΠΎΠ΄;</li>
<li>ΡΠ°Π²Π½ΠΎΠΌΠ΅ΡΠ½ΡΠ΅ Π±Π»ΠΎΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ β¦</li></ul><p>ΠΠ° ΡΡΠ΅ΡΡΠ΅ΠΌ ΠΊΡΡΡΠ΅ ΠΏΠΎ ΠΏΡΠ΅Π΄ΠΌΠ΅ΡΡ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΎΠΏΡΠΈΠΌΠΈΠ·Π°ΡΠΈΠΈ Π΄Π΅Π»Π°Π»ΠΈ Π»Π°Π±ΠΎΡΠ°ΡΠΎΡΠ½ΡΡ ΡΠ°Π±ΠΎΡΡ Π½Π°
ΡΠ΅ΠΌΡ Β«ΠΡΡΠΈΡΠ»ΠΈΡΠ΅Π»ΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ ΠΎΠ΄Π½ΠΎΠΌΠ΅ΡΠ½ΠΎΠΉ ΠΎΠΏΡΠΈΠΌΠΈΠ·Π°ΡΠΈΠΈΒ».
ΠΠ°Π΄Π°ΡΠ° Π·Π°ΠΊΠ»ΡΡΠ°Π»Π°ΡΡ Π² ΠΏΠΎΠΈΡΠΊΠ΅ Π±Π΅Π·ΡΡΠ»ΠΎΠ²Π½ΠΎΠ³ΠΎ ΠΌΠΈΠ½ΠΈΠΌΡΠΌΠ° ΡΡΠ½ΠΊΡΠΈΠΈ
<tt class="docutils literal">f(x) = pow(x, 3) β x + pow(e, <span class="pre">-x)</span></tt> Π½Π° Π½Π°ΡΠ°Π»ΡΠ½ΠΎΠΌ ΠΈΠ½ΡΠ΅ΡΠ²Π°Π»Π΅ <tt class="docutils literal">[0, 1]</tt>
Ρ ΡΠΎΡΠ½ΠΎΡΡΡΡ <tt class="docutils literal">0.00001</tt>.</p>
<p>ΠΡΡΠΈΡΠ»Π΅Π½ΠΈΡ ΠΏΡΠΎΠΈΠ·Π²ΠΎΠ΄ΠΈΠ»ΠΈΡΡ ΡΠ΅ΡΠ΅Π·:</p>
<ul class="simple">
<li>ΠΏΠ°ΡΡΠΈΠ²Π½ΡΠΉ ΠΌΠ΅ΡΠΎΠ΄;</li>
<li>ΡΠ°Π²Π½ΠΎΠΌΠ΅ΡΠ½ΡΠ΅ Π±Π»ΠΎΡΠ½ΡΠ΅ ΠΌΠ΅ΡΠΎΠ΄Ρ;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ Π·ΠΎΠ»ΠΎΡΠΎΠ³ΠΎ ΡΠ΅ΡΠ΅Π½ΠΈΡ;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ ΡΠΈΡΠ΅Π» Π€ΠΈΠ±ΠΎΠ½Π°ΡΡΠΈ;</li>
<li>ΠΌΠ΅ΡΠΎΠ΄ ΠΊΠ°ΡΠ°ΡΠ΅Π»ΡΠ½ΠΎΠΉ.</li>
</ul>
<p>ΠΠ°ΠΈΠ»ΡΡΡΠΈΠΌ ΠΏΠΎ ΠΊΠΎΠ»ΠΈΡΠ΅ΡΡΠ²Ρ ΡΠΊΡΠΏΠ΅ΡΠΈΠΌΠ΅Π½ΡΠΎΠ² ΠΎΠΊΠ°Π·Π°Π»ΡΡ ΠΌΠ΅ΡΠΎΠ΄ ΡΠΈΡΠ΅Π» Π€ΠΈΠ±ΠΎΠ½Π°ΡΡΠΈ,
Π½Π°ΠΈΡ
ΡΠ΄ΡΠΈΠΌ β ΠΏΠ°ΡΡΠΈΠ²Π½ΡΠΉ ΠΌΠ΅ΡΠΎΠ΄. ΠΠ°ΠΈΠ»ΡΡΡΠΈΠΌ ΠΏΠΎ Π²ΡΠ΅ΠΌΠ΅Π½ΠΈ ΡΠ°Π±ΠΎΡΡ ΠΎΠΊΠ°Π·Π°Π»ΡΡ ΠΌΠ΅ΡΠΎΠ΄
Π·ΠΎΠ»ΠΎΡΠΎΠ³ΠΎ ΡΠ΅ΡΠ΅Π½ΠΈΡ, Π½Π°ΠΈΡ
ΡΠ΄ΡΠΈΠΌ β ΠΏΠ°ΡΡΠΈΠ²Π½ΡΠΉ ΠΌΠ΅ΡΠΎΠ΄. Π’Π°ΠΊ ΠΆΠ΅ Π±ΡΠ»ΠΎ ΡΡΡΠ°Π½ΠΎΠ²Π»Π΅Π½ΠΎ, ΡΡΠΎ
Π΄Π»Ρ Π΄Π°Π½Π½ΠΎΠΉ ΡΡΠ½ΠΊΡΠΈΠΈ Π±Π»ΠΎΡΠ½ΡΠΉ ΠΌΠ΅ΡΠΎΠ΄ ΡΠ°Π±ΠΎΡΠ°Π΅Ρ Π»ΡΡΡΠ΅ Ρ Π±Π»ΠΎΠΊΠ°ΠΌΠΈ ΡΠ°Π·ΠΌΠ΅ΡΠ°ΠΌΠΈ 3 ΠΈ 8, Π΄Π»Ρ
Π½Π΅ΡΠ΅ΡΠ½ΠΎΠ³ΠΎ ΠΈ ΡΠ΅ΡΠ½ΠΎΠ³ΠΎ ΡΠΎΠΎΡΠ²Π΅ΡΡΡΠ²Π΅Π½Π½ΠΎ.</p>
<p>ΠΡΠ»ΠΈ ΠΊΠΎΠΌΡ-Π½ΠΈΠ±ΡΠ΄Ρ ΠΏΡΠΈΠ³ΠΎΠ΄ΠΈΠ»ΠΎΡΡ β <a class="reference external" href="https://dl.dropbox.com/u/15875449/fx.zip">Π°ΡΡ
ΠΈΠ² Ρ ΠΈΡΡ
ΠΎΠ΄Π½ΠΈΠΊΠ°ΠΌΠΈ</a> Π½Π° php.</p>
ΠΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ Π½Π°ΠΆΠ°ΡΠΈΡ ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΠΈ ΠΊΠ»Π°Π²ΠΈΡ ΡΡΠ΅Π΄ΡΡΠ²Π°ΠΌΠΈ BIOS Π½Π° Π°ΡΡΠ΅ΠΌΠ±Π»Π΅ΡΠ΅2009-12-03T00:00:00+07:002009-12-03T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2009-12-03:/definition-of-pressing-of-a-combination-of-keys-by-means-BIOS-on-the-assembler.html<p>ΠΠΎ ΡΡΠ΅Π±Π΅ ΠΏΠΎΠ½Π°Π΄ΠΎΠ±ΠΈΠ»ΠΎΡΡ Π½Π°ΠΏΠΈΡΠ°ΡΡ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ Π½Π° Π°ΡΡΠ΅ΠΌΠ±Π»Π΅ΡΠ΅, ΠΊΠΎΡΠΎΡΠ°Ρ Π΄ΠΎΠ»ΠΆΠ½Π°
ΡΠ°ΡΠΏΠΎΠ·Π½Π°ΡΡ Π½Π°ΠΆΠ°ΡΠΈΠ΅ Β«Π³ΠΎΡΡΡΠ΅ΠΉΒ» ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΠΈ ΠΊΠ»Π°Π²ΠΈΡ <tt class="docutils literal">LeftCtrl+RightShift+F3</tt> ΠΈ
ΡΠ΅Π°Π³ΠΈΡΠΎΠ²Π°ΡΡ Π½Π° Π½Π΅Π³ΠΎ Π·Π²ΡΠΊΠΎΠ²ΡΠΌ ΡΠΈΠ³Π½Π°Π»ΠΎΠΌ. ΠΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠΈ/ΠΏΡΠΈΠΌΠ΅ΡΠΎΠ² ΠΏΠΎ ΡΡΠΎΠΉ ΡΠ΅ΠΌΠ΅
ΠΌΠ°Π»ΠΎΠ²Π°ΡΠΎ, ΠΏΠΎ ΡΡΠΎΠΌΡ ΡΠ΅ΡΠΈΠ» ΠΎΠΏΡΠ±Π»ΠΈΠΊΠΎΠ²Π°ΡΡ ΡΠ²ΠΎΡ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΊΡ.</p>
<div class="highlight"><pre><span></span><span class="nf">masm</span><span class="w"></span>
<span class="nf">.model</span><span class="w"> </span><span class="nv">small</span><span class="w"></span>
<span class="nf">.stack</span><span class="w"> </span><span class="mi">256</span><span class="w"></span>
<span class="nf">.data</span><span class="w"></span>
<span class="w"> </span><span class="nf">Msg_about</span><span class="w"> </span><span class="nv">db</span><span class="w"> </span><span class="s">'Π Π°ΡΠΏΠΎΠ·Π½Π°ΡΡ Π½Π°ΠΆΠ°ΡΠΈΠ΅ Β«Π³ΠΎΡΡΡΠ΅ΠΉΒ» ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΠΈ ΠΊΠ»Π°Π²ΠΈΡ'</span><span class="p">,</span><span class="w"> </span><span class="mh">0Ah</span><span class="p">,</span><span class="w"> </span><span class="mh">0Dh β¦</span></pre></div><p>ΠΠΎ ΡΡΠ΅Π±Π΅ ΠΏΠΎΠ½Π°Π΄ΠΎΠ±ΠΈΠ»ΠΎΡΡ Π½Π°ΠΏΠΈΡΠ°ΡΡ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ Π½Π° Π°ΡΡΠ΅ΠΌΠ±Π»Π΅ΡΠ΅, ΠΊΠΎΡΠΎΡΠ°Ρ Π΄ΠΎΠ»ΠΆΠ½Π°
ΡΠ°ΡΠΏΠΎΠ·Π½Π°ΡΡ Π½Π°ΠΆΠ°ΡΠΈΠ΅ Β«Π³ΠΎΡΡΡΠ΅ΠΉΒ» ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΠΈ ΠΊΠ»Π°Π²ΠΈΡ <tt class="docutils literal">LeftCtrl+RightShift+F3</tt> ΠΈ
ΡΠ΅Π°Π³ΠΈΡΠΎΠ²Π°ΡΡ Π½Π° Π½Π΅Π³ΠΎ Π·Π²ΡΠΊΠΎΠ²ΡΠΌ ΡΠΈΠ³Π½Π°Π»ΠΎΠΌ. ΠΠ½ΡΠΎΡΠΌΠ°ΡΠΈΠΈ/ΠΏΡΠΈΠΌΠ΅ΡΠΎΠ² ΠΏΠΎ ΡΡΠΎΠΉ ΡΠ΅ΠΌΠ΅
ΠΌΠ°Π»ΠΎΠ²Π°ΡΠΎ, ΠΏΠΎ ΡΡΠΎΠΌΡ ΡΠ΅ΡΠΈΠ» ΠΎΠΏΡΠ±Π»ΠΈΠΊΠΎΠ²Π°ΡΡ ΡΠ²ΠΎΡ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΠΊΡ.</p>
<div class="highlight"><pre><span></span><span class="nf">masm</span><span class="w"></span>
<span class="nf">.model</span><span class="w"> </span><span class="nv">small</span><span class="w"></span>
<span class="nf">.stack</span><span class="w"> </span><span class="mi">256</span><span class="w"></span>
<span class="nf">.data</span><span class="w"></span>
<span class="w"> </span><span class="nf">Msg_about</span><span class="w"> </span><span class="nv">db</span><span class="w"> </span><span class="s">'Π Π°ΡΠΏΠΎΠ·Π½Π°ΡΡ Π½Π°ΠΆΠ°ΡΠΈΠ΅ Β«Π³ΠΎΡΡΡΠ΅ΠΉΒ» ΠΊΠΎΠΌΠ±ΠΈΠ½Π°ΡΠΈΠΈ ΠΊΠ»Π°Π²ΠΈΡ'</span><span class="p">,</span><span class="w"> </span><span class="mh">0Ah</span><span class="p">,</span><span class="w"> </span><span class="mh">0Dh</span><span class="w"></span>
<span class="w"> </span><span class="kd">db</span><span class="w"> </span><span class="s">'LeftCtrl+RightShift+F3'</span><span class="p">,</span><span class="w"> </span><span class="mh">0Ah</span><span class="p">,</span><span class="w"> </span><span class="mh">0Dh</span><span class="w"></span>
<span class="w"> </span><span class="kd">db</span><span class="w"> </span><span class="s">'ΠΈ ΡΠ΅Π°Π³ΠΈΡΠΎΠ²Π°ΡΡ Π½Π° Π½Π΅Π³ΠΎ Π·Π²ΡΠΊΠΎΠ²ΡΠΌ ΡΠΈΠ³Π½Π°Π»ΠΎΠΌ'</span><span class="p">,</span><span class="w"> </span><span class="mh">0Ah</span><span class="p">,</span><span class="w"> </span><span class="mh">0Dh</span><span class="p">,</span><span class="w"> </span><span class="s">'$'</span><span class="w"></span>
<span class="nf">.code</span><span class="w"></span>
<span class="nl">start:</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΠ½ΠΈΡΠΈΠ°Π»ΠΈΠ·Π°ΡΠΈΡ ΡΠ΅Π³ΠΌΠ΅Π½ΡΠ½ΠΎΠ³ΠΎ ΡΠ΅Π³ΠΈΡΡΡΠ° ds</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ax</span><span class="p">,</span><span class="w"> </span><span class="err">@</span><span class="nv">data</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ds</span><span class="p">,</span><span class="w"> </span><span class="nb">ax</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΠΈΠ΄Π΅ΠΎΡΠ΅ΠΆΠΈΠΌ 3 (ΠΎΡΠΈΡΡΠΊΠ° ΡΠΊΡΠ°Π½Π° ΠΈ ΡΡΡΠ°Π½ΠΎΠ²ΠΊΠ° ΠΊΡΡΡΠΎΡΠ° Π² 0, 0)</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ax</span><span class="p">,</span><span class="w"> </span><span class="mh">0003h</span><span class="w"></span>
<span class="w"> </span><span class="nf">int</span><span class="w"> </span><span class="mh">10h</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΡΠ²ΠΎΠ΄ ΡΠΎΠΎΠ±ΡΠ΅Π½ΠΈΡ Π½Π° ΡΠΊΡΠ°Π½</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ah</span><span class="p">,</span><span class="w"> </span><span class="mi">9</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">dx</span><span class="p">,</span><span class="w"> </span><span class="nv">offset</span><span class="w"> </span><span class="nv">Msg_about</span><span class="w"></span>
<span class="w"> </span><span class="nf">int</span><span class="w"> </span><span class="mh">21h</span><span class="w"></span>
<span class="w"> </span><span class="c1">; Π§ΡΠ΅Π½ΠΈΠ΅ ΡΠΈΠΌΠ²ΠΎΠ»Π° Ρ ΠΎΠΆΠΈΠ΄Π°Π½ΠΈΠ΅ΠΌ</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ah</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="nf">int</span><span class="w"> </span><span class="mh">16h</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΡΠΎΠ²Π΅ΡΠΊΠ° Π½Π°ΠΆΠ°ΡΠΈΡ Ctrl+F3</span><span class="w"></span>
<span class="w"> </span><span class="nf">cmp</span><span class="w"> </span><span class="nb">ah</span><span class="p">,</span><span class="w"> </span><span class="mh">60h</span><span class="w"></span>
<span class="w"> </span><span class="nf">jne</span><span class="w"> </span><span class="nv">exit</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΠΎΠ»ΡΡΠ΅Π½ΠΈΠ΅ ΡΠΎΡΡΠΎΡΠ½ΠΈΡ ΠΊΠ»Π°Π²ΠΈΠ°ΡΡΡΡ</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ah</span><span class="p">,</span><span class="w"> </span><span class="mh">12h</span><span class="w"></span>
<span class="w"> </span><span class="nf">int</span><span class="w"> </span><span class="mh">16h</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΡΠΎΠ²Π΅ΡΠΊΠ° Π½Π°ΠΆΠ°ΡΠΈΡ LeftCtrl</span><span class="w"></span>
<span class="w"> </span><span class="nf">test</span><span class="w"> </span><span class="nb">ah</span><span class="p">,</span><span class="w"> </span><span class="mb">1b</span><span class="w"></span>
<span class="w"> </span><span class="nf">jz</span><span class="w"> </span><span class="nv">exit</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΡΠΎΠ²Π΅ΡΠΊΠ° Π½Π°ΠΆΠ°ΡΠΈΡ RightShift</span><span class="w"></span>
<span class="w"> </span><span class="nf">test</span><span class="w"> </span><span class="nb">al</span><span class="p">,</span><span class="w"> </span><span class="mb">1b</span><span class="w"></span>
<span class="w"> </span><span class="nf">jz</span><span class="w"> </span><span class="nv">exit</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ah</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="c1">; ΠΡΠ²ΠΎΠ΄ ΡΠΈΠΌΠ²ΠΎΠ»Π°</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">dl</span><span class="p">,</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="c1">; Π‘ΠΈΠ³Π½Π°Π»</span><span class="w"></span>
<span class="w"> </span><span class="nf">int</span><span class="w"> </span><span class="mh">21h</span><span class="w"></span>
<span class="w"> </span><span class="c1">; ΠΠ°Π²Π΅ΡΡΠ΅Π½ΠΈΠ΅ ΠΏΡΠΎΠ³ΡΠ°ΠΌΠΌΡ, Π²ΠΎΠ·Π²ΡΠ°Ρ ΡΠΏΡΠ°Π²Π»Π΅Π½ΠΈΡ ΠΠ‘</span><span class="w"></span>
<span class="w"> </span><span class="nl">exit:</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="nb">ax</span><span class="p">,</span><span class="w"> </span><span class="mh">4c00h</span><span class="w"></span>
<span class="w"> </span><span class="nf">int</span><span class="w"> </span><span class="mh">21h</span><span class="w"></span>
<span class="nf">end</span><span class="w"> </span><span class="nv">start</span><span class="w"></span>
</pre></div>
<p>ΠΠ»Ρ ΡΡΠ΅Π½ΠΈΡ ΡΠΈΠΌΠ²ΠΎΠ»Π° ΠΈΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΡΡΠ½ΠΊΡΠΈΡ <tt class="docutils literal">16h BIOS</tt>.</p>
<p>AH = 0</p>
<p>ΠΠ° Π²ΡΡ
ΠΎΠ΄Π΅ Π² AL = ASCII-ΠΊΠΎΠ΄ ΡΠΈΠΌΠ²ΠΎΠ»Π°, 0 ΠΈΠ»ΠΈ ΠΏΡΠ΅ΡΠΈΠΊΡ ΡΠΊΠ°Π½-ΠΊΠΎΠ΄Π°, ΠΠ = ΡΠΊΠ°Π½-ΠΊΠΎΠ΄
Π½Π°ΠΆΠ°ΡΠΎΠΉ ΠΊΠ»Π°Π²ΠΈΡΠΈ ΠΈΠ»ΠΈ ΡΠ°ΡΡΠΈΡΠ΅Π½Π½ΡΠΉ ASCII-ΠΊΠΎΠ΄.</p>
<p>ΠΠ°Π»Π΅Π΅ ΠΏΡΠΎΠΈΠ·Π²ΠΎΠ΄ΠΈΡΡΡ ΡΡΠ°Π²Π½Π΅Π½ΠΈΠ΅ ΡΠ΅Π³ΠΈΡΡΡΠ° AH ΡΠΎ ΡΠΊΠ°Π½-ΠΊΠΎΠ΄ΠΎΠΌ 60h (Π½Π°ΠΆΠ°ΡΠΈΠ΅ Ctrl+F3).</p>
<p>ΠΠΎΡΠΎΠΌ ΠΏΠΎΠ»ΡΡΠ°Π΅ΠΌ ΡΠΎΡΡΠΎΡΠ½ΠΈΡ ΠΊΠ»Π°Π²ΠΈΠ°ΡΡΡΡ. ΠΡΠΏΠΎΠ»ΡΠ·ΡΠ΅ΡΡΡ ΡΡΠ½ΠΊΡΠΈΡ <tt class="docutils literal">16h BIOS</tt>.</p>
<p>AH = 12h</p>
<p>ΠΠ° Π²ΡΡ
ΠΎΠ΄Π΅ Π² AX Π·Π°Π½ΠΎΡΠΈΡΡΡ ΡΠΎΡΡΠΎΡΠ½ΠΈΠ΅ ΠΊΠ»Π°Π²ΠΈΠ°ΡΡΡΡ. ΠΠ°Ρ ΠΈΠ½ΡΠ΅ΡΠ΅ΡΡΠ΅Ρ ΡΠΎΠ»ΡΠΊΠΎ ΠΏΠ΅ΡΠ²ΡΠΉ
Π±ΠΈΡ AH (LeftCtrl) ΠΈ ΠΏΠ΅ΡΠ²ΡΠΉ Π±ΠΈΡ AL (RightShift).</p>
<div class="section" id="section-1">
<h2>Π‘ΠΎΡΡΠΎΡΠ½ΠΈΠ΅ ΠΊΠ»Π°Π²ΠΈΠ°ΡΡΡΡ</h2>
<div class="section" id="al">
<h3>AL</h3>
<ul class="simple">
<li>ΠΠΈΡ 7: Ins</li>
<li>ΠΠΈΡ 6: CapsLock</li>
<li>ΠΠΈΡ 5: NumLock</li>
<li>ΠΠΈΡ 4: ScrollLock</li>
<li>ΠΠΈΡ 3: Alt (Π»ΡΠ±ΠΎΠΉ Alt Π΄Π»Ρ ΡΡΠ½ΠΊΡΠΈΠΈ 02h, ΡΠ°ΡΡΠΎ ΡΠΎΠ»ΡΠΊΠΎ Π»Π΅Π²ΡΠΉ Alt Π΄Π»Ρ 12h/22h)</li>
<li>ΠΠΈΡ 2: Ctrl (Π»ΡΠ±ΠΎΠΉ)</li>
<li>ΠΠΈΡ 1: LeftShift</li>
<li>ΠΠΈΡ 0: RightShift</li>
</ul>
</div>
<div class="section" id="ah">
<h3>AH</h3>
<ul class="simple">
<li>ΠΠΈΡ 7: SysRq</li>
<li>ΠΠΈΡ 6: CapsLock</li>
<li>ΠΠΈΡ 5: NumLock</li>
<li>ΠΠΈΡ 4: ScrollLock</li>
<li>ΠΠΈΡ 3: RightAlt</li>
<li>ΠΠΈΡ 2: RightCtrl</li>
<li>ΠΠΈΡ 1: LeftAlt</li>
<li>ΠΠΈΡ 0: LeftCtrl</li>
</ul>
</div>
</div>
ΠΠΎΠ΄Π΅Π»ΠΈΡΠΎΠ²Π°Π½ΠΈΠ΅ ΠΎΠ΄Π½ΠΎΠΊΠ°Π½Π°Π»ΡΠ½ΠΎΠΉ Π‘ΠΠ Ρ ΠΎΡΠΊΠ°Π·Π°ΠΌΠΈ2009-05-30T00:00:00+07:002009-05-30T00:00:00+07:00Marsel Mavletkulovtag:marselester.com,2009-05-30:/modeling-single-channel-queue-with-refusals.html<p>ΠΠ°Π½Π° ΠΎΠ΄Π½ΠΎΠΊΠ°Π½Π°Π»ΡΠ½Π°Ρ ΡΠΈΡΡΠ΅ΠΌΠ° ΠΌΠ°ΡΡΠΎΠ²ΠΎΠ³ΠΎ ΠΎΠ±ΡΠ»ΡΠΆΠΈΠ²Π°Π½ΠΈΡ Ρ ΠΎΡΠΊΠ°Π·Π°ΠΌΠΈ. Π Π½Π΅Π΅ ΠΏΠΎΡΡΡΠΏΠ°ΡΡ
Π·Π°ΡΠ²ΠΊΠΈ ΡΠ΅ΡΠ΅Π· ΠΏΡΠΎΠΌΠ΅ΠΆΡΡΠΎΠΊ Π²ΡΠ΅ΠΌΠ΅Π½ΠΈ <tt class="docutils literal">n</tt>, Π³Π΄Π΅ <tt class="docutils literal">n</tt> β ΡΠ»ΡΡΠ°ΠΉΠ½Π°Ρ Π²Π΅Π»ΠΈΡΠΈΠ½Π°,
ΠΏΠΎΠ΄ΡΠΈΠ½Π΅Π½Π½Π°Ρ ΡΠ°Π²Π½ΠΎΠΌΠ΅ΡΠ½ΠΎΠΌΡ Π·Π°ΠΊΠΎΠ½Ρ ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ. ΠΡΠ΅ΠΌΡ ΠΎΠ±ΡΠ»ΡΠΆΠΈΠ²Π°Π½ΠΈΡ Π·Π°ΡΠ²ΠΊΠΈ
ΡΠΈΡΡΠ΅ΠΌΠΎΠΉ <tt class="docutils literal">m</tt> ΡΠ°ΠΊΠΆΠ΅ ΡΠ²Π»ΡΠ΅ΡΡΡ ΡΠ»ΡΡΠ°ΠΉΠ½ΠΎΠΉ Π²Π΅Π»ΠΈΡΠΈΠ½ΠΎΠΉ Ρ ΠΏΠΎΠΊΠ°Π·Π°ΡΠ΅Π»ΡΠ½ΡΠΌ Π·Π°ΠΊΠΎΠ½ΠΎΠΌ
ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ. ΠΡΠ»ΠΈ ΠΊ ΠΌΠΎΠΌΠ΅Π½ΡΡ ΠΏΡΠΈΡ
ΠΎΠ΄Π° Π·Π°ΡΠ²ΠΊΠΈ ΠΊΠ°Π½Π°Π» Π·Π°Π½ΡΡ, Π·Π°ΡΠ²ΠΊΠ° ΠΏΠΎΠΊΠΈΠ΄Π°Π΅Ρ
ΡΠΈΡΡΠ΅ΠΌΡ Π½Π΅ΠΎΠ±ΡΠ»ΡΠΆΠ΅Π½Π½ΠΎΠΉ.</p>
<p>ΠΠ·Π½Π°ΡΠ°Π»ΡΠ½ΠΎ ΠΊΠΎΠ΄ Π±ΡΠ» β¦</p><p>ΠΠ°Π½Π° ΠΎΠ΄Π½ΠΎΠΊΠ°Π½Π°Π»ΡΠ½Π°Ρ ΡΠΈΡΡΠ΅ΠΌΠ° ΠΌΠ°ΡΡΠΎΠ²ΠΎΠ³ΠΎ ΠΎΠ±ΡΠ»ΡΠΆΠΈΠ²Π°Π½ΠΈΡ Ρ ΠΎΡΠΊΠ°Π·Π°ΠΌΠΈ. Π Π½Π΅Π΅ ΠΏΠΎΡΡΡΠΏΠ°ΡΡ
Π·Π°ΡΠ²ΠΊΠΈ ΡΠ΅ΡΠ΅Π· ΠΏΡΠΎΠΌΠ΅ΠΆΡΡΠΎΠΊ Π²ΡΠ΅ΠΌΠ΅Π½ΠΈ <tt class="docutils literal">n</tt>, Π³Π΄Π΅ <tt class="docutils literal">n</tt> β ΡΠ»ΡΡΠ°ΠΉΠ½Π°Ρ Π²Π΅Π»ΠΈΡΠΈΠ½Π°,
ΠΏΠΎΠ΄ΡΠΈΠ½Π΅Π½Π½Π°Ρ ΡΠ°Π²Π½ΠΎΠΌΠ΅ΡΠ½ΠΎΠΌΡ Π·Π°ΠΊΠΎΠ½Ρ ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ. ΠΡΠ΅ΠΌΡ ΠΎΠ±ΡΠ»ΡΠΆΠΈΠ²Π°Π½ΠΈΡ Π·Π°ΡΠ²ΠΊΠΈ
ΡΠΈΡΡΠ΅ΠΌΠΎΠΉ <tt class="docutils literal">m</tt> ΡΠ°ΠΊΠΆΠ΅ ΡΠ²Π»ΡΠ΅ΡΡΡ ΡΠ»ΡΡΠ°ΠΉΠ½ΠΎΠΉ Π²Π΅Π»ΠΈΡΠΈΠ½ΠΎΠΉ Ρ ΠΏΠΎΠΊΠ°Π·Π°ΡΠ΅Π»ΡΠ½ΡΠΌ Π·Π°ΠΊΠΎΠ½ΠΎΠΌ
ΡΠ°ΡΠΏΡΠ΅Π΄Π΅Π»Π΅Π½ΠΈΡ. ΠΡΠ»ΠΈ ΠΊ ΠΌΠΎΠΌΠ΅Π½ΡΡ ΠΏΡΠΈΡ
ΠΎΠ΄Π° Π·Π°ΡΠ²ΠΊΠΈ ΠΊΠ°Π½Π°Π» Π·Π°Π½ΡΡ, Π·Π°ΡΠ²ΠΊΠ° ΠΏΠΎΠΊΠΈΠ΄Π°Π΅Ρ
ΡΠΈΡΡΠ΅ΠΌΡ Π½Π΅ΠΎΠ±ΡΠ»ΡΠΆΠ΅Π½Π½ΠΎΠΉ.</p>
<p>ΠΠ·Π½Π°ΡΠ°Π»ΡΠ½ΠΎ ΠΊΠΎΠ΄ Π±ΡΠ» Π½Π°ΠΏΠΈΡΠ°Π½ Π½Π° php, Π²ΠΎΡ <a class="reference external" href="https://github.com/marselester/single-channel-queuing">ΠΏΡΠΈΠΌΠ΅Ρ Π½Π° python 3</a>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">random</span>
<span class="k">class</span> <span class="nc">RequestPoll</span><span class="p">:</span>
<span class="sd">"""Iterator that yields random requests and keeps statistic."""</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">time_to_finish</span><span class="p">,</span> <span class="n">intensity_of_service_flow</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">time_to_finish</span> <span class="o">=</span> <span class="n">time_to_finish</span>
<span class="bp">self</span><span class="o">.</span><span class="n">intensity_of_service_flow</span> <span class="o">=</span> <span class="n">intensity_of_service_flow</span>
<span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="s1">'Total: </span><span class="si">{}</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">total</span><span class="p">())</span> <span class="o">+</span>
<span class="s1">'Processed: </span><span class="si">{}</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">qty_of_processed_requests</span><span class="p">)</span> <span class="o">+</span>
<span class="s1">'Refused: </span><span class="si">{}</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">qty_of_refused_requests</span><span class="p">)</span> <span class="o">+</span>
<span class="s1">'Proportion of processed requests: </span><span class="si">{}</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">proportion_of_processed_requests</span><span class="p">())</span> <span class="o">+</span>
<span class="s1">'Probability of refuse: </span><span class="si">{}</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">probability_of_refuse</span><span class="p">())</span> <span class="o">+</span>
<span class="s1">'Absolute bandwidth: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">abs_bandwidth</span><span class="p">())</span>
<span class="p">)</span>
<span class="k">def</span> <span class="fm">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">time_when_channel_will_be_free</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">qty_of_processed_requests</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">qty_of_refused_requests</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">def</span> <span class="fm">__next__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">time_to_finish</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">StopIteration</span>
<span class="n">time_when_request_came_in</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span> <span class="o">+=</span> <span class="n">time_when_request_came_in</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_can_system_process_request</span><span class="p">():</span>
<span class="bp">self</span><span class="o">.</span><span class="n">qty_of_processed_requests</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">time_for_which_request_has_taken_channel</span> <span class="o">=</span> <span class="n">random</span> \
<span class="o">.</span><span class="n">expovariate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">intensity_of_service_flow</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">time_when_channel_will_be_free</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span> \
<span class="o">+</span> <span class="n">time_for_which_request_has_taken_channel</span>
<span class="k">return</span> <span class="s1">'request added to queue at </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">qty_of_refused_requests</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="s1">'requests refused at </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">_can_system_process_request</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">system_uptime</span> <span class="o">>=</span> <span class="bp">self</span><span class="o">.</span><span class="n">time_when_channel_will_be_free</span>
<span class="k">def</span> <span class="nf">total</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">qty_of_processed_requests</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">qty_of_refused_requests</span>
<span class="k">def</span> <span class="nf">abs_bandwidth</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">qty_of_processed_requests</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">time_to_finish</span>
<span class="k">def</span> <span class="nf">proportion_of_processed_requests</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">qty_of_processed_requests</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">total</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">probability_of_refuse</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">qty_of_refused_requests</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">total</span><span class="p">()</span>
<span class="n">intensity_of_service_flow</span> <span class="o">=</span> <span class="mf">0.5</span>
<span class="n">time_to_finish</span> <span class="o">=</span> <span class="mi">70</span>
<span class="n">request_poll</span> <span class="o">=</span> <span class="n">RequestPoll</span><span class="p">(</span><span class="n">time_to_finish</span><span class="p">,</span> <span class="n">intensity_of_service_flow</span><span class="p">)</span>
<span class="k">for</span> <span class="n">request</span> <span class="ow">in</span> <span class="n">request_poll</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">request_poll</span><span class="p">)</span>
</pre></div>