File: //usr/share/doc/rsyslog/html/tutorials/hash_sampling.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Hash-based Sampling — rsyslog 8.2006.0 documentation</title>
<link rel="stylesheet" href="../_static/classic.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/rsyslog.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '8.2006.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Development" href="../development/index.html" />
<link rel="prev" title="Random sampling" href="random_sampling.html" />
</head>
<body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../development/index.html" title="Development"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="random_sampling.html" title="Random sampling"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">rsyslog 8.2006.0 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Tutorials</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="hash-based-sampling">
<h1>Hash-based Sampling<a class="headerlink" href="#hash-based-sampling" title="Permalink to this headline">¶</a></h1>
<p>Rsyslog supports various sampling mechanisms. These can be used on client systems to save servers from getting overwhelmed. Here we introduce a new sampling mechanism “Hash-based sampling”.</p>
<p>Let’s consider the following setup of systems running services and generating logs.
There are three services A, B and C.
These services work together to create a request processing pipeline.
User request lands up at system A, which processes the request, generates logs and forwards it to service B.
B processes the request it received from A, generates logs and forwards it to C and so on and so forth.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">+-+-+-+-+-+-+</span> <span class="o">+-+</span> <span class="o">+-+</span> <span class="o">+-+</span> <span class="o">+-+-++-+-+</span>
<span class="o">|</span><span class="n">UserRequest</span><span class="o">|</span> <span class="o">-></span> <span class="o">|</span><span class="n">A</span><span class="o">|</span> <span class="o">-></span> <span class="o">|</span><span class="n">B</span><span class="o">|</span> <span class="o">-></span> <span class="o">|</span><span class="n">C</span><span class="o">|</span> <span class="o">-></span> <span class="o">|</span> <span class="n">Database</span> <span class="o">|</span>
<span class="o">+-+-+-+-+-+-+</span> <span class="o">+-+</span> <span class="o">+-+</span> <span class="o">+-+</span> <span class="o">+-+-++-+-+</span>
</pre></div>
</div>
<dl class="docutils">
<dt>Consider three sample user requests</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid1"</span><span class="p">,</span> <span class="s2">"data"</span><span class="p">:</span> <span class="s2">"payload_1_original"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid2"</span><span class="p">,</span> <span class="s2">"data"</span><span class="p">:</span> <span class="s2">"payload_2_original"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"data"</span><span class="p">:</span> <span class="s2">"payload_3_original"</span><span class="p">}</span>
</pre></div>
</div>
</dd>
<dt>Service A generated logs</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid1"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"A"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"none"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid2"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"A_prime"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"Unknow flow"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"A_double_prime"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"Parsing Failed"</span><span class="p">}</span>
</pre></div>
</div>
</dd>
<dt>Service B generated logs</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid1"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"B"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"none"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid2"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"A_Failed_at_1st_Stage"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"none"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"B_Failed"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"Field not found"</span><span class="p">}</span>
</pre></div>
</div>
</dd>
<dt>Service C generated logs</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid1"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"C"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"none"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid2"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"NO_OP"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"none"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"C_Failed"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"NullPointer"</span><span class="p">}</span>
</pre></div>
</div>
</dd>
</dl>
<p>We can sample logs based on request_id thereby getting <strong>all-or-none</strong> logs associated with a request_id. This is a kind of transactional guarantee on logs for a request ID. This helps to create an end to end picture of pipeline even on sampled data.</p>
<dl class="docutils">
<dt>Using hash64mod for sampling</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span>if (hash64mod($!msg!request_id, 100) <= 30) then {
//send out
}
</pre></div>
</div>
</dd>
<dt>With hash64mod sampling, we get sampled logs like</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"data"</span><span class="p">:</span> <span class="s2">"payload_3_original"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"A_double_prime"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"Parsing Failed"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"B_Failed"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"Field not found"</span><span class="p">}</span>
<span class="p">{</span><span class="s2">"request_id"</span><span class="p">:</span> <span class="s2">"reqid3"</span><span class="p">,</span> <span class="s2">"datetime"</span><span class="p">:</span> <span class="s2">"<time>"</span><span class="p">,</span> <span class="s2">"Processing_state"</span> <span class="p">:</span> <span class="s2">"C_Failed"</span><span class="p">,</span> <span class="s2">"Exception"</span> <span class="p">:</span> <span class="s2">"NullPointer"</span><span class="p">}</span>
</pre></div>
</div>
</dd>
</dl>
<p>We will get all-or-none logs associated to hash-string (request ID in this case or combination of field_1 & field_2 & field_3) giving a full view of a request’s life cycle in the cluster across various systems. The only constraint in hash-based sampling is, it will work only if we have the same unique identifier (single field or combination of fields) in logs across services.</p>
<p>Same as hash64mod, we can use hash64 to add hash as tags that can be used later to filter logs.</p>
<dl class="docutils">
<dt>Using hash64 for filtering later</dt>
<dd><div class="first last highlight-default"><div class="highlight"><pre><span></span>set $.hash = hash64($!msg!field_1 & $!msg!field_2 & $!msg!field_3)
set $!tag= $syslogtag & $.hash;
//send out
</pre></div>
</div>
</dd>
</dl>
<p>Hash-based sampling is helpful on a single system too. Say a single process generates 4 log lines for each transaction it commits. We can get all-or-none guarantees around these 4 logs generated for each transaction. Thereby giving a complete picture of a transaction (while sampling).</p>
<dl class="docutils">
<dt><strong>Note</strong></dt>
<dd><ul class="first last simple">
<li>Hash-based sampling can be used only when logs have <strong>same unique identifier</strong> across services.</li>
<li>Default hash implementation is djb2 hash and xxhash can be enabled using compile-time flag (hash implementation can change release to release, check changelog).</li>
<li>For parsing log to generate json to act on individual fields use mmnormalize.</li>
</ul>
</dd>
</dl>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p>Help with configuring/using <code class="docutils literal"><span class="pre">Rsyslog</span></code>:</p>
<ul class="last simple">
<li><a class="reference external" href="http://lists.adiscon.net/mailman/listinfo/rsyslog">Mailing list</a> - best route for general questions</li>
<li>GitHub: <a class="reference external" href="https://github.com/rsyslog/rsyslog/">rsyslog source project</a> - detailed questions, reporting issues
that are believed to be bugs with <code class="docutils literal"><span class="pre">Rsyslog</span></code></li>
<li>Stack Exchange (<a class="reference external" href="https://stackexchange.com/filters/327462/rsyslog">View</a>, <a class="reference external" href="https://serverfault.com/questions/ask?tags=rsyslog">Ask</a>)
- experimental support from rsyslog community</li>
</ul>
</div>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p>Contributing to <code class="docutils literal"><span class="pre">Rsyslog</span></code>:</p>
<ul class="last simple">
<li>Source project: <a class="reference external" href="https://github.com/rsyslog/rsyslog/blob/master/README.md">rsyslog project README</a>.</li>
<li>Documentation: <a class="reference external" href="https://github.com/rsyslog/rsyslog-doc/blob/master/README.md">rsyslog-doc project README</a></li>
</ul>
</div>
<p>Copyright 2008-2020 <a class="reference external" href="https://rainer.gerhards.net/">Rainer Gerhards</a>
(<a class="reference external" href="https://www.rainer-gerhards.de/grossrinderfeld/">Großrinderfeld</a>),
and Others.</p>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h4>Previous topic</h4>
<p class="topless"><a href="random_sampling.html"
title="previous chapter">Random sampling</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="../development/index.html"
title="next chapter">Development</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/tutorials/hash_sampling.rst.txt"
rel="nofollow">Show Source</a></li>
<li><a href="https://github.com/rsyslog/rsyslog-doc/edit/master/source/tutorials/hash_sampling.rst"
rel="nofollow">Edit on GitHub</a></li>
</ul>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="../development/index.html" title="Development"
>next</a> |</li>
<li class="right" >
<a href="random_sampling.html" title="Random sampling"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="../index.html">rsyslog 8.2006.0 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="index.html" >Tutorials</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
</div>
</body>
</html>