<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en_US"><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://easabbir.github.io/blog/feed.xml" rel="self" type="application/atom+xml" /><link href="https://easabbir.github.io/" rel="alternate" type="text/html" hreflang="en_US" /><updated>2026-11-12T12:29:19+06:00</updated><id>https://easabbir.github.io/blog/feed.xml</id><title type="html">Ehsanul Alam Sabbir</title><subtitle>Staff SQA Engineer writing about test automation, performance testing, API testing and embedded firmware QA on real hardware.</subtitle><author><name>Ehsanul Alam Sabbir</name></author><entry><title type="html">From “To Be Validated” to Validated</title><link href="https://easabbir.github.io/blog/from-to-be-validated-to-validated/" rel="alternate" type="text/html" title="From “To Be Validated” to Validated" /><published>2026-11-10T10:09:00+06:00</published><updated>2026-11-10T10:09:00+06:00</updated><id>https://easabbir.github.io/blog/from-to-be-validated-to-validated</id><content type="html" xml:base="https://easabbir.github.io/blog/from-to-be-validated-to-validated/"><![CDATA[<p>The ticket was one line: “Add input echo in terminal. Echoed text should follow the logging text formatting.” Its status was To be validated, and it was assigned to QA. Here is how we turned that line into a test with a clear verdict.</p>

<h2 id="1-find-out-what-the-formatting-means">1. Find out what “the formatting” means</h2>

<p>The protocol specification defines the log line: module, level and message separated by tabs, ended by CR LF. It also says the echo must be at DEBUG level and contain the command exactly as received. The firmware source showed a deliberate choice: echo at INFO with a <code class="language-plaintext highlighter-rouge">&gt; </code> marker. Two sources, two answers.</p>

<h2 id="2-ask-dont-assume">2. Ask, don’t assume</h2>

<p>We wrote the conflict down with three options: check the format only, enforce the full spec, or lock in the firmware’s behaviour. The first decision was format only. On review, the level was added: the echo must be at DEBUG, as the spec says.</p>

<h2 id="3-write-the-test-from-the-spec">3. Write the test from the spec</h2>

<ul>
  <li>Four inputs: a plain command, a module command, a command with arguments and an unknown command.</li>
  <li>Read the raw serial bytes up to the echo, and match the exact line structure.</li>
  <li>Assert module CLI, level DEBUG, and that the line ends with the command as typed.</li>
  <li>A docstring with numbered steps, so the next reader knows what is verified.</li>
</ul>

<h2 id="4-verify-the-test-itself">4. Verify the test itself</h2>

<p>The format checks passed on the device. A deliberately broken expectation failed with a readable message. Offline checks confirmed the pattern accepts the spec’s form and rejects malformed lines.</p>

<h2 id="5-report-a-precise-verdict">5. Report a precise verdict</h2>

<p>The result is not “it works” or “it doesn’t”. The echo follows the log format; it is logged at INFO where the spec requires DEBUG. The ticket goes back with that sentence, the spec reference and a test that will turn green the moment the firmware is fixed.</p>

<h2 id="6-ship-it-the-normal-way">6. Ship it the normal way</h2>

<p>A feature branch from the main development branch, small focused commits, a pull request with the test results and the known deviation in the description, and review by a colleague before merge.</p>

<blockquote>
  <p><strong>Takeaway:</strong> Validation is not a thumbs-up. It is a precise statement of what matches the specification and what doesn’t, backed by a test that keeps checking it.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="qa-process" /><category term="validation" /><category term="workflow" /><summary type="html"><![CDATA[QA in an embedded feature workflow]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">When Expected Errors Look Like Failures</title><link href="https://easabbir.github.io/blog/when-expected-errors-look-like-failures/" rel="alternate" type="text/html" title="When Expected Errors Look Like Failures" /><published>2026-11-10T10:08:00+06:00</published><updated>2026-11-10T10:08:00+06:00</updated><id>https://easabbir.github.io/blog/when-expected-errors-look-like-failures</id><content type="html" xml:base="https://easabbir.github.io/blog/when-expected-errors-look-like-failures/"><![CDATA[<p>A common smoke check after a device reset is “no ERROR lines in the boot log”. It is simple and usually right. On our project it would have failed forever, even with perfect firmware.</p>

<h2 id="why">Why</h2>

<p>The firmware runs a built-in self-test at every boot. Some of its checks are negative: they deliberately write to a read-only object and request an unknown object ID, then verify that both are rejected. The rejection is logged by the lower layer at ERROR level, and the self-test then logs a PASS for that check:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[STORE]  [ERROR] write rejected: object is read-only
[BOOT]   [INFO]  PASS  read-only object rejects a write
[STORE]  [ERROR] unknown object id
[BOOT]   [INFO]  PASS  unknown object id is rejected
[BOOT]   [INFO]  SELFTEST PASS: 12 passed, 0 failed
</code></pre></div></div>

<p>Two ERROR lines, both proof that the device works. The reset test asserted that there were no ERROR lines at all, so it could never pass. It was failing earlier for a different reason, which hid this one. We found it by probing the self-test command directly and then checking old boot logs.</p>

<h2 id="how-to-handle-it">How to handle it</h2>

<ul>
  <li><strong>Allow-list the expected errors</strong> in the test: ignore exactly the known self-test lines and fail on anything else. Precise, but must be updated when the self-test changes.</li>
  <li><strong>Change the firmware</strong> so expected rejections inside a self-test are logged at a lower level. Cleaner, but a developer decision.</li>
  <li><strong>Assert on the summary</strong>, <code class="language-plaintext highlighter-rouge">SELFTEST PASS ... 0 failed</code>, and keep a separate, narrower error check for the rest of the boot sequence.</li>
</ul>

<p>Which one is right is a team decision, so we raised it instead of quietly loosening the test. One more lesson came with it: when a new self-test was added a few days later, the count of checks changed. Never hard-code the count; assert on “0 failed”.</p>

<blockquote>
  <p><strong>Takeaway:</strong> “No errors” is a claim about the whole log. Before you make it, find out which errors the system produces on purpose.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="assertions" /><category term="self-test" /><category term="boot-log" /><summary type="html"><![CDATA[Designing assertions around intentional negative checks]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">The Hardware Lab Is Part of the System Under Test</title><link href="https://easabbir.github.io/blog/the-lab-is-part-of-the-system/" rel="alternate" type="text/html" title="The Hardware Lab Is Part of the System Under Test" /><published>2026-11-10T10:07:00+06:00</published><updated>2026-11-10T10:07:00+06:00</updated><id>https://easabbir.github.io/blog/the-lab-is-part-of-the-system</id><content type="html" xml:base="https://easabbir.github.io/blog/the-lab-is-part-of-the-system/"><![CDATA[<p>Over two weeks we tested on five units of the same product. Several of our most confusing results had nothing to do with the firmware. Here are the lab problems we met, how each one looked from the test report, and how we told it apart from a real defect.</p>

<h2 id="1-com-ports-renumber-on-every-reconnect">1. COM ports renumber on every reconnect</h2>

<p>Each new unit, and sometimes the same unit after re-plugging, appeared on new COM port numbers. Even the port carrying the CLI moved from the second modem port to the first on one unit. <strong>Fix:</strong> the bench configuration file lists the ports per bench, and we find the CLI by sending a harmless info command to each port of the device’s serial number before every session.</p>

<h2 id="2-another-program-holds-the-port">2. Another program holds the port</h2>

<p>Every test reported skipped: interface not available. The cause was a serial terminal still connected to the port; Windows allows only one owner. <strong>Fix:</strong> check for terminal sessions first and treat “all skipped” as a failed run, never as a pass.</p>

<h2 id="3-the-device-drops-off-usb-during-a-reset">3. The device drops off USB during a reset</h2>

<p>After a reset command, the serial driver reported I/O errors and the ports briefly disappeared, because the module re-enumerates its USB interfaces while rebooting. <strong>Fix:</strong> the framework’s reset window expects this, waits for the device and reconnects. These errors are noise, not a firmware fault, and the test report should say so.</p>

<h2 id="4-a-unit-without-the-application">4. A unit without the application</h2>

<p>Two units answered the modem’s standard <code class="language-plaintext highlighter-rouge">AT</code> command with <code class="language-plaintext highlighter-rouge">OK</code> but never answered the CLI, and listening for 20 seconds showed no boot output. The module worked; the application was simply not flashed. <strong>Fix:</strong> a quick health check that separates “module alive” from “application running” before anyone opens a bug.</p>

<h2 id="5-running-the-wrong-selection-flashes-the-device">5. Running the wrong selection flashes the device</h2>

<p>When the CI pipeline gained a setup step that flashes firmware, running the whole project folder would also collect that step. <strong>Fix:</strong> run the product test folder explicitly on a desk bench.</p>

<blockquote>
  <p><strong>Takeaway:</strong> Before blaming the firmware, rule out the lab: ports, port owners, USB re-enumeration, and whether the application is even there. A five-minute bench check saves hours of debugging the wrong layer.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="hardware-lab" /><category term="debugging" /><category term="usb-serial" /><summary type="html"><![CDATA[Telling lab problems apart from firmware bugs]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Reading Firmware Source as a Tester</title><link href="https://easabbir.github.io/blog/reading-firmware-source-as-a-tester/" rel="alternate" type="text/html" title="Reading Firmware Source as a Tester" /><published>2026-11-10T10:06:00+06:00</published><updated>2026-11-10T10:06:00+06:00</updated><id>https://easabbir.github.io/blog/reading-firmware-source-as-a-tester</id><content type="html" xml:base="https://easabbir.github.io/blog/reading-firmware-source-as-a-tester/"><![CDATA[<p>Testers are often told to treat the system as a black box. For embedded firmware that is a waste of information. The source code of a command handler is a precise list of the inputs it accepts and the ways it says no. Reading it, read-only, turned our negative tests from guesses into a checklist.</p>

<h2 id="what-we-look-for">What we look for</h2>

<ul>
  <li><strong>Argument parsing:</strong> how many arguments, which formats, what happens with extra ones. One helper accepted decimal digits only, with no sign and an overflow check. That gives four cases immediately: a letter, a negative number, a value one past the 32-bit limit, and an extra argument.</li>
  <li><strong>Limits:</strong> the input line buffer had a fixed size, and a longer line was discarded with a specific error message. That is a precise, low-risk test of input robustness.</li>
  <li><strong>Error codes:</strong> the framework’s error enumeration maps each failure to the code the device prints, so a test can assert the exact code instead of “some error”.</li>
  <li><strong>Inconsistencies:</strong> some commands rejected extra arguments, others silently ignored them. Worth a question to the developers, and worth knowing before you send a reset command with an extra argument.</li>
</ul>

<h2 id="an-example">An example</h2>

<p>For a log-level command, the source showed: one decimal argument, nothing after it, and a value below the number of levels. The fault-injection cases follow directly:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"log"            missing value           -&gt; bad parameter
"log x"          not a number            -&gt; bad parameter
"log 9"          out of range            -&gt; bad parameter
"log 1 2"        extra argument          -&gt; bad parameter
"log 4294967296" overflows 32 bits       -&gt; bad parameter
</code></pre></div></div>

<p>Each case was then confirmed on the device with a read-only probe before it went into a test.</p>

<h2 id="ground-rules">Ground rules</h2>

<ul>
  <li>The source explains behaviour; the <strong>specification</strong> still decides what is correct. When the source disagrees with the spec, report it.</li>
  <li>Treat the firmware repository as <strong>read-only</strong> reference. Note the branch and commit you read.</li>
  <li>Check which branch holds the real code. In our case the default branch only held a shared framework; all application code lived on a feature branch.</li>
  <li>Never send destructive commands to learn their behaviour. Read the source for those instead.</li>
</ul>

<blockquote>
  <p><strong>Takeaway:</strong> You don’t need to write firmware to read it. An hour with the command handlers produces better edge cases than a day of guessing at the prompt.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="firmware" /><category term="negative-testing" /><category term="white-box" /><summary type="html"><![CDATA[Where the best negative test cases come from]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Fault-Injection Tests That Clean Up After Themselves</title><link href="https://easabbir.github.io/blog/fault-injection-tests-that-clean-up/" rel="alternate" type="text/html" title="Fault-Injection Tests That Clean Up After Themselves" /><published>2026-11-10T10:05:00+06:00</published><updated>2026-11-10T10:05:00+06:00</updated><id>https://easabbir.github.io/blog/fault-injection-tests-that-clean-up</id><content type="html" xml:base="https://easabbir.github.io/blog/fault-injection-tests-that-clean-up/"><![CDATA[<p>Fault-injection tests send the device things it should reject: unknown commands, bad parameters, values out of range. The device should refuse them and carry on. Proving the “carry on” part is where our first version went wrong.</p>

<h2 id="the-first-version">The first version</h2>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">unknown_command_FI</span><span class="p">(</span><span class="n">log_fetcher</span><span class="p">,</span> <span class="n">param</span><span class="p">):</span>
    <span class="n">error_logs</span> <span class="o">=</span> <span class="n">log_fetcher</span><span class="p">.</span><span class="n">send_and_get_until_pattern</span><span class="p">(</span><span class="n">param</span><span class="p">,</span> <span class="n">expected_error</span><span class="p">,</span> <span class="p">...)</span>
    <span class="n">recovery_logs</span> <span class="o">=</span> <span class="n">log_fetcher</span><span class="p">.</span><span class="n">send_and_get_until_pattern</span><span class="p">(</span><span class="s">"info"</span><span class="p">,</span> <span class="s">"name:"</span><span class="p">,</span> <span class="p">...)</span>

    <span class="k">assert</span> <span class="n">error_logs</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="n">expected_error</span><span class="p">)</span>
    <span class="k">assert</span> <span class="n">recovery_logs</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="s">"name:"</span><span class="p">)</span>
</code></pre></div></div>

<p>It passed. A reviewer pointed out the problem: if the first assertion fails, or anything raises an exception before the end, the recovery step never runs. On a shared device that means the next test may start with the hardware in an unknown state, and its failure gets blamed on the wrong thing.</p>

<h2 id="recovery-belongs-in-teardown">Recovery belongs in teardown</h2>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">pytest</span><span class="p">.</span><span class="n">fixture</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">verify_cli_recovery</span><span class="p">(</span><span class="n">log_fetcher</span><span class="p">):</span>
    <span class="k">yield</span>
    <span class="n">logs</span> <span class="o">=</span> <span class="n">log_fetcher</span><span class="p">.</span><span class="n">send_and_get_until_pattern</span><span class="p">(</span><span class="s">"info"</span><span class="p">,</span> <span class="s">"name:"</span><span class="p">,</span> <span class="n">module</span><span class="o">=</span><span class="s">"CLI"</span><span class="p">)</span>
    <span class="k">assert</span> <span class="n">logs</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="s">"name:"</span><span class="p">),</span> <span class="sa">f</span><span class="s">"No response after fault injection: </span><span class="si">{</span><span class="n">logs</span><span class="si">}</span><span class="s">"</span>

<span class="k">def</span> <span class="nf">unknown_command_FI</span><span class="p">(</span><span class="n">log_fetcher</span><span class="p">,</span> <span class="n">verify_cli_recovery</span><span class="p">,</span> <span class="n">param</span><span class="p">):</span>
    <span class="p">...</span>
</code></pre></div></div>

<p>Everything after <code class="language-plaintext highlighter-rouge">yield</code> runs whether the test passes or fails. If the device does not recover, pytest reports a teardown error on that test, which is exactly where the problem was caused.</p>

<h2 id="we-proved-it-on-the-device">We proved it on the device</h2>

<p>We broke the test’s assertion on purpose and checked the device log. The recovery command was sent right after the failure was reported, and the device answered. Then we restored the test.</p>

<h2 id="the-rule-we-follow-now">The rule we follow now</h2>

<ul>
  <li>Every test that changes device state gets a fixture that restores it: log level, time, configuration, stored objects.</li>
  <li>Every fault-injection test gets a recovery check in teardown, not at the end of the test body.</li>
  <li>Reuse the framework’s existing restore fixtures where they exist, for example one that resets the log level.</li>
</ul>

<p>The same review produced a smaller rule worth keeping: use a plain string for a simple substring match, and compile a regular expression only when you need one.</p>

<blockquote>
  <p><strong>Takeaway:</strong> On shared hardware, a test is responsible for the state it leaves behind. Put that responsibility where it cannot be skipped: in fixture teardown.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="fault-injection" /><category term="pytest-fixtures" /><category term="code-review" /><summary type="html"><![CDATA[A code-review lesson about teardown on real devices]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Who Tests the Tests?</title><link href="https://easabbir.github.io/blog/who-tests-the-tests/" rel="alternate" type="text/html" title="Who Tests the Tests?" /><published>2026-11-10T10:04:00+06:00</published><updated>2026-11-10T10:04:00+06:00</updated><id>https://easabbir.github.io/blog/who-tests-the-tests</id><content type="html" xml:base="https://easabbir.github.io/blog/who-tests-the-tests/"><![CDATA[<p>A green test proves one thing: it did not fail this time. It does not prove the test can fail. On hardware this matters even more, because a timeout, an empty buffer or a skipped step can all quietly produce a pass.</p>

<h2 id="the-habit">The habit</h2>

<p>After every new test passes on the device, we break its expectation on purpose and run it again:</p>

<ul>
  <li>Change the expected error text to something the device never prints.</li>
  <li>Remove a valid value from an allowed set, for example drop INFO from the accepted log levels.</li>
  <li>Point a check at a module that never logs the line.</li>
</ul>

<p>The test must fail, and the failure message must show what the device actually sent. Then we restore the file and confirm the diff is exactly the intended change. It takes a minute per test.</p>

<h2 id="why-it-pays-off-on-hardware">Why it pays off on hardware</h2>

<ul>
  <li><strong>Timeouts look like success.</strong> A fetch that waits two seconds and returns whatever arrived can make an assertion on an empty collection pass, depending on how it is written.</li>
  <li><strong>Skips look like progress.</strong> When a serial port is locked by another program, every test is skipped. A CI summary with no failures says nothing about the firmware.</li>
  <li><strong>Recovery must run on failure.</strong> One mutation check confirmed something different: that our fixture teardown still sends its recovery command after the test fails. The device log showed the command arriving right after the failure was reported.</li>
</ul>

<h2 id="reading-the-failure-message">Reading the failure message</h2>

<p>A good failure message is part of the test. Ours print the received lines, for example <code class="language-plaintext highlighter-rouge">'WRONG TEXT: abc' missing in ['unknown command: abc']</code>. When the real firmware regresses, the engineer reading the CI report sees the cause without opening a log viewer.</p>

<h2 id="offline-checks-when-the-hardware-is-away">Offline checks when the hardware is away</h2>

<p>When no unit was available, we tested the parsing logic of a new test against captured device output and hand-made bad inputs. It does not replace a device run, but it means the first hardware run tests the firmware, not our regular expression.</p>

<blockquote>
  <p><strong>Takeaway:</strong> Make it part of “done”: a new test is finished when it has passed on the device <strong>and</strong> failed on purpose with a readable message.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="mutation-testing" /><category term="test-quality" /><category term="pytest" /><summary type="html"><![CDATA[Mutation checks for hardware integration tests]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Testing a CLI the Way a Machine Reads It</title><link href="https://easabbir.github.io/blog/testing-a-cli-the-way-a-machine-reads-it/" rel="alternate" type="text/html" title="Testing a CLI the Way a Machine Reads It" /><published>2026-11-10T10:03:00+06:00</published><updated>2026-11-10T10:03:00+06:00</updated><id>https://easabbir.github.io/blog/testing-a-cli-the-way-a-machine-reads-it</id><content type="html" xml:base="https://easabbir.github.io/blog/testing-a-cli-the-way-a-machine-reads-it/"><![CDATA[<p>Our device logs every line in a fixed format: a module name, a level and the message, separated by tabs and ended by carriage return plus line feed. Test tools, log viewers and CI parsers all depend on that format. So when a feature ticket said “echoed input should follow the logging text formatting”, the question was how to test it properly.</p>

<h2 id="the-loose-test-that-passed-anyway">The loose test that passed anyway</h2>

<p>An existing test sent <code class="language-plaintext highlighter-rouge">help</code> and asserted that the word <code class="language-plaintext highlighter-rouge">help</code> came back. It passed. It would also have passed if the device echoed raw text with no formatting at all, which is exactly what the feature was meant to fix.</p>

<h2 id="reading-raw-bytes-not-parsed-logs">Reading raw bytes, not parsed logs</h2>

<p>A log parser is forgiving by design: it trims spaces and accepts variations. To test the format itself, we read the raw serial bytes up to the echo line and matched them against the specified structure:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">LOG_LINE</span> <span class="o">=</span> <span class="n">re</span><span class="p">.</span><span class="nb">compile</span><span class="p">(</span>
    <span class="sa">r</span><span class="s">"\[(?P&lt;module&gt;[A-Z_]{2,15}) *\]\t"</span>       <span class="c1"># [MODULE]&lt;TAB&gt;
</span>    <span class="sa">r</span><span class="s">"\[(?P&lt;level&gt;DEBUG|INFO|WARN|ERROR)\] ?\t"</span>  <span class="c1"># [LEVEL]&lt;TAB&gt;
</span>    <span class="sa">r</span><span class="s">"(?P&lt;line&gt;[^\r\n]*)\r\n"</span><span class="p">)</span>                  <span class="c1"># LINE&lt;CR&gt;&lt;LF&gt;
</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">LOG_LINE</span><span class="p">.</span><span class="n">fullmatch</span><span class="p">(</span><span class="n">echo_line</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">match</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span>
<span class="k">assert</span> <span class="n">match</span><span class="p">[</span><span class="s">"module"</span><span class="p">]</span> <span class="o">==</span> <span class="s">"CLI"</span>
<span class="k">assert</span> <span class="n">match</span><span class="p">[</span><span class="s">"level"</span><span class="p">]</span> <span class="o">==</span> <span class="s">"DEBUG"</span>
<span class="k">assert</span> <span class="n">match</span><span class="p">[</span><span class="s">"line"</span><span class="p">].</span><span class="n">endswith</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
</code></pre></div></div>

<p>We ran it for four kinds of input: a plain command, a command with a module prefix, a command with arguments, and an unknown command. Every input line must be echoed, not only the valid ones.</p>

<h2 id="checking-the-checker">Checking the checker</h2>

<p>Before trusting the pattern, we fed it real captured lines and deliberately wrong ones: raw text with no header, spaces instead of tabs, the wrong module, an invalid level, and no echo at all. Each was rejected with a message that shows the offending line. The spec’s own example was accepted.</p>

<h2 id="what-it-found">What it found</h2>

<p>The echo followed the log format, so the main part of the feature was done. But the spec also requires the echo at DEBUG level, and the firmware used INFO. With the level check in place the test fails, and the ticket goes back with a precise finding instead of a vague “looks fine”.</p>

<blockquote>
  <p><strong>Takeaway:</strong> When a requirement is about format, test the bytes. A parser that tidies the output before your assertion runs will hide exactly the defect you are looking for.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="cli-testing" /><category term="serial" /><category term="regex" /><category term="log-format" /><summary type="html"><![CDATA[Why "the text appears somewhere" is not a test]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Spec, Source, Device</title><link href="https://easabbir.github.io/blog/spec-source-device/" rel="alternate" type="text/html" title="Spec, Source, Device" /><published>2026-11-10T10:02:00+06:00</published><updated>2026-11-10T10:02:00+06:00</updated><id>https://easabbir.github.io/blog/spec-source-device</id><content type="html" xml:base="https://easabbir.github.io/blog/spec-source-device/"><![CDATA[<p>Where does a test’s expected value come from? In many projects the honest answer is “from whatever the system did when the test was written”. That locks in today’s behaviour, bugs included. On our embedded project we use three sources, in a fixed order of authority.</p>

<h2 id="1-the-specification-decides-what-is-correct">1. The specification decides what is correct</h2>

<p>The command list, the CLI protocol document and the subsystem requirements say what the device must do. If a test and the spec disagree, the test is wrong until someone changes the spec.</p>

<h2 id="2-the-firmware-source-explains-why-and-where-the-edges-are">2. The firmware source explains why and where the edges are</h2>

<p>Reading the command handlers tells us which arguments are validated, which error code each failure returns, and how long a line may be. That turns a vague “test invalid input” into a precise list of negative cases.</p>

<h2 id="3-the-device-confirms-both">3. The device confirms both</h2>

<p>A read-only probe on real hardware shows the exact bytes. Only then do we write the assertion.</p>

<h2 id="what-it-caught">What it caught</h2>

<p>An existing test checked the system-information command for fields named <code class="language-plaintext highlighter-rouge">git_sha</code> and <code class="language-plaintext highlighter-rouge">build_ver</code>. The firmware printed <code class="language-plaintext highlighter-rouge">git sha</code> with a space, and no build field at all. Before we had the documents, it looked like a firmware bug. The specification showed the opposite: the firmware matched the spec, and the test was wrong on two fields.</p>

<p>The same cross-check worked the other way round. The spec says the CLI echoes each command at DEBUG level, exactly as received. The firmware echoed at INFO with a <code class="language-plaintext highlighter-rouge">&gt; </code> prefix, on purpose, according to its source comments. An existing test only checked that the command text appeared somewhere, so it passed. The deviation was invisible until we compared all three sources.</p>

<h2 id="making-it-a-routine">Making it a routine</h2>

<ul>
  <li>Keep the documents in one known place and always use the <strong>latest revision</strong>. Check for new ones before each round of test design.</li>
  <li>Note which <strong>firmware branch and commit</strong> the source analysis was based on.</li>
  <li>When two sources disagree, <strong>write it down and ask</strong>. Do not resolve it silently in the test.</li>
  <li>Report problems in the documents themselves too: copy-paste headings, missing syntax sections, revision numbers that don’t match the change log.</li>
</ul>

<blockquote>
  <p><strong>Takeaway:</strong> A test encodes a claim about correct behaviour. The three-source order makes sure that claim comes from the spec, not from whatever the firmware happened to do on the day the test was written.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="test-design" /><category term="specification" /><category term="firmware" /><summary type="html"><![CDATA[A three-source approach to embedded test design]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Testing Firmware on Real Hardware</title><link href="https://easabbir.github.io/blog/testing-firmware-on-real-hardware/" rel="alternate" type="text/html" title="Testing Firmware on Real Hardware" /><published>2026-11-10T10:01:00+06:00</published><updated>2026-11-10T10:01:00+06:00</updated><id>https://easabbir.github.io/blog/testing-firmware-on-real-hardware</id><content type="html" xml:base="https://easabbir.github.io/blog/testing-firmware-on-real-hardware/"><![CDATA[<p>In web QA, a failing test usually means a bug. In embedded QA, it might be the firmware, the test, the specification, the USB cable, or a terminal window someone left open on the test port. That difference shapes everything about how we test embedded products at Vivasoft.</p>

<p>This is the first article in a series about our embedded testing work. It describes a recent project: integration testing for a cellular IoT gateway, an application running directly on an LTE module that turns it into a communications co-processor for a smart-metering device.</p>

<h2 id="what-we-test-and-how-we-reach-it">What we test, and how we reach it</h2>

<p>The firmware exposes a debug command-line interface (CLI) over USB serial. Every command and every log line travels over that link, so the CLI is our window into the device. Our tests send commands, read the device’s log output, and assert on what comes back. There are no mocks: every test runs against a real unit on the bench.</p>

<ul>
  <li><strong>pytest</strong> as the test runner, extended by the client’s in-house framework for serial devices, log capture and device resets.</li>
  <li><strong>Layered configuration:</strong> product constants, per-build information from the firmware release, and a per-bench file with the local COM ports. The same tests run on any bench.</li>
  <li><strong>Structured logs:</strong> every byte on the serial link is recorded and parsed into module, level and message, so a failing test links straight to the exact device traffic.</li>
  <li><strong>CI:</strong> a Jenkins pipeline flashes the target firmware and runs the suite on a dedicated test PC.</li>
</ul>

<h2 id="what-a-test-looks-like">What a test looks like</h2>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">pytest_item</span>
<span class="k">def</span> <span class="nf">unknown_command_FI</span><span class="p">(</span><span class="n">log_fetcher</span><span class="p">,</span> <span class="n">verify_cli_recovery</span><span class="p">,</span> <span class="n">param</span><span class="p">):</span>
    <span class="s">"""
    Verify that unknown commands are rejected and the CLI keeps responding
    1. Send the unknown command
    2. Assert an ERROR "unknown command: &lt;command&gt;" is logged
    3. Teardown: assert the CLI still answers a known command
    """</span>
    <span class="n">logs</span> <span class="o">=</span> <span class="n">log_fetcher</span><span class="p">.</span><span class="n">send_and_get_until_pattern</span><span class="p">(</span><span class="n">param</span><span class="p">,</span> <span class="sa">f</span><span class="s">"unknown command: </span><span class="si">{</span><span class="n">param</span><span class="si">}</span><span class="s">"</span><span class="p">,</span>
                                                  <span class="n">module</span><span class="o">=</span><span class="s">"CLI"</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">LogLevel</span><span class="p">.</span><span class="n">ERROR</span><span class="p">)</span>
    <span class="k">assert</span> <span class="n">logs</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="sa">f</span><span class="s">"unknown command: </span><span class="si">{</span><span class="n">param</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
</code></pre></div></div>

<p>Each test follows a happy-path (<code class="language-plaintext highlighter-rouge">_HP</code>) or fault-injection (<code class="language-plaintext highlighter-rouge">_FI</code>) naming convention, has a docstring with numbered steps, and puts device recovery in fixture teardown, so a failure never leaves the hardware in a bad state for the next test.</p>

<h2 id="three-habits-that-made-the-difference">Three habits that made the difference</h2>

<ul>
  <li><strong>Spec first, then source, then device.</strong> Expected behaviour comes from the specification. The firmware source explains the edge cases. The device confirms both. When they disagree, we report it instead of quietly picking one.</li>
  <li><strong>Probe before asserting.</strong> Before writing a test, we send the command by hand with read-only probes and record the exact response.</li>
  <li><strong>Test the tests.</strong> After a test passes, we break its expectation on purpose and confirm it fails with a clear message.</li>
</ul>

<blockquote>
  <p><strong>Takeaway:</strong> The rest of this series takes each of these habits apart, with the real problems they caught: a test that was wrong about the spec, a feature that looked done but wasn’t, and a lab full of COM ports that renumber themselves.</p>
</blockquote>]]></content><author><name>Ehsanul Alam Sabbir</name></author><category term="Embedded QA" /><category term="firmware" /><category term="iot" /><category term="pytest" /><category term="hardware-testing" /><category term="ci" /><summary type="html"><![CDATA[How we built an integration test suite for a cellular IoT gateway]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://easabbir.github.io/images/og-card.jpg" /><media:content medium="image" url="https://easabbir.github.io/images/og-card.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>