{"id":381,"date":"2013-12-13T13:00:02","date_gmt":"2013-12-13T13:00:02","guid":{"rendered":"http:\/\/writeasync.net\/?p=381"},"modified":"2013-11-26T19:38:17","modified_gmt":"2013-11-26T19:38:17","slug":"memorychannel-integration-test","status":"publish","type":"post","link":"http:\/\/writeasync.net\/?p=381","title":{"rendered":"MemoryChannel integration test"},"content":{"rendered":"<p>In a <a title=\"MemoryChannel and concurrency\" href=\"http:\/\/writeasync.net\/?p=311\">previous post<\/a>, I discussed the concurrency issues with the initial <code>MemoryChannel<\/code> implementation and how unit tests were insufficient to uncover them.<\/p>\n<p>I came up with these basic requirements\/invariants to guide my integration test design:<\/p>\n<ul>\n<li>Data from separately sent buffers <strong>must not<\/strong> be mixed or interleaved. That is, if sender 1 writes <code>\"FFFF\"<\/code> and sender 2 writes <code>\"CCCC\"<\/code>, the receiver may only see <code>\"CCCCFFFF\"<\/code> or <code>\"FFFFCCCC\"<\/code> (depending on how the writes were serialized).<\/li>\n<li>Assuming the channel is fully drained, all sent buffers <strong>must<\/strong> eventually be delivered; data <strong>must not<\/strong> be lost.<\/li>\n<li>The requirements above <strong>must<\/strong> hold given a single receiver and one or more senders operating concurrently.<\/li>\n<\/ul>\n<p>The basic test flow would thus be as follows:<\/p>\n<ol>\n<li>Create <code>MemoryChannel<\/code>.<\/li>\n<li>Start senders on background threads; run until canceled.<\/li>\n<li>Start async receiver; validate data after each receive operation.<\/li>\n<li>Run for some predetermined duration.<\/li>\n<li>Cancel senders; wait for tasks to complete.<\/li>\n<li>Dispose <code>MemoryChannel<\/code>; this will unblock the last receive operation.<\/li>\n<li>Validate sent and received data size.<\/li>\n<\/ol>\n<p>In the <a href=\"https:\/\/github.com\/brian-dot-net\/writeasync\/commit\/961f95afabf5f02d65d3eb434c1fa9fcf1588350\" title=\"&quot;Parameterize receive buffer size&quot;\">final refactoring<\/a>, the main body of the test code ended up as follows:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\nMemoryChannel channel = new MemoryChannel();\r\nint&#x5B;] sentDataSizes = new int&#x5B;] { 11, 19, 29, 41, 53, 71, 89, 101 };\r\n\r\nusing (CancellationTokenSource cts = new CancellationTokenSource())\r\n{\r\n    DataOracle oracle = new DataOracle();\r\n    Sender&#x5B;] senders = this.CreateSenders(channel, sentDataSizes, oracle);\r\n\r\n    ValidatingReceiver receiver = new ValidatingReceiver(channel, this.logger, this.receiveBufferSize, oracle);\r\n\r\n    Task&#x5B;] senderTasks = new Task&#x5B;senders.Length];\r\n    Task receiverTask = this.StartSendersAndReceiver(cts.Token, senders, receiver, senderTasks);\r\n    Thread.Sleep(this.duration);\r\n\r\n    cts.Cancel();\r\n    Task.WaitAll(senderTasks);\r\n\r\n    channel.Dispose();\r\n    receiverTask.Wait();\r\n\r\n    ValidateTransferredByteCount(senderTasks, receiverTask);\r\n}\r\n<\/pre>\n<p>My data validation strategy uses sent buffers of prime number sizes filled with a specific byte value per sender and a larger power of two for receive buffer size; for example, sender 1 uses a buffer of size 11 filled with <code>0x1<\/code>, sender 2 uses a buffer of size 19 filled with <code>0x2<\/code> and so on, with a receiver asking for 256 bytes at a time. My thinking was that it would be easier to detect state corruption if I used numbers like these (I admittedly have not done any rigorous mathematical proofs of this&#8230;). I implemented a <a title=\"&quot;Refactoring - move validation into ValidatingReceiver&quot;\" href=\"https:\/\/github.com\/brian-dot-net\/writeasync\/commit\/fa1423bb49d5df8d597674b748c6b8c67116b8da\"><code>ValidatingReceiver<\/code><\/a> which scans all received buffers and looks for runs of the same byte value. On each byte value change, the results are fed to a <a title=\"&quot;Add basic received data oracle&quot;\" href=\"https:\/\/github.com\/brian-dot-net\/writeasync\/commit\/218c004f976532e1dc69bec8c6466ad9b19fa0f3\"><code>DataOracle<\/code><\/a> which knows the mapping of expected byte values to buffer length multiples:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\npublic void VerifyLastSeen(byte lastSeen, int lastCount)\r\n{\r\n    int expectedCountMultiple;\r\n    if (!this.patterns.TryGetValue(lastSeen, out expectedCountMultiple))\r\n    {\r\n        string message = string.Format(\r\n            CultureInfo.InvariantCulture,\r\n            &quot;State corruption detected; byte 0x{0:X} was unexpected.&quot;,\r\n            lastSeen);\r\n        throw new InvalidOperationException(message);\r\n    }\r\n\r\n    if (lastCount % expectedCountMultiple != 0)\r\n    {\r\n        string message = string.Format(\r\n            CultureInfo.InvariantCulture,\r\n            &quot;State corruption detected; count of {0} for byte 0x{1:X} is not a multiple of {2}.&quot;,\r\n            lastCount,\r\n            lastSeen,\r\n            expectedCountMultiple);\r\n        throw new InvalidOperationException(message);\r\n    }\r\n}\r\n<\/pre>\n<p>By the <a title=\"&quot;Add missing file Delay.cs&quot;\" href=\"https:\/\/github.com\/brian-dot-net\/writeasync\/commit\/c86f1c3010e2d99ce094374863c92aff48efb554\">final commit<\/a>, I had a fully automated integration test app which gave me reasonable confidence that my logic was correct:<br \/>\n<code><br \/>\n[ . . . ]<br \/>\n[0035.048\/T01] Receive loop with 8 senders, 5.0 sec, send before receive=True, receive buffer=256...<br \/>\n[0035.048\/T01] Sender B=11\/F=0x1 starting...<br \/>\n[0035.048\/T01] Sender B=19\/F=0x2 starting...<br \/>\n[0035.048\/T01] Sender B=29\/F=0x3 starting...<br \/>\n[0035.049\/T01] Sender B=41\/F=0x4 starting...<br \/>\n[0035.049\/T01] Sender B=53\/F=0x5 starting...<br \/>\n[0035.049\/T01] Sender B=71\/F=0x6 starting...<br \/>\n[0035.049\/T01] Sender B=89\/F=0x7 starting...<br \/>\n[0035.050\/T01] Sender B=101\/F=0x8 starting...<br \/>\n[0035.050\/T01] Receiver starting...<br \/>\n[0040.050\/T35] Sender B=41\/F=0x4 completed. Sent 409631 bytes.<br \/>\n[0040.050\/T39] Sender B=101\/F=0x8 completed. Sent 1009091 bytes.<br \/>\n[0040.051\/T36] Sender B=53\/F=0x5 completed. Sent 529470 bytes.<br \/>\n[0040.051\/T38] Sender B=89\/F=0x7 completed. Sent 889288 bytes.<br \/>\n[0040.051\/T37] Sender B=71\/F=0x6 completed. Sent 709290 bytes.<br \/>\n[0040.051\/T33] Sender B=19\/F=0x2 completed. Sent 189886 bytes.<br \/>\n[0040.051\/T32] Sender B=11\/F=0x1 completed. Sent 109956 bytes.<br \/>\n[0040.051\/T34] Sender B=29\/F=0x3 completed. Sent 289826 bytes.<br \/>\n[0040.053\/T01] Receiver completed. Received 4136438 bytes.<br \/>\n[0040.053\/T01] Done.<br \/>\n<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In a previous post, I discussed the concurrency issues with the initial MemoryChannel implementation and how unit tests were insufficient to uncover them. I came up with these basic requirements\/invariants to guide my integration test design: Data from separately sent&hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,61,51],"tags":[],"class_list":["post-381","post","type-post","status-publish","format-standard","hentry","category-async","category-concurrency","category-testing"],"_links":{"self":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/381","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=381"}],"version-history":[{"count":0,"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/381\/revisions"}],"wp:attachment":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=381"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=381"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=381"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}