{"id":4001,"date":"2015-06-24T13:00:49","date_gmt":"2015-06-24T13:00:49","guid":{"rendered":"http:\/\/writeasync.net\/?p=4001"},"modified":"2015-06-24T01:02:43","modified_gmt":"2015-06-24T01:02:43","slug":"long-haul-two-years-later","status":"publish","type":"post","link":"http:\/\/writeasync.net\/?p=4001","title":{"rendered":"Long haul: two years later"},"content":{"rendered":"

When I presented my thoughts on long-haul testing<\/a> for PNSQC<\/a> in 2013, I had been steeped in the methodology for probably five years. It’s now two years later; what has changed?<\/p>\n

One thing I noticed while researching back then which still remains true today is the relatively sparse selection of published resources on the topic. One of the first web search results on long-haul specifically is a write-up (which I referenced in my paper) about the approach as used in the Windows CE testing strategy<\/a>. Repeating the search now, I did find one fairly comprehensive treatment of endurance testing by Steven Woody<\/a> for a 2009 Sticky Minds article. (If I had been aware of this one at the time, I definitely would have referenced it; either I overlooked it or it was somehow not readily available on the web.) I suspect this is partly a terminology issue; in the industry at large, “long-haul testing” seems somewhat less common than terms like “soak testing”<\/a> or “endurance testing”<\/a>.<\/p>\n

An aspect that I would give even more emphasis to today is the quality of the long-haul feedback loop. Broad, long-running tests are at their best when they act as gap detectors. As I said recently in “Destroy all* tests<\/a>“, it is good to have tests which can periodically uncover new issues. It is decidedly not<\/em> good, however, to see this and keep moving forward without making some sort of change in your testing strategy. Nearly any bug a long-haul test can find could also be found in a faster, more precise test — if only you knew to look. A single worrisome defect might open your mind to whole categories of issues which are simply awaiting the right set of unit tests to flush them out.<\/p>\n

In my paper, I mentioned only in passing the execution environment of a long-haul test, giving a few examples of test topologies (multi-threaded, multi-machine, etc.). The Me of 2015 would say, loudly, do as much as you can as close to production as you can<\/strong>. In the more cloud-focused world, trying to maintain distinct and specialized test environments can be a liability. If your software already runs on a distributed server farm<\/a>, it is worth exploring whether you can reserve real production capacity for validation purposes, e.g. by generating synthetic load<\/a>. You would need a secure and reliable way of isolating the impact of this traffic and perhaps a partitioning strategy to ensure your test load lands on the right nodes. If using “real prod” is not in the cards, a fully maintained staging environment would be the next best option. Note that it is a good idea to treat stage just like prod and use all the same deployment, access control, and incident management strategies; you don’t want to go down the road of “it worked on my [environment]” only to find that everything is different (and broken) in the real world.<\/p>\n

Would I still recommend long-haul testing? Absolutely! I would probably just “under-engineer” it a bit compared to how I described it in the paper, while stressing the above tenets.<\/p>\n","protected":false},"excerpt":{"rendered":"

When I presented my thoughts on long-haul testing for PNSQC in 2013, I had been steeped in the methodology for probably five years. It’s now two years later; what has changed? One thing I noticed while researching back then which… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51],"tags":[],"class_list":["post-4001","post","type-post","status-publish","format-standard","hentry","category-testing"],"_links":{"self":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/4001","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4001"}],"version-history":[{"count":0,"href":"http:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/4001\/revisions"}],"wp:attachment":[{"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4001"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4001"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4001"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}