[{"content":"Shipping an AI-built product? I help startups harden it for production and set up AI coding agents that fit how their team works. More at Lykotek →\nProjects # cache-apt-pkgs-action: Cache APT packages in GitHub Actions. ts-apt: TypeScript library for APT and APT-fast package operations on Debian-based Linux systems. vscode-keep-sorted: VSCode linter extension for google/keep-sorted. PcapngFile: Reads PCAP Next Generation files and generates CLR objects from its data. fluentcpp: This library can be used to express information in a functional way that allows for easy auto-complete and shorter expressions. ","date":"15 October 2026","externalUrl":null,"permalink":"/","section":"","summary":"Shipping an AI-built product? I help startups harden it for production and set up AI coding agents that fit how their team works. More at Lykotek →\nProjects # cache-apt-pkgs-action: Cache APT packages in GitHub Actions. ts-apt: TypeScript library for APT and APT-fast package operations on Debian-based Linux systems. vscode-keep-sorted: VSCode linter extension for google/keep-sorted. PcapngFile: Reads PCAP Next Generation files and generates CLR objects from its data. fluentcpp: This library can be used to express information in a functional way that allows for easy auto-complete and shorter expressions. ","title":"","type":"page"},{"content":"Some computer based reading I have enjoyed over the years. For everything else I\u0026rsquo;m reading, see My GoodReads.\nCyberpunk \u0026amp; Hard Sci-Fi # Cyberpunk # Akira - In the backdrop of post-WWIII Neo-Tokyo, two friends\u0026rsquo; relationship becomes darker as one realizes their strong psychokinetic powers. Blade Runner / Do Androids Dream of Electric Sheep? - Backdrop is a detective in futuristic 2019 charged with killing replicants. Contrasts human vs artificial intelligence (AI), the central theme is the examination of humanity. CyberStorm - A massive cyberattack cuts New York off from power and communications in the middle of a blizzard, leaving one family to survive the collapse. Daemon and Freedom™ - After a legendary game designer dies, a distributed program he left behind starts dismantling and rebuilding society. The Ghost in the Shell - In the near future, humans are augmented with cybernetic improvements to traits such as vision, strength, and intelligence. Story follows counter-terrorism operative and cyborg, Major Motoko Kusanagi. MaddAddam Trilogy - A corporate dystopian future where civil society doesn\u0026rsquo;t exist, just the corps. The story follows multiple characters involved in a bio attack. Neuromancer - Set in the future, a washed-up computer hacker, Henry Case is hired for one last job. This brings him up against a powerful AI. Reamde - Ransomware spread through a massively multiplayer online game pulls the game creator\u0026rsquo;s family into a global chase with hackers, the Russian mob, and terrorists. Service Model - A robot valet that has apparently murdered its master sets out across a collapsing world of malfunctioning automation looking for a new purpose. Ubik - In a future where the dead are kept in \u0026ldquo;half-life\u0026rdquo;, the reality around a team of anti-psychic agents starts to decay after a job goes wrong. Upgrade - A gene-crime investigator discovers his own genome has been upgraded, the first step in a plan to upgrade all of humanity. Hard Sci-Fi # Blindsight - A crew of transhuman specialists makes first contact with an alien intelligence that may not be conscious at all, questioning whether consciousness is needed for intelligence. Bobiverse - A software engineer wakes up as the uploaded mind of a self-replicating Von Neumann probe and starts copying himself across the galaxy. Children of Time - Humanity\u0026rsquo;s last survivors reach a terraformed planet where an uplift experiment has produced an evolving civilization of spiders. Darwin\u0026rsquo;s Radio - Retroviruses dormant in human DNA activate and trigger what may be the next step in human evolution. The Expanse - In a colonized solar system, a missing-person case and a destroyed ice hauler uncover an alien protomolecule, with physics and orbital mechanics kept grounded throughout the nine-book series. Project Hail Mary - A science teacher wakes alone on a spacecraft with no memory and has to work out, from first principles, how to save Earth from a star-dimming microbe. Seveneves - The Moon shatters, giving Earth two years before a debris bombardment, and humanity engineers its survival in orbit over the next five thousand years. Computer Science # The Art of Computer Programming Synopsis: Multi volume books meant to cover all the different mathematics used to write algorithms. Volumes are separated based on math concept. Opinion: Don\u0026rsquo;t expect to read these back to back. Take it in parts and for topics that you are interested in. For example I found the section on hashing interesting when I was trying to understand it better. Introduction to Algorithms (CLRS) Synopsis: Canonical Algorithms teaching book. Opinion: I have a personal affinity for this book due to its simplicity in format and diagrams. IMO it isn\u0026rsquo;t too wordy like other books and gives me what I need to know quickly. Compilers: Principles, Techniques, and Tools (Dragon Book) Synopsis: Provides information for building a compiler and the different features involved. Opinion: Really loved this book in school. Gave me a good sense of compiler basics. Unfortunately it doesn\u0026rsquo;t cover more advanced topics like type reification but it shouldn\u0026rsquo;t be expected as starter book. Computer History # The Cathedral \u0026amp; the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary - Observations by Eric S. Raymond of Linux development and Open Source projects in general while managing his own (i.e. fetchmail). Cult of the Dead Cow: How the Original Hacking Supergroup Might Just Save the World Synopsis: Hacktivism where its followers are battling electoral misinformation, making personal data safer, and battling to keep technology a force for good instead of for surveillance and oppression. Opinion: As a 90\u0026rsquo;s kid of the nascent personal Internet this is very nostalgic for me. Great read if you want to get a sense of the hacking ethos at the time. The Dream Machine: J.C.R. Licklider and the Revolution That Made Computing Personal - Outlines Licklider\u0026rsquo;s visionary dream of a \u0026ldquo;human-computer symbiosis\u0026rdquo; transforming the course of modern science and led to the development of the personal computer. Turing\u0026rsquo;s Cathedral: The Origins of the Digital Universe Synopsis: A history of computation and the creation of the computer. Opinion: Gave me a deep sense of the contribution that philosophy and mathematics had on computers, and a new appreciation of its history. Software # Design \u0026amp; Architecture # Clean Code Design Patterns: Elements of Reusable Object-Oriented Software (Gang of Four Book) Object-Oriented Analysis and Design with Applications Patterns of Enterprise Application Architecture Refactoring: Improving the Design of Existing Code Maintenance \u0026amp; Management # Mythical Man-Month - Presents the multiple ideas of Fred Brooks with the central theme that \u0026ldquo;adding manpower to a late software project makes it later\u0026rdquo; (Brooks\u0026rsquo;s Law). Site Reliability Engineering - Outlines concepts used by Google SRE for managing large scale systems. These concepts have served me very well in my time at Google. If you want a good sense of devops posture, this is a must read. Landmark Papers # An archive of lectures by ACM Turing Award recipients can be found in Turing Lecture Award Papers.\nComputation # Leibniz, Gottfried Wilhelm (1703). \u0026ldquo;Explanation of Binary Arithmetic\u0026rdquo; Babbage, Charles (1824). \u0026ldquo;Note on the Application of Machinery to the Computation of Astronomical and Mathematical Tables\u0026rdquo; Boole, George (1854). \u0026ldquo;The Laws of Thought: Chapter V\u0026rdquo; Cantor, Georg (1874). \u0026ldquo;On a Property of the Collection of All Real Algebraic Numbers\u0026rdquo; Hilbert, David (1900). \u0026ldquo;Mathematical Problems\u0026rdquo; Gödel, Kurt (1931). \u0026ldquo;On Formally Undecidable Propositions of Principia Mathematica and Related Systems I\u0026rdquo; Turing, A.M. (1936). \u0026ldquo;On Computable Numbers, With an Application to the Entscheidungsproblem\u0026rdquo; von Neumann, John (1945). \u0026ldquo;First Draft of a Report on the EDVAC\u0026rdquo; Shannon, Claude (1948). \u0026ldquo;A Mathematical Theory of Communication\u0026rdquo; McCarthy, John et al. (1955). \u0026ldquo;A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence\u0026rdquo; Cook, Stephen (1971). \u0026ldquo;The Complexity of Theorem-Proving Procedures\u0026rdquo; Engineering # Codd, E.F. (1970). \u0026ldquo;A Relational Model of Data for Large Shared Data Banks\u0026rdquo; Cerf, Vinton and Kahn, Robert (1974). \u0026ldquo;A Protocol for Packet Network Intercommunication\u0026rdquo; Backus, John (1978). \u0026ldquo;Can Programming Be Liberated From the von Neumann Style?\u0026rdquo; Fielding, Roy (2000). \u0026ldquo;Architectural Styles and the Design of Network-based Software Architectures\u0026rdquo; (REST Dissertation) Dean, Jeffrey and Ghemawat, Sanjay (2004). \u0026ldquo;MapReduce: Simplified Data Processing on Large Clusters\u0026rdquo; DeCandia, Giuseppe et al. (2007). \u0026ldquo;Dynamo: Amazon\u0026rsquo;s Highly Available Key-value Store\u0026rdquo; ","date":"15 October 2026","externalUrl":null,"permalink":"/reading/","section":"","summary":"Reading list for topics related to Software.","title":"Reading","type":"page"},{"content":"","date":"25 December 2025","externalUrl":null,"permalink":"/tags/keep-sorted/","section":"Tags","summary":"","title":"Keep-Sorted","type":"tags"},{"content":"","date":"25 December 2025","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"25 December 2025","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"25 December 2025","externalUrl":null,"permalink":"/tags/vscode/","section":"Tags","summary":"","title":"VSCode","type":"tags"},{"content":"","date":"25 December 2025","externalUrl":null,"permalink":"/tags/vscode-extension/","section":"Tags","summary":"","title":"Vscode Extension","type":"tags"},{"content":" VSCode Keep Sorted Extension 0.0.1 Released I recently released lightweight VS Code extension that automatically keeps sorted blocks in your code organized using Google\u0026rsquo;s Keep Sorted formatter, seeing a gap in an available extension that handles code blocks generically and with options to modify behavior.\nThe extension has pretty comprehensive code coverage, diagnostics and error handling to create a more reliable extension.\nIt is a language agnostic sorting tool, working in any programming language or file type that provides:\n🔍 Automatic Detection: Recognizes // keep-sorted start and // keep-sorted end blocks in any file type 💡 Quick Fix Actions: To sort just the block, the entire file or the entire workspace. ✨ Autocomplete for Options: Get intelligent suggestions for keep-sorted options when typing on a keep-sorted start line 📊 Real-time Diagnostics Shows warnings with squiggly underlines for unsorted content Logging integration with VSCode with severity levels for general information and debugging ⚡ Performance Optimized Uses native binaries for fast sorting operations for x86, ARM, Darwin and Windows No outside libraries required to be brought into the development environment (i.e. Go compiler). 🎛️ Flexible Configuration: File exclusion patterns, enable/disable toggle, and optional file logging See http://github.com/awalsh128/vscode-keep-sorted/README.md for the complete documentation of the extension.\nExamples # Basic Sorting # JavaScript/TypeScript Imports # // keep-sorted start import { Component } from \u0026#34;@angular/core\u0026#34;; import { HttpClient } from \u0026#34;@angular/common/http\u0026#34;; import { Observable } from \u0026#34;rxjs\u0026#34;; import fs from \u0026#34;fs\u0026#34;; import path from \u0026#34;path\u0026#34;; // keep-sorted end Python Imports # # keep-sorted start import json import os import sys from datetime import datetime from typing import List, Dict # keep-sorted end Case-Insensitive Sorting # Use case=no to sort without considering case:\n// keep-sorted start case=no import { ComponentA } from \u0026#34;./components\u0026#34;; import { utilityB } from \u0026#34;./utils\u0026#34;; import { ServiceC } from \u0026#34;./services\u0026#34;; import { helperD } from \u0026#34;./helpers\u0026#34;; // keep-sorted end Numeric Sorting # Use numeric=yes to sort numbers by their numeric value instead of lexically:\n# keep-sorted start numeric=yes item_1 = \u0026#34;first\u0026#34; item_5 = \u0026#34;fifth\u0026#34; item_10 = \u0026#34;tenth\u0026#34; item_50 = \u0026#34;fiftieth\u0026#34; item_100 = \u0026#34;hundredth\u0026#34; # keep-sorted end Block Sorting # Use block=yes to sort complex multi-line blocks like structs or objects:\n// keep-sorted start block=yes const widgets = [ { name: \u0026#34;alpha\u0026#34;, value: 100, }, { name: \u0026#34;beta\u0026#34;, value: 200, }, { name: \u0026#34;delta\u0026#34;, value: 150, }, ]; // keep-sorted end Prefix Ordering # Use prefix_order to control the order of specific prefixes:\n// keep-sorted start prefix_order=INIT_,,FINAL_ INIT_DATABASE, INIT_LOGGING, PROCESS_DATA, VALIDATE_INPUT, FINAL_CLEANUP, FINAL_SHUTDOWN; // keep-sorted end Ignore Prefixes # Use ignore_prefixes to ignore certain prefixes when sorting:\n// keep-sorted start ignore_prefixes=const,let,var const apple = 1; let banana = 2; var cherry = 3; const date = 4; // keep-sorted end Custom Grouping # Use group_prefixes to keep related lines together:\n\u0026lt;!-- keep-sorted start group_prefixes=and,with --\u0026gt; hamburger with lettuce and tomatoes peanut butter and jelly spaghetti with meatballs \u0026lt;!-- keep-sorted end --\u0026gt; Newline Separated # Use newline_separated=yes to add blank lines between sorted items:\n# keep-sorted start newline_separated=yes def calculate_total(): pass def process_data(): pass def validate_input(): pass # keep-sorted end Skip Lines # Use skip_lines=N to skip header lines (useful for tables):\n\u0026lt;!-- keep-sorted start skip_lines=2 --\u0026gt; | Name | Value | | ----- | ----- | | Alpha | 100 | | Beta | 200 | | Delta | 150 | | Gamma | 175 | \u0026lt;!-- keep-sorted end --\u0026gt; Sticky Comments # Comments stick with the line below them by default:\n# keep-sorted start # Configuration for production PROD_URL = \u0026#34;https://prod.example.com\u0026#34; # Configuration for staging STAGING_URL = \u0026#34;https://staging.example.com\u0026#34; # Configuration for development DEV_URL = \u0026#34;http://localhost:3000\u0026#34; # keep-sorted end Regular Expression Sorting # Use by_regex to sort based on specific patterns:\n// keep-sorted start by_regex=\\w+; String bar; Object baz; List\u0026lt;String\u0026gt; foo; // keep-sorted end Remove Duplicates # Duplicates are removed by default, but you can disable it:\n# keep-sorted start remove_duplicates=no - apple - apple - banana - banana - cherry # keep-sorted end Usage # IMPORTANT: See google/keep-sorted#options for complete documentation on all the options that are available.\nSetting Up Keep Sorted Blocks # Add keep-sorted markers around the content you want to keep sorted:\n// keep-sorted start import { zebra } from \u0026#34;./animals\u0026#34;; import { apple } from \u0026#34;./fruits\u0026#34;; import { car } from \u0026#34;./vehicles\u0026#34;; // keep-sorted end The extension will detect when these blocks are not properly sorted and provide automatic fixes.\nSorting Methods # 1. Quick Fixes (Lightbulb) # When the extension detects unsorted content:\nA warning squiggle appears under the unsorted block Click the lightbulb 💡 icon or press Ctrl+. / Cmd+. Choose from available actions: \u0026ldquo;Sort all lines in block (keep-sorted)\u0026rdquo; - Sorts just the selected block \u0026ldquo;Sort all lines in file (keep-sorted)\u0026rdquo; - Sorts all keep-sorted blocks in the file 2. Command Palette # Use Ctrl+Shift+P / Cmd+Shift+P and search for:\n\u0026ldquo;Keep Sorted: Fix Current File\u0026rdquo; - Sort all keep-sorted blocks in the active file \u0026ldquo;Keep Sorted: Fix Entire Workspace\u0026rdquo; - Sort all keep-sorted blocks across all workspace files 3. Problems Panel # Open the Problems panel (Ctrl+Shift+M / Cmd+Shift+M) Look for keep-sorted warnings Use \u0026ldquo;Quick Fix\u0026rdquo; or \u0026ldquo;Fix All\u0026rdquo; options directly from the panel Supported File Types # Keep Sorted works with any file type and is language agnostic. Common use cases include:\nimport statements, lists, top level file types and classes, and many more. Troubleshooting # Extension Not Working? # Check Extension Status\nEnsure the extension is enabled in VS Code Verify keep-sorted.enabled is set to true in settings No Diagnostics Appearing?\nMake sure your blocks use the correct syntax: // keep-sorted start and // keep-sorted end Check if files are excluded via keep-sorted.exclude patterns Verify the file is saved (diagnostics appear after save) Commands Not Available?\nOpen Command Palette (Ctrl+Shift+P / Cmd+Shift+P) and search for \u0026ldquo;Keep Sorted\u0026rdquo; If commands don\u0026rsquo;t appear, try reloading VS Code (Developer: Reload Window) Verify the extension is activated (check status bar or Extensions view) Performance Issues?\nAdd problematic file patterns to keep-sorted.exclude Check VS Code\u0026rsquo;s Output panel (select \u0026ldquo;Keep Sorted\u0026rdquo; from dropdown) for error messages Viewing Extension Logs\nOpen the Output panel: View → Output or Ctrl+Shift+U / Cmd+Shift+U Select \u0026ldquo;Keep Sorted\u0026rdquo; from the dropdown menu Logs show extension activity, errors, and diagnostic information For persistent logs, configure keep-sorted.logFilepath in settings Fixes Are Malformed # This is usually due to the underlying Google keep-sorted command itself and have an incorrect block setup.\nAn example where the block isn\u0026rsquo;t inside the list.\n# keep-sorted start items = [ \u0026#39;zebra\u0026#39;, \u0026#39;alpha\u0026#39;, \u0026#39;delta\u0026#39;, \u0026#39;beta\u0026#39;, ] # keep-sorted end Breaks the syntax because non-alphanumeric characters have precedence in the sort order.\n# keep-sorted start ], items = [ \u0026#39;zebra\u0026#39;, \u0026#39;alpha\u0026#39;, \u0026#39;delta\u0026#39;, \u0026#39;beta\u0026#39; # keep-sorted end The command is working correctly but the declaration is incorrect. The correct block setup should be.\nitems = [ # keep-sorted start \u0026#39;zebra\u0026#39;, \u0026#39;alpha\u0026#39;, \u0026#39;delta\u0026#39;, \u0026#39;beta\u0026#39;, # keep-sorted end ] To ensure your statements are correct see google/keep-sorted#options for more information.\nGetting Help # Report Issues: GitHub Issues Feature Requests: GitHub Discussions Documentation: Keep Sorted Project License # This project is licensed under the Apache 2.0 License - see the LICENSE.md file for details.\n","date":"25 December 2025","externalUrl":null,"permalink":"/vscode-keep-sorted/","section":"Posts","summary":" VSCode Keep Sorted Extension 0.0.1 Released I recently released lightweight VS Code extension that automatically keeps sorted blocks in your code organized using Google’s Keep Sorted formatter, seeing a gap in an available extension that handles code blocks generically and with options to modify behavior.\n","title":"VSCode Keep Sorted Extension 0.0.1","type":"posts"},{"content":"","date":"14 February 2022","externalUrl":null,"permalink":"/tags/interview/","section":"Tags","summary":"","title":"Interview","type":"tags"},{"content":"If you have experience in the software engineering field, an interview process can test the experience you have; projects, production scenarios, designs. Unfortunately, for junior positions and those out of college, there isn\u0026rsquo;t much to test for. Since most are computer scientists, you could test knowledge of algorithms and data structures but does that really prove the ability of a candidate? Knowing the upper bound complexity of an algorithm\u0026rsquo;s runtime is important but what about when a constant multiplying factor means the difference between production being up and down. We don\u0026rsquo;t have a solution for the Traveling Salesman Problem but we have very good and practical approximations (e.g. flight scheduling).\nAs an engineer, we have to think about what is the best approximation and trade offs among different solutions when posed with a hard problem. While the CS field greatly informs software engineering and provides it with a fundamental basis to extend from, it doesn\u0026rsquo;t wholly prepare a junior software engineer for working in the field, especially in the parts that matter the most.\nSo what does this have to do with interviewing software engineers? Well it\u0026rsquo;s hard because they have little to no experience. That isn\u0026rsquo;t their fault. It is just the difficult situation that interviewers and interviewees find themselves in. What\u0026rsquo;s worse is that there are good jobs out there but the candidate ends up having a terrible interviewer.\nDynamics # As someone that has given many interviews, my worst nightmare is leading the candidate down a road I can\u0026rsquo;t pull them out of due to a bad question or guidance. For me the worst feeling is holding an interview that wastes the candidate\u0026rsquo;s time and the opportunity to accomplish their career goals. As an interviewer it is my responsibility that the problem be clear to the candidate and something that reflects actual skills needed.\nLack of Consistent Feedback # I have had interviewers go out of the room to take a bathroom break, sit on their phones, or just stare silently while I worked on the problem. An interview is just as much a chance to see how the candidate would work with you as you them. It is also an opportunity to evaluate their teamwork skills. This kind of work is not an island and having a poor team dynamic can ruin projects.\nAnother problem is when the interviewer lets you hang yourself with your own solution. It is important that the interviewer provide guidance and redirection. Not doing so wastes the entire time left instead of giving a candidate an opportunity to show what they can do after correction. I had an interview with a principal engineer once for their architecture interview (they break them up into categories). In it they asked me to find a way to count the number of unique exceptions in a production system. I went right into showing an architecture of designing a logging system and how the processing would work at scale. They had a line of questioning talking about the logging system itself and then at the end told me I could just hash the exceptions. I was pissed. If the interviewer had told me it would be a simple data structures question I would have answered it right away but was completely misled about the parameters of the interview. It was my fault that I made a simple solution a complex one, but the opportunity was completely missed on seeing what I could do, even if that meant I got marked down.\nToy Questions # IMO the worst kind of questions are the ones that don\u0026rsquo;t reflect real work at all. How is reversing a linked list or rotating a matrix going to show that I am a good engineer? Some people say that this just reveals how you think about things and while there is some merit to that, it can be very perilous. It requires the candidate to be well studied in these problem spaces first. A better question reflects not only how they approach something but the problem they are trying to solve. What compounds this problem is when an interviewer has a single toy question they ask all the time and evaluates performance based on their own long time familiarity with the problem. The worst is when the problem requires just one key observation to solving it.\nPoor Personality # I have high anxiety and have had interviews go terribly wrong. This is also why I try really hard to improve myself in the process and be an interviewer that gives the candidate the best opportunity of proving themselves. I have co-workers and friends that I consider to be very high quality engineers that would never had the chance if someone didn\u0026rsquo;t accommodate them in terms of personality.\nSo what do I mean by poor personality? The interviewer has the power to make or break in these situations. Imagine some of the difficult people that you had to deal with in your life. Now have them be your interviewer. Dispassionate, judgemental, uncooperative, critical, the stuff of nightmares.\nFeedback Goes Both Ways # If you have an interviewer like this don\u0026rsquo;t be afraid to call it out afterwards. Some of these kind of interviewers may score you well but they should be getting feedback that this isn\u0026rsquo;t how to handle it. Be polite but be pointed about the behavior. It may not matter but many companies want this kind of feedback. At the end of the day, any person committed to wanting great candidates cares about the process that gets them to that.\nIt\u0026rsquo;s Not All Bad # For the bad interviews that I have had or heard about, there are plenty of, if not more, good ones too. The interviewers were attentive, guiding and gave problems that were multi layered and faceted. Some of them I came out wanting to be on their team because the collaboration went so well.\nThe reason I call out the bad interviewers is because candidates tend to put the blame on themselves. There is plenty that can be done to make sure you pass an interview but it is important to know what to look for too. Who wants to work for a company whose values are reflected in their employees this way? The process is as much about you evaluating them as they you and a good interviewer should know this. Although this isn\u0026rsquo;t a blame game either. Even a bad interviewer could be the gatekeeper to a great job and you will never have to see them again after being hired.\nHow Should I Prepare # The bad interviewer is a caricature. An embodiment of all the things that could go wrong. Really most have a mix of qualities both positive and negative, intentional and not. You will just have to make sure you do well in spite of the circumstances given.\nBecome a data structures and algorithms gymnast. I call it this because it is the most rote part. It bites some candidates because they just didn\u0026rsquo;t do the work. Like a gymnast you need a bunch of practice and be able to contort yourself in different ways because every problem will be some variation.\nProblem Spaces # What is the problem that is trying to be solved more generally speaking? A problem space doesn\u0026rsquo;t refer to any specific solution but rather the end goal that needs to be met. If you become familiar with these, you can also identify some of the common components that are used to solve them. Below is an approximate sample of the problem spaces out there.\nStrings # Google Search - string interview questions\nFor example, in bio-informatic key sequencing, the problem space is in looking for specific patterns (e.g. genome sequencing). Most all sequences can be represented using a string abstraction. While the example of key sequencing is a real world example, it can be generalized to ask questions about strings. The generalized problem space would be strings. By strings I don\u0026rsquo;t mean words or text, I mean a sequence of symbols in the most abstract way.\nExcerpt of a Gene Sequence\nGiven the string problem space, some combinatorial factors, with examples, could be length, affix, repetition, similarity, and cardinality. Combinations of those being:\nLeast common subsequence. Largest common prefix. Most unique string. Longest repeated string. Number of unique characters (i.e. cardinality). Words that share the most characters. This is just a small example. Least common subsequence is a popular one but why stop there? If you can do the combinations then you will have mastery. Just remember an interviewer is trying to mix it up too.\nEncoding # Google Search - data compression interview questions\nClaude Shannon - The OG of Information Theory\nHow would you represent data in memory? Combinatorial factors could be cardinality, word size, and direction. Some combinations:\nNumber of unique values encountered. Counters of letters encountered. Compression for streaming data. Compression for bounded data. Record a sequence of values containing A, B and C. Lookup \u0026amp; Counting # Google Search - hashing interview questions Google Search - computer counting interview questions What is the best way to lookup data with the constraints given? Combinatorial factors could be cardinality, length, and match type (whole, partial).\nSpecific string of unbounded length. Strings that share the first 3 characters. Specific count of a single character in a corpus. Words who share the same 3 letters. Notice how there is some overlap here with strings, that is because lookup frequently is a subset of the string problem space. We could easily apply common solutions from the string problem space to these as well as hashing and graphs (i.e. trie).\nGraphs # Google Search - graph interview questions\nSpotting Spammers in Social Graph\nHow do you find specific relationships between entities? Combinatorial factors could be value cardinality, size, edge direction, edge weights and cycles.\nLeast cost path between two points. Detect all cycles. All paths to a point from all other points. Paths that never leads to a cycle. Reverse direction to every odd numbered node and detect all cycles. Remove edges in a cycle that would have the most cost. Keep in mind that a linked list and tree are just graphs with constraints. If you learn graphs, you learn trees and linked lists. It\u0026rsquo;s just with trees their is direction so it can be flipped.\nSorting # Google Search - sorting interview questions\nHow do you derive meaning (how is it practical?) by ordering some collections of values? Combinatorial factors could be value cardinality, stability, partiality, and comparability.\nGroup cards by color. Sort integers by partial order where some integer k can be 4 positions away from k+1 and k-1. Sort a stream of natural numbers. Group floats by their whole number and the order they appear (stable). Order a deck of cards by rank, color and suit. Dynamic Programming # Honestly I don\u0026rsquo;t study this and despise its use in interviews. Rather jadedly I would speculate that most interviewers wouldn\u0026rsquo;t even be able to come up with a DP solution from first principles. Identifying optimal sub-structure really just takes a lot of experience IMO. Much like induction proofs.\nI find a better approach is to work a real world problem from several solution approaches and then use the DP method intuitively. This comes after really understanding the problem and consequently its sub-structure when applied in successive steps (either iteratively or recursively).\nSpace \u0026amp; Runtime Are Cross Cutting # Google Search - space and time complexity analysis exercises\nFor all of these problem spaces, it is important to factor in time and space complexity as well. For example, merge sort is best suited for space because the re-ordering happens in blocks. However quick sort is considered the most performant for in-memory on average even though quick sort has the largest worst case. For lookups a bounded set of keys could just use an array (e.g. frequency counting letters).\nBonus points for pointing out engineering ramifications for choosing a solution even if it has the same complexity. It may be just a constant multiplier difference (e.g. 4n and n) but it could mean the difference of thousands of dollars in a production system\u0026rsquo;s resource usage.\nSome tips when analyzing complexity.\nHow do the variables grow as you take them to infinity? If given m and n, does one become insignificant / constant. Sometimes a candidate will give something like O(m*n) when in reality it is just O(m). Another way to look at it below. Think of complexity on average but be explicit. Maybe you are dealing with a set of strings. Let\u0026rsquo;s say the size of the set is m and the largest length of a string is n, is the space complexity really O(m*n). Possibly in a theoretical sense but realistically m will be the overwhelming factor; it would be O(m) on average. Tell the interviewer this though, don\u0026rsquo;t just give them the answer O(m) Think Out Loud # While going through your thought process, make sure you share it with the interviewer like you would in a job. It\u0026rsquo;s like spreading out your ideas on a table, picking each one up, investigating it, saying what the pros and cons are, and then going with the best candidate at the moment while also being open to improvement. Maybe your idea isn\u0026rsquo;t the best one but put something out there that you can execute on and refine later. This will give an opportunity for the interviewer to see how you approach problems and judge the qualities of it.\nAsk A Lot Of Questions # Really ask them. The worst scenario is that you assume too much and go down a bad path. Probing the problem shows that you are thinking about all the possible aspects and applying critical thinking. I have had candidates ask questions that I didn\u0026rsquo;t even ask myself about the problem (impressive). In my experience, the better candidates were very inquisitive. Be curious and learn more.\nThink Before You Type # Maybe you\u0026rsquo;re nervous, maybe you think you know the answer and you go straight to typing. Stop! Don\u0026rsquo;t do it. I have been guilty of this myself. Writing code is a matter of self expression for me, a way of speaking. Although I have learned in time that it can also be a way to code yourself into a corner or hit a dead end that requires lots of back tracking and deletion of code. It can also be a bad sign that you are a coder that doesn\u0026rsquo;t think before you act and can potentially waste large hours of time on bad paths. This is why design is highly valued in software engineering. It forces people to think about the hard problems before they run up against them in code. Same applies to an interview. I have seen candidates rush into a problem only to stop and realize they really didn\u0026rsquo;t understand it. Don\u0026rsquo;t do this. Stop and go over your design with the interviewer. Walk through the different approaches, considering each and then selecting the route you want to take. This complements the idea of thinking out loud.\nPractice Coding .. A LOT # When walking through mock interviews, always write out your solutions. In this COVID world, that means typing on a keyboard but it doesn\u0026rsquo;t hurt to do both. Especially if you get called up for an in-person interview.\nPick a Language # Don\u0026rsquo;t pick a language you think they would want you to write, pick one you are good at and know how to use both syntactically and idiomatically. Unfortunately there are companies that want you to code in a specific languages. Personally I think that is a poor choice for the company since they should be looking for talent but sometimes that\u0026rsquo;s just the way it goes. For me, I would pick Python because it is the easiest to write IMO.\nFor the language you use, make sure you know:\nthe syntax of the language, the idioms used, and the framework functions / classes involved. For example, know how to work with sequences (e.g. lists), manipulate symbols (e.g. casing strings), and things similar. This will come up naturally as you work on solutions. Whatever language you end up picking, always code out the solution in it. It will build up that muscle so you can easily translate your ideas to code quickly.\nBreak Up the Problem # Start top down. Meaning start with the top level function and then break up the problem into sub functions. Deal with easy ones first and then slowly work your way towards the hardest. A divide and conquer approach shows you can compartmentalize and break out the problem into sub problems. It is something that is impressive and a lot of candidates don\u0026rsquo;t do.\ndef solve_problem(): easy_values = solve_easy_subproblem() hard_values = solve_hard_subproblem() return do_trivial_thing(easy_values, hard_values) def solve_hard_subproblem() easypartofhard_values = solve_easypartofhard_subproblem() somewhathard_values = solve_somewhathard_subproblem() return do_code_timeconsuming_thing(easypartofhard_values, somewhathard_values) An additional upside to this is that if you don\u0026rsquo;t complete the solution in time, the interviewer is able to at least evaluate the parts that you wholly finished. This is especially helpful if you solved the most conceptually hard parts but just didn\u0026rsquo;t have enough time for the time consuming code part.\nManage Your Emotions # For some, interviewing is not emotionally taxing but for others it can be debilitating. Personally, this is a big problem for me. I froze up completely for an interview once. I couldn\u0026rsquo;t even answer a trivial problem. I was so panicked that I couldn\u0026rsquo;t even think straight. The interviewer was very kind about it all but there was nothing to evaluate, I failed. Some things that helped me:\nPractice until it is instinctual. Do gymnastics until you can cooly answer any interview question or at least get close to it. Interview problems are just combinations; if you learn the fundamentals and apply them in different arrangements, they aren\u0026rsquo;t so surprising. Mock interview a lot. Have a friend or use an online resource to constantly perform mock interviews. It will help you with anxiety if you have it and keep you accustomed to the interview format. When I have held mock interviews I have intentionally been cold but polite. It wasn\u0026rsquo;t to be mean but to prepare them. If you have different tiers of jobs you are interested in, interview with the lowest ones first. Less is on the line if you fail and it is the best thing because it is real. You can always try again. Most companies allow a candidate to try again after some time has passed. Just because you failed an interview doesn\u0026rsquo;t mean you are a bad candidate. It just means you were off or weren\u0026rsquo;t prepared enough. I\u0026rsquo;ll be honest, I failed my first interview with Google, and thought I wasn\u0026rsquo;t good enough. Five years later I tried again and got the job. You are good enough. Imposter syndrome is a big thing. Especially when you work with people that you perceive to be intelligent and/or well accomplished. Sure there are people that have an obvious raw high intelligence but in my experience that is not the majority, and even then, very incongruent. Many intelligent people who I have met are highly driven. Education is a large factor in intelligence. Also intelligence is not some monolith; people express their intelligence through different characteristics as I alluded to with the word incongruent. In other words, intelligence is fluid and if you are driven, you are good enough. Resources To Use # Competitive Programming # Sites like TopCoder are nice and a way for you to get your chops solving different problems but it is first and foremost a competitive programming site. It is meant to present challenges that are new and novel, eliciting the tersest amount of code in the smallest amount of time. This isn\u0026rsquo;t to say if you got good at this you wouldn\u0026rsquo;t do well in an interview, the opposite, but it requires you to intuit the ability to solve based on working through a lot of problems rather than learning the underlying problem spaces.\nInterview Question Databases # This pertains to sites like CareerCup where questions are posted that came from actual interviews with tech companies. Keep in mind that a lot of these get posted because they were difficult for the candidate. That means you are more likely to see harder questions and have false expectations of what is required from you. Take these sites with a grain of salt.\nOnline Exercises # Online IDEs and sets of exercises that you can drill on are useful (e.g. GeeksForGeeks, HackerRank). I usually pick the data structures and algorithms exercises and just grind on them several times until it becomes second nature. Experiment with different sites and see which one works best for you. The key is that you are able to get through the exercises fairly quickly. A good site should be able to build you up by difficulty so that you can get muscle strength.\nConclusion # That was a lot but I hope it helps. This is rather a brain dump for me of all the thoughts I have had on interviewing in my career. If I were to summarize in just a few points, it would be to:\nDrill and drill and drill everyday for several weeks. Go through all the data structures and algos. Go through all the problem spaces and make up combinations. Mock interview and challenge yourself to socially uncomfortable situations. Lastly and most importantly is to know that you can do this. In my experience, lack of preparation has been the biggest hit on candidates not getting a job. Put in the time and it will help a lot in getting there. Good luck!\n","date":"14 February 2022","externalUrl":null,"permalink":"/interviews-are-the-worst/","section":"Posts","summary":"If you have experience in the software engineering field, an interview process can test the experience you have; projects, production scenarios, designs. Unfortunately, for junior positions and those out of college, there isn’t much to test for. Since most are computer scientists, you could test knowledge of algorithms and data structures but does that really prove the ability of a candidate? Knowing the upper bound complexity of an algorithm’s runtime is important but what about when a constant multiplying factor means the difference between production being up and down. We don’t have a solution for the Traveling Salesman Problem but we have very good and practical approximations (e.g. flight scheduling).\n","title":"Interviews are the Worst","type":"posts"},{"content":"","date":"23 October 2021","externalUrl":null,"permalink":"/tags/bootstrap/","section":"Tags","summary":"","title":"Bootstrap","type":"tags"},{"content":"Compilers was one of my favorite classes in school alongside OS and architecture. It gave me a deeper understanding of what happened with my code behind the scenes and insight into how to create my own domain specific language. Knowing how compilers work create the link between the abstractions that developers use everyday and the machine executing them below.\nPhases # Compilation comes into five phases.\nScanner - Breaks up text into tokens. Parser - Arranges the tokens into relationships that form a tree. Symbol Table - Attributes metadata like type and scope to identifiers (e.g. variables, functions, classes). Intermediate Representation (IR) - Representation of the code that is typically a series of statements / commands that can be optimized before becoming the assembly code. Compiled Code - The actual assembly code that gets generated at the end as a series of opcodes with their corresponding data and control bits. Of course this description makes a lot of assumptions and doesn\u0026rsquo;t account for different approaches to compilation but serves well for the purpose of learning and demonstration. For the following phase breakdowns we will use a C like language with a simple set of constructs as a toy use case. Let\u0026rsquo;s get started\u0026hellip;\nScanner # This is also known as the lexer or tokenizer. It\u0026rsquo;s responsibility is to take text and create indivisible units / tokens out of it. For example, the following\u0026hellip;\nint x = 123; \u0026hellip;would be tokenized as int, x, =, 123, and ;.\nLet\u0026rsquo;s take a more complex example.\nvoid foo(int\u0026amp; bar) { bar = 123; } This would be tokenized as int, foo, (, int, \u0026amp;, bar, ), {, bar, =, 123, ;, and }.\nAs a contrast, let\u0026rsquo;s look at a Python like snippet\u0026hellip;\ndef foo(bar): bar = 123 Since these languages treat spaces as part of the language (unlike C-like ones), this would be tokenized as def, foo, (, bar, ), :, \\n, \\s, \\s, bar, =, and 123. This approach is a bit nuanced though as any spaces after : and before newline are not tokenized since they have no meaning for the compiled language (same goes for extraneous newlines). This is why the last C-like example could be expressed as.\nvoid foo(int\u0026amp;bar){bar=123;} The only case where spaces have any meaning is to separate identifiers.\nNow let\u0026rsquo;s take an example we will use going forward for all the phases.\nif (x \u0026gt; 2) { y = 1; } else { y = 2; } Parser # The parser phase transforms the tokens into relationships between them (semantical) where it takes the flat list of tokens and puts them into a tree structure (i.e. Abstract Syntax Tree (AST)). Here is what our example looks like.\nSomething useful to know is the difference between the scanner and parser phase.\nThe scanner phase is the lexical component, it is concerned with the structure of symbols / tokens (in this case a combination of letters, numbers and punctuation). The parser phase is the semantic component, it is concerned with the meaning of symbols as they relate in the context of one another. For example\u0026hellip; void foo() { int x = (1+1) + 1; } The parenthesis are the symbols present in the function declaration and the function body but they have completely different meaning in their relationship with the rest of the symbols.\nOne is in the context of a function declaration foo and denotes the start and end of declared parameters. The other imposes order of operation on the expression assigned to the variable x. This is what makes the parsing / semantic phase different than the scanner / lexical phase.\nSymbol Table # The symbol table provides information about the identifiers such as type and scope. Let\u0026rsquo;s take the canonical example we have been using and encapsulate it into a function so we can give it some scope.\nvoid foo(int x) { int y; if (x \u0026gt; 2) { y = 1; } else { y = 2; } } The code gets stored in the symbol table as\u0026hellip;\nSymbol Type Scope foo function, void global x int function parameter y int block scope A symbol table is defined and typically stored in each compilation unit that corresponds to an object file (e.g. *.o, *.obj). These tables are not only used for metadata but will also serve as lookups for other object files that are trying to resolve their own symbols in the linker phase.\nSlight Detour to Undefined Reference # Note this is around the phase where you will get errors like undefined reference. The linker in these cases is trying to link the symbol references between the object files. An example of an undefined reference.\n// This says, I promise this thing exists. Just keep compiling and // it will show up when the linker searches for it. extern int foo(); int bar() { int x = foo(); return x + 2; } The symbol table will be generated as\u0026hellip;\nSymbol Type Scope foo function, void extern (note this) bar function, int global x int block scope If foo isn\u0026rsquo;t included anywhere either directly or indirectly through headers, an undefined reference will get reported. In these cases, the linker couldn\u0026rsquo;t resolve the symbol scope, which has an extern placemarker waiting for linkage.\nBack on Track # Let\u0026rsquo;s now assume the extern int foo(); is defined in another file. The resolved symbol table becomes.\nSymbol Type Scope foo function, void int (note this has changed) bar function, int global x int block scope Intermediate Representation (IR) # Generated from the parser graph and symbol table is the intermediate code / representation. It is the code that gets generated before translating to assembly. The reason this exists as opposed to direct assembly translation is so that optimizations can be made before final code expression.\nIR generators come in different types and take on different degrees of responsibility. Examples of IRs are LLVM, Java bytecode, and Microsoft CIL.\n.NET CIL # Applying the .NET Common Intermediate Language (CIL) to our canonical example below\u0026hellip;\nvoid foo(int x) { int y; if (x \u0026gt; 2) { y = 1; } else { y = 2; } } \u0026hellip;becomes\u0026hellip;\n.method private hidebysig instance void foo(int32 x) cil managed { .maxstack 2 .locals init (int32 V_0, bool V_1) IL_0000: nop IL_0001: ldarg.1 IL_0002: ldc.i4.2 IL_0003: cgt IL_0005: ldc.i4.0 IL_0006: ceq IL_0008: stloc.1 IL_0009: ldloc.1 IL_000a: brtrue.s IL_0012 IL_000c: nop IL_000d: ldc.i4.1 IL_000e: stloc.0 IL_000f: nop IL_0010: br.s IL_0016 IL_0012: nop IL_0013: ldc.i4.2 IL_0014: stloc.0 IL_0015: nop IL_0016: ret } This is an instance of a stack based bytecode representation as opposed to a register based approach. The two approaches are not strictly one or the other, but typically a virtual machine uses one predominantly. The more popular JVM and CIL are stack based but sometimes leverage registers. For example, the CIL will store simple constants like integers or perform calculations using registers since it is faster in terms of memory access than the L1+ cache.\nCompiled Code # The last and final step is the machine code itself. This is usually the assembly code that gets ran. Using our previous example for the CIL code. This bytecode gets translated into x86 code as\u0026hellip;\nC.foo(Int32) L0000: push ebp L0001: mov ebp, esp L0003: sub esp, 0x10 L0006: xor eax, eax L0008: mov [ebp-0xc], eax L000b: mov [ebp-0x10], eax L000e: mov [ebp-4], ecx L0011: mov [ebp-8], edx L0014: cmp dword ptr [0x1adec1a8], 0 L001b: je short L0022 L001d: call 0x661dfc10 L0022: nop L0023: cmp dword ptr [ebp-8], 2 L0027: setg al L002a: movzx eax, al L002d: mov [ebp-0x10], eax L0030: cmp dword ptr [ebp-0x10], 0 L0034: je short L0042 L0036: nop L0037: mov dword ptr [ebp-0xc], 1 L003e: nop L003f: nop L0040: jmp short L004b L0042: nop L0043: mov dword ptr [ebp-0xc], 2 L004a: nop L004b: nop L004c: mov esp, ebp L004e: pop ebp L004f: ret Into the Abyss # Let\u0026rsquo;s go even further now and see what the encoded values look like in the instruction register of an x86 CPU. On the left are the line numbers and the contents of the register, and on the right is the assembly equivalent. Notice how some of the hex matches value to the right.\n0: 55 push ebp 1: 89 e5 mov ebp,esp 3: 83 ec 10 sub esp,0x10 6: 31 c0 xor eax,eax 8: 89 45 f4 mov DWORD PTR [ebp-0xc],eax b: 89 45 f0 mov DWORD PTR [ebp-0x10],eax e: 89 4d fc mov DWORD PTR [ebp-0x4],ecx 11: 89 55 f8 mov DWORD PTR [ebp-0x8],edx 14: 83 3d a8 c1 de 1a 00 cmp DWORD PTR ds:0x1adec1a8,0x0 1b: 74 05 je 22 \u0026lt;L0022\u0026gt; 1d: e8 0c fc 1d 66 call 661dfc2e \u0026lt;L004f+0x661dfbdf\u0026gt; 22: 90 nop 23: 83 7d f8 02 cmp DWORD PTR [ebp-0x8],0x2 27: 0f 9f c0 setg al 2a: 0f b6 c0 movzx eax,al 2d: 89 45 f0 mov DWORD PTR [ebp-0x10],eax 30: 83 7d f0 00 cmp DWORD PTR [ebp-0x10],0x0 34: 74 0c je 42 \u0026lt;L0042\u0026gt; 36: 90 nop 37: c7 45 f4 01 00 00 00 mov DWORD PTR [ebp-0xc],0x1 3e: 90 nop 3f: 90 nop 40: eb 09 jmp 4b \u0026lt;L004b\u0026gt; 42: 90 nop 43: c7 45 f4 02 00 00 00 mov DWORD PTR [ebp-0xc],0x2 4a: 90 nop 4b: 90 nop 4c: 89 ec mov esp,ebp 4e: 5d pop ebp 4f: c3 ret The byte layout from left to right is opcode, control bits, data. Imagine being the person that had to execute all of this manually.\nThis was considered a step up from this.\nBecause they didn\u0026rsquo;t want to do this.\n01010101 1000100111100101 100000111110110000010000 11000111000000 100010010100010111110100 100010010100010111110000 100010010100110111111100 100010010101010111111000 10000011001111011010100011000001110111100001101000000000 111010000000101 1110100000001100111111000001110101100110 10010000 10000011011111011111100000000010 11111001111111000000 11111011011011000000 100010010100010111110000 10000011011111011111000000000000 111010000001100 10010000 11000111010001011111010000000001000000000000000000000000 10010000 10010000 1110101100001001 10010000 11000111010001011111010000000010000000000000000000000000 10010000 10010000 1000100111101100 01011101 11000011 In fact the whole concept of an assembler was once considered by some as a wasteful endevour. It was using computational resources to create instructions to use computational resources. The whole point is that it used computational resources! In today\u0026rsquo;s world where we are pretty wasteful of computational resources by orders of magnitude, this was not taken for granted back then when the idea of computational time sharing was a thing. We have come a long way.\nClosing Thoughts # So now we have come full circle from\u0026hellip;\nvoid foo(int x) { int y; if (x \u0026gt; 2) { y = 1; } else { y = 2; } } \u0026hellip;to the binary seen above.\nThe compiler is a great thing, it allows us to use abstractions that make development much easier and express ideas in ways that humans are more able to grasp. Turning the realm of controlled electrons into that of the symbolic and philosophical, permitting us to process and interpret the world around us.\nReferences \u0026amp; More Reading # To start exploring on how to create a language for yourself, check out Antlr (lexer) and Bison (parser). My First Language Frontend with LLVM Tutorial - Exactly what it says. Lecture notes on Writing a Toy Compiler. ASTs - What they are and how to use them - Engaging blog post on how ASTs work and dives further into other topics also covered here. GodBolt: Compiler Explorer - Fantastic online IDE to view compiled assembly from high level languages. Let\u0026rsquo;s make a Teeny Tiny compiler, part 1 - First part of a series showing how to make a compiler using Python. Trivial but approachable and demonstrative to learning compilers. ","date":"23 October 2021","externalUrl":null,"permalink":"/compiler-basics/","section":"Posts","summary":"Introduction of compiler basics using a toy example.","title":"Compiler Basics","type":"posts"},{"content":"","date":"23 October 2021","externalUrl":null,"permalink":"/tags/compilers/","section":"Tags","summary":"","title":"Compilers","type":"tags"},{"content":"","date":"23 October 2021","externalUrl":null,"permalink":"/tags/doxygen/","section":"Tags","summary":"","title":"Doxygen","type":"tags"},{"content":"I just released a new project on GitHub called doxygen-themes which allows users to customize Doxygen styling and colors using some example templates, including a stylesheet boiled down to a few color variables.\nContributions are definitely welcome. For the lazy, here is the README.md below.\nSome sample Doxygen themes using Bootstrap 5 with different approaches to CSS use. Not all of the HTML is Bootstrap though, just the header and footer with Bootstrap like coloring applied to the body elements.\nClick here for the demo landing page..\nDocumentation # This project is a demonstration of possible approaches to theming Doxygen generated websites. Frameworks other than Bootstrap can be used, this is just an example of what is possible.\nApplied directly. Root variables applied. Simple color palette propagated. Pre-requisites # CMake - Build the project and create the website. Doxygen - Create the website documentation. Components # All components use CDN links to keep project file overhead low.\nBootstrap - Styles the header and footer. Also provides the primary colors for the Doxygen classes. Font Awesome - Header icons. jQuery - Required by Bootstrap and used for the custom palette picker. Sample CSS # Below is the CSS used for the Neon Pink theme.\n:root { --color-body: #bda9a9; --color-hyperlink: #ffffff; --color-title-background: #8a1253; --color-title-text: #ffffff; --color-footer-header-background: #c51350; --color-footer-header-text: #ffffff; --color-section-header-background: #e8751a; --color-section-header-text: #ffffff; --color-section-subheader-background: #fda403; --color-section-subheader-text: #ffffff; --color-section-text: #ffffff; --color-section-background: #bd9a9a; } Notice how there is a lot of #ffffff in there. This could probably be further simplified.\nFuture Ideas # There are a lot of CSS features that can be found. Some ways to simplify the palette even more.\nFilter Functions - Allows for different affects and interesting functions like brightness. HSL Scheme - Makes darkening and lightening easier. Arithmetic - Useful for modifying color values. ","date":"23 October 2021","externalUrl":null,"permalink":"/doxygen-themes-release/","section":"Posts","summary":"Customizable and modernized Doxygen themes.","title":"Doxygen Themes","type":"posts"},{"content":"GitHub actions don\u0026rsquo;t support running tests on created actions from the same repository. In order to do this, another repository needs to be created. There is a way to test locally using nektos/act but this doesn\u0026rsquo;t address code being committed upstream and continuously integrated. Let\u0026rsquo;s get to it.\nFor this walkthrough, there will be two endpoints:\nAction repository where your created action under test lives. Test repository that houses the tests. The general idea is to have your action repository publish an event that says a push has occurred, relay it to the test repository which triggers tests on the test repository. For this example I am using my own recently published action awalsh128/cache-apt-pkgs-action. Feel free to swap out this with your equivalent action.\nThe steps needed are to create:\na shared secret, a staging (or another) branch besides master on the action repository, a repository to run tests on your action code, tests in your test repository, a publish action in your action repository, a subscriber action in your test repository. More details on each step below.\nCreate Shared Secret # A Personal Access Token (PAT) will allow the publisher (action repository) to trigger the event that the subscriber (test repository) will then act on. This shared secret will be needed so they can interact.\nInstructions to setup your PAT. Use the arguments below.\nProfile \u0026gt; Settings \u0026gt; Developer settings \u0026gt; Personal Access Token \u0026gt; Generate New Token\nField Value Note Publish push requests to awalsh128/cache-apt-pkgs-action-ci Expiration (whatever you choose this to be) Access: repo_public Once you complete the configuration it will then show you what the value is. Note this isn\u0026rsquo;t my actual value, so don\u0026rsquo;t get any ideas. \u0026gt;:|\nWe can now share this PAT / secret with the publisher.\nPublisher # Turning our attention to the publisher side (action repository) of the event that will trigger the tests on the test repository.\nStore Shared Secret on Publisher # In order for the action repository to publish the event to the test repository, it will need access to it. The PAT will need to be stored in a variable so it can be used in the action without actually revealing it\u0026rsquo;s value.\nTo create a scecret, you can use a URL like below. Replace the cache-apt-pkgs-action-ci repository with your own test repository name.\nhttps://github.com/awalsh128/cache-apt-pkgs-action-ci/settings/secrets/actions/new\nUse the arguments below when creating the secret.\nSettings \u0026gt; Secrets \u0026gt; New repository secret\nField Value Name TRIGGER_PUBLISH_STAGING_PR_TOKEN Secret (value taken from Create Shared Secret) Here\u0026rsquo;s an example of the PAT we generated before.\nCreate a staging branch in awalsh128/cache-apt-pkgs-action # This will be used as a testing branch. Broken code can live here without going to master. It doesn\u0026rsquo;t have to be called staging, call it whatever you want (e.g. dev, whatever).\ngit checkout -b staging git push origin staging git push --set-upstream origin staging Create Publish Action # In the action repository, create .github/workflows/staging_push.yml workflow that will trigger on any push to staging.\nname: Publish Staging Push Event on: # Allow manual triggering for debugging. workflow_dispatch: # Publish when we see a push to staging. push: branches: - staging jobs: publish_event: runs-on: ubuntu-latest name: Publish staging push steps: # Note the event_type and URL secrets passed so the action repository # is allowed to post to the test repository. - run: | curl -i \\ -X POST \\ -H \u0026#34;Accept: application/vnd.github.v3+json\u0026#34; \\ -H \u0026#34;Authorization: token ${{ secrets.PUBLISH_PUSH_TOKEN }}\u0026#34; \\ https://api.github.com/repos/awalsh128/cache-apt-pkgs-action-ci/dispatches \\ -d \u0026#39;{\u0026#34;event_type\u0026#34;:\u0026#34;staging_push\u0026#34;}\u0026#39; Subscriber # Now on the subscriber (test repository), setup the tests and trigger to respond to the staging push.\nCreate Repository and Test # Create awalsh128/cache-apt-pkgs-action-ci repository (ci = continuous integration) for testing. Create a workflow .github/workflows/tests.yml that subscribes to staging_push events and runs tests. name: Staging Push Tests on: # Allow for manual dispatches so we can test the workflow if needed. workflow_dispatch: repository_dispatch: # Name of the event that will by pubsub\u0026#39;d. types: [staging_push] jobs: install: runs-on: ubuntu-latest name: Install and cache. steps: - uses: actions/checkout@v2 # Note that the action uses the @staging version. # Allows testing to happen on that branch so it can get pulled into master once it passes. - uses: awalsh128/cache-apt-pkgs-action@staging with: packages: xdot rolldice That\u0026rsquo;s it. Now anytime code is pushed to staging on your action repository, it will trigger tests on your test repository.\nMore Resources # Github: Create Repository Dispatch Event Github: Create Personal Access Token Trigger Github Actions between Repositories ","date":"12 October 2021","externalUrl":null,"permalink":"/automated-testing-for-github-actions/","section":"Posts","summary":"Create a testing harness for your created GitHub Actions.","title":"Automated Testing for GitHub Actions","type":"posts"},{"content":"","date":"12 October 2021","externalUrl":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"GitHub","type":"tags"},{"content":"","date":"12 October 2021","externalUrl":null,"permalink":"/tags/testing/","section":"Tags","summary":"","title":"Testing","type":"tags"},{"content":"","date":"10 October 2021","externalUrl":null,"permalink":"/tags/code-analysis/","section":"Tags","summary":"","title":"Code Analysis","type":"tags"},{"content":"LGTM is a nice little integration for your GitHub projects that performs code analysis and alerts if any violations are found. And example of what an alert screen looks like on the LGTM site below (using my fluentcpp project as a demonstration).\nYou will need to grant the application access to your repositories using their guide Integration with GitHub Apps.\nYou can then setup badges on your project site that show the status of your code quality.\nIf more customization is needed, an .lgtml.yml file can be created in the root of the repository per their instructions. You can also download their canonical template too to get started.\nHere\u0026rsquo;s a search of GitHub to look at other example configuration files. For example in my project I needed a more recent version of CMake. This is what mine looks like so it is in the environment before LGTM performs its analysis.\nextraction: cpp: after_prepare: - \u0026#34;mkdir custom_cmake\u0026#34; # Need later version of CMake than lgtm has. - \u0026#34;wget --quiet -O - https://cmake.org/files/v3.16/cmake-3.16.3-Linux-x86_64.tar.gz | tar --strip-components=1 -xz -C custom_cmake\u0026#34; - \u0026#34;export PATH=$(pwd)/custom_cmake/bin:${PATH}\u0026#34; index: build_command: - cd $LGTM_SRC - mkdir build; cd build - cmake .. -DCMAKE_BUILD_TYPE=RELWITHDEBINFO - make Here\u0026rsquo;s another taken from GitHub that excludes specified directories from analysis.\npath_classifiers: plugins: - plugins/ extraction: javascript: # https://lgtm.com/help/lgtm/javascript-extraction#customizing-index # The `index` step extracts information from the files in the codebase. index: # Specify a list of files and folders to exclude from extraction. exclude: - bower_components/ - docs/assets/js/plugins/ - plugins/ \u0026hellip; or from another repository more simply.\nextraction: javascript: index: filters: - exclude: \u0026#34;dist\u0026#34; Here\u0026rsquo;s yet another repository. This one disables CMake and sets the configuration options before build.\nextraction: cpp: prepare: packages: # to avoid confusion with libopenafs-dev which also provides a des.h - libssl-dev after_prepare: # make sure lgtm.com doesn\u0026#39;t use CMake (which generates and runs tests) - rm -f CMakeLists.txt - ./buildconf configure: # enable as many optional features as possible command: ./configure --enable-ares --with-libssh2 --with-gssapi --with-librtmp --with-libmetalink --with-libmetalink I think you get the point. Just give the app permission, customize as needed with the .lgtml.yml and setup your cute badges.\n","date":"10 October 2021","externalUrl":null,"permalink":"/integrating-lgtm/","section":"Posts","summary":"Integrate your GitHub projects with a code analysis tool.","title":"Code Analysis for GitHub Projects","type":"posts"},{"content":"","date":"9 October 2021","externalUrl":null,"permalink":"/tags/c++/","section":"Tags","summary":"","title":"C++","type":"tags"},{"content":"","date":"9 October 2021","externalUrl":null,"permalink":"/tags/fluent-c++/","section":"Tags","summary":"","title":"Fluent C++","type":"tags"},{"content":" Motivation # I had been wanting to make a library to express data transformation in a fluent way. The STL already has some functions for this. For example, if I wanted to transform a sequence and then filter based on a conditional I would do this.\nstd::vector\u0026lt;int\u0026gt; xs; // Add int to xs. std::transform( xs.begin(), xs.end(), [](auto x) { return ++x; }); std::vector\u0026lt;int\u0026gt; filtered; filtered.reserve(xs.size()); std::copy_if( xs.begin(), xs.end(), std::back_inserter(filtered), [](auto x) { return x % 2 == 0; }); This is indeed a functional approach to expressing data transformations. It applies functions to data sets and without side effects (as long as you don\u0026rsquo;t cause any in the lambda). Although, there can also be a lot of boilerplate if you just wants the entire range of the container. Also, it is not always obvious from first glance what the inputs and outputs are to the function because it doesn\u0026rsquo;t follow the typical f(a,b) = c mathematical format. This doesn\u0026rsquo;t make it less functional but perhaps less readable.\nThe library that I put together uses a different approach that addresses some of these concerns.\n#include \u0026lt;fluentcpp/query.h\u0026gt; std::vector\u0026lt;int\u0026gt; xs; // Add int to xs. std::vector\u0026lt;int\u0026gt; filtered = fcpp::query(std::move(xs)) .select([](auto x) { return ++x; }) .where([](auto x) { return x % 2 == 0; }); This style is patterned after the .NET LINQ framework something I had become very accustomed to before programming in C++ full time.\nInitial Release # Version 0.1 is now available on GitHub. As mentioned in the repository README.md, you can easily install the library with the instructions below.\nGo to the release you want to install. Download the fluentcpp-\u0026lt;tag\u0026gt;.tar.gz file. Decompress the file in the directory tar -xvzf fluentcpp-\u0026lt;tag\u0026gt;.tar.gz. Run the install command sudo ./install.sh. The library files are now installed on your system and can be used as in the examples. Caveats # Since this is an initial release, there are some drawbacks to using this over the STL functional library or just older C++ constructs (e.g. for-loops).\nMemory semantics are strictly by reference and move. This is intentional to accomodate the strictest object constructor and assignment declarations (i.e. no copy and move). Strict memory semantics has implications for fundamental types where copy is more efficient. Performance has not been benchmarked yet so it is unknown how it performs against the alternatives. Static analysis hints can still seem obscure. While they depend on the STL concepts enforced, the wording can be confusing across the interface. Feedback # I hope this proves useful to people and can become a wider community effort. It is primarily a proof of concept work and needs more to demonstrate it can be used in production code with similar performance to the status quo.\n","date":"9 October 2021","externalUrl":null,"permalink":"/fluentcpp-v01-release/","section":"Posts","summary":"Fluent style querying for C++.","title":"Fluent C++ v0.1 Release","type":"posts"},{"content":"","date":"9 October 2021","externalUrl":null,"permalink":"/tags/fluent-programming/","section":"Tags","summary":"","title":"Fluent Programming","type":"tags"},{"content":"","date":"7 October 2021","externalUrl":null,"permalink":"/tags/cmake/","section":"Tags","summary":"","title":"CMake","type":"tags"},{"content":" Motivation # There are already some tutorials and examples of setting up a CMake project already but I work very much ny example and then deep dive later. It\u0026rsquo;s just how my brain works I guess. So let\u0026rsquo;s skim the waters.\nAssumptions # This example uses the follow dev stack.\nCMake (obviously) C++ code. Catch2 testing framework (GitHub). The file layout has subdirectory for source code and tests, each with their own CMake file and one at the root level.\nCMakeLists.txt tests ├── CMakeLists.txt └── mylib_test.cpp src ├── CMakeLists.txt ├── mylib.h └── mylib.h Configuration # Here is the whole configuration in its respective files.\n/CMakeLists.txt\ncmake_minimum_required(VERSION 3.16.3) include(GNUInstallDirs) # Make CMAKE_INSTALL_*DIR variables available. set(CMAKE_CXX_STANDARD 20) # Use standard C++20 for compilation. project(mylib VERSION 0.1 DESCRIPTION \u0026#34;My library is a library that you can use as a library.\u0026#34;) # Make the src directory available for include lookup. include_directories(src) # Tell CMake to include the /src/CMakeLists.txt file in the build. add_subdirectory(src) # Must live in the top level file or tests won\u0026#39;t be found. include(CTest) # Automatically invokes enable_testing() # Tell CMake to include the /tests/CMakeLists.txt file in the build. add_subdirectory(tests) /src/CMakeLists.txt\nmessage(STATUS \u0026#34;Building and installing library.\u0026#34;) # Add library to build. # # STATIC rolls all code (including related includes) into a single compiled object. # SHARED rolls only mylib code into a single compiled object. # # https://stackoverflow.com/questions/2649334/difference-between-static-and-shared-libraries # # If you are using a different type of library than STATIC, make sure to look into how the target properties and # install will be different. For example https://cmake.org/cmake/help/latest/command/install.html # add_library(mylib STATIC mylib.h mylib.cpp) set_target_properties(mylib PROPERTIES VERSION ${PROJECT_VERSION} # Already set in the parent CMakeLists.txt via \u0026#39;project\u0026#39;. PUBLIC_HEADER mylib.h # Pulic headers you intend other projects to include. CXX_STANDARD_REQUIRED 20) # Enforce the C++20 standard to compile against. install(TARGETS mylib # Used for STATIC library headers. ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} COMPONENT lib # Rolled code lives here. PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/mylib COMPONENT dev) /tests/CMakeLists.txt\ninclude(FetchContent) message(STATUS \u0026#34;Building tests.\u0026#34;) # Pull in the Catch2 framework. FetchContent_Declare( Catch2 GIT_REPOSITORY https://github.com/catchorg/Catch2.git GIT_TAG v3.0.0-preview3) FetchContent_MakeAvailable(Catch2) # Most all test frameworks create a binary that then runs the tests. # This is a simple expression of that and nothing special to tests themselves. # # In this framework, Catch2 provides a main function as executable entry point. add_executable(query_tests mylib_test.cpp) # Link the included libraries in for the tests. target_link_libraries(my_tests PUBLIC Catch2 Catch2WithMain mylib) add_test(NAME tests COMMAND query_tests) More Resources # Here are some more examples to help you along.\nDev Cafe: CMake Cookbook Examples GitHub: CMakeLists.txt Search CMake Documentation add_library add_test install set_target_properties ","date":"7 October 2021","externalUrl":null,"permalink":"/cmake-simple-library/","section":"Posts","summary":"An example of a simple library using CMake.","title":"CMake Simple Library Example","type":"posts"},{"content":"I created a GitHub action awalsh128/cache-apt-packages that allows caching of Advanced Package Tool (APT) package dependencies to improve workflow execution time instead of installing the packages on every run.\nHere is an example using it to cache Doxygen dependencies.\nname: Create Documentation on: push jobs: build_and_deploy_docs: runs-on: ubuntu-latest name: Build Doxygen documentation and deploy steps: - uses: actions/checkout@v2 - uses: awalsh128/cache-apt-pkgs-action-action@v1 with: packages: dia doxygen doxygen-doc doxygen-gui doxygen-latex graphviz mscgen - name: Build run: | cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}} - name: Deploy uses: JamesIves/github-pages-deploy-action@4.1.5 with: branch: gh-pages folder: ${{github.workspace}}/build/website This action is a composition of actions/cache and the apt utility. For more information, see the repository README on GitHub.\n","date":"18 September 2021","externalUrl":null,"permalink":"/github-actions-cache-apt-pkgs-v1-released/","section":"Posts","summary":"A GitHub action to cache APT packages for future workflow runs..","title":"GitHub Actions - Cache APT Packages v1 Released","type":"posts"},{"content":"I was looking to validate and throw on invalid arguments in my code. This meant I had to write something like.\nvoid foo(size_t size) { if (size \u0026gt; this.current_size) { throw std::invalid_argument(\u0026#34;Size cannot be greater than current size.\u0026#34;); } ... } Not so bad really but I also wanted to create a dynamically created message.\nvoid foo(size_t size) if (size \u0026gt; this.current_size) { std::stringstream text; text \u0026lt;\u0026lt; \u0026#34;Size \u0026#34; \u0026lt;\u0026lt; size \u0026lt;\u0026lt; \u0026#34; cannot be greater than current size of \u0026#34; \u0026lt;\u0026lt; this.current_size \u0026lt;\u0026lt; \u0026#34;.\u0026#34;; throw std::invalid_argument(text.str()); } ... } We could shorten it\u0026hellip;\nvoid validate(bool condition) { if (condition) return; std::stringstream text; text \u0026lt;\u0026lt; \u0026#34;Size \u0026#34; \u0026lt;\u0026lt; size \u0026lt;\u0026lt; \u0026#34; cannot be greater than current size of \u0026#34; \u0026lt;\u0026lt; this.current_size \u0026lt;\u0026lt; \u0026#34;.\u0026#34;; throw std::invalid_argument(text.str()); } void foo(size_t size) validate(size \u0026gt; this.current_size); ... } \u0026hellip; but it isn\u0026rsquo;t very re-usable since the message is hardcoded.\nC++20 offers the ability to use std::format.\nvoid foo(size_t size) if (size \u0026gt; this.current_size) { throw std::invalid_argument(std::format(\u0026#34;Size {} cannot be greater than current size of {}.\u0026#34;, size, this.current_size)); } ... } Pretty nice, although for people not on that standard they are left with the previous validation approach. There is a different way to do it however.\nvoid foo(size_t size) invariant::eval(size \u0026gt; this.current_size) \u0026lt;\u0026lt; \u0026#34;Size \u0026#34; \u0026lt;\u0026lt; size \u0026lt;\u0026lt; \u0026#34; cannot be greater than current size of \u0026#34; \u0026lt;\u0026lt; this.current_size \u0026lt;\u0026lt; \u0026#34;.\u0026#34;; ... } Here is how it is implemented.\nclass invariant { private: const bool condition_; std::stringstream message_; invariant(bool condition) : condition_(condition), message_() {} invariant() = delete; invariant(const invariant \u0026amp;) = default; public: ~invariant() noexcept(false) { if (!condition_) { // Living dangerously, guaranteed not to live on the stack though. throw std::invalid_argument(message_.str()); } } template \u0026lt;typename Text\u0026gt; friend invariant \u0026amp;\u0026amp;operator\u0026lt;\u0026lt;(invariant \u0026amp;\u0026amp;item, const Text \u0026amp;text) { item.message_ \u0026lt;\u0026lt; text; return std::move(item); } static invariant eval(bool condition) { return invariant(condition, \u0026#34;\u0026#34;); } }; Not so bad maybe for pre C++20 but why have a throw in the destructor? This can be dangerous since it is called when the stack unwinds.\n~invariant() noexcept(false) { if (!condition_) { // Living dangerously, guaranteed not to live on the stack though. throw std::invalid_argument(message_.str()); } } The key is to make sure it never gets on the stack in the first place. This means making it never becomes an lvalue. Herb Sutter discusses this strategy by ensuring that the object only ever can be an rvalue reference than cannot be copied or constructed. Here are the pieces that make this possible.\ninvariant(bool condition) : condition_(condition), message_() {} invariant() = delete; invariant(const invariant \u0026amp;) = default; Lastly is getting the dynamic message input.\ntemplate \u0026lt;typename Text\u0026gt; friend invariant \u0026amp;\u0026amp;operator\u0026lt;\u0026lt;(invariant \u0026amp;\u0026amp;item, const Text \u0026amp;text) { item.message_ \u0026lt;\u0026lt; text; return std::move(item); } Note how an rvalue reference is passed in and out. This ensures that it is never copied while also modifying the object\u0026rsquo;s message state.\nThis exercise was fun and informative for myself but can be quite dangerous if not understood well and modified without such knowledge. There is also a safer alternative (well why didn\u0026rsquo;t you say so?!) using a variadic template via parameter pack. You can find the answers to string concatenation on this StackOverflow post. An adaptation from the top answer would look like.\ntemplate\u0026lt; typename ...Args\u0026gt; void invariant(bool condition, const Args\u0026amp;... args) { if (condition) return; std::stringstream message; using List= int[]; (void)List{0, ((void)(message \u0026lt;\u0026lt; args), 0 ) ...}; throw std::invalid_argument(message.str()); } ","date":"3 September 2021","externalUrl":null,"permalink":"/validate-and-throw-with-multi-type-message/","section":"Posts","summary":"Assert a condition and throw with a dynamic message in a single line in C++.","title":"C++ validate and throw with a dynamic message in a single line.","type":"posts"},{"content":"When I first started using C++, I was really confused as to how to pass things around and reason about best practices for memory. In C#, primitives were passed by value and all else by memory reference (ignoring nuance). C++ was a whole new world of aliases, pointers, smart pointers, [r/l]value references, forwarding, moves, etc.\nThis post is a simplified approach to helping C# and Java devs get an introductory understanding of C++ memory semantics so they can confidently create signatures correctly and use it as a stepping stone for more advances techniques. In the following post we will compare C++ against C# for simplicity so we don\u0026rsquo;t have to constantly bring up both C# and Java.\nThe layout of this post will follow:\nAn example of a non-idiomatic approach to C++ and show why it is inefficient. Define the storage durations so that we can understand how C++ handles objects in memory. Define the types used to communicate the memory semantics involved. Walk through examples, styles and use of the different types described. Warning We will tend to generalize ideas and lack preciseness in some areas. This is intentional to keep the reader on track. For example, I will not point out memory nuance, or undefined behavior treatments from different compilers. Also, it is assumed that you already understand what pointers, const and references are.\nThe Literal Approach (AKA Wrong Approach) # Let\u0026rsquo;s start with a common misconception that memory semantics are interchageable with all other languages. Memory semantics and how the language approaches it are important. We can\u0026rsquo;t just take our understanding from C# and literally apply it to C++.\nFor example:\nclass Object { public int x; } void Foo(Object o); var o = new Object(); Foo(o); Could literally be transformed into:\nclass Object { public: int x; }; void Foo(Object* o); void Bar() { auto* o = new Object(); Foo(o); } This is a terrible idea, don\u0026rsquo;t do this. The main mistake is confusing C# class memory semantics with C++\u0026rsquo;s.\nYou just can\u0026rsquo;t change the C# reference to a C++ pointer and be done. This go against the C++ philosophy to prefer the stack. Especially since this is an option not available in C# due to the division between value and reference types Object is very small (about 8 bytes) and is best allocated on the stack since it is faster (see gotw/009). The scope of the Object instance isn\u0026rsquo;t well defined but if we assume it is short lived there is no need for dynamic allocation. Here is a better way to do it:\nObject o; FooReadOnly(o); // Simple and fast copy. FooWrite(\u0026amp;o); // Simple and fast copy of stack address pointer. Storage Duration # As noted, there is a strong preference to allocate on the stack since it is much faster for small objects. In C# it is generally assumed that fundamental types are allocated on the stack and reference types on the heap. This is optional in the C++ language as the developer can decide to allocate classes on the stack or on the heap. Therefore it is helpful for the new developer to understand how storage duration / lifetime works in C++.\nGoing forward we will refer to allocation in terms of the storage duration. Concepts like stack and heap are implementation concerns. While useful when first learning these durations it is better to speak in terms of the traits of these durations than the lower level understanding. For example stack storage is fast but we might as well just say automatic storage is fast instead.\nAutomatic # The object lifetime is tied to the code block it is allocated in and is deallocated at the end of the block. We should aim to use this storage duration whenever possible. As noted previously, it is faster (see gotw/009)\nvoid Foo() { int x = 1; // Object is allocated. // ... } // Object is deallocated. Dynamic # The object lifetime is tied to the declaration of an instance and the explicit deallocation of the instance (see examples).\nWarning Missed deallocation can results in Out of Memory (OOM) errors. This is one of the most dangerous durations since the [de]allocation is controlled by the dev.\nvoid Foo(Object* o) { // ... } void Bar() { auto* o = new Object(); Foo(o.get()); delete o; } Thread # The object lifetime is tied to the thread allocation and deallocation. In the example below we will not consider race conditions.\n#include \u0026lt;threads\u0026gt; // Object is statically allocated. See Static section below for more information. thread_local int x = 1; void Foo(int y) { // x is allocated on the thread with original declared value for every new thread and // incremented. x += y; } void Bar() { std::thread t1(Foo, 2); // Thread instance of x becomes 3 inside Foo. std::thread t2(Foo, 3); // Thread instance of x becomes 4 inside Foo. t1.join(); // Thread instance of x is deallocated once thread joins and deallocates. t2.join(); } Static # The object lifetime is tied to that of the program. Note the thread_local declaration in the section above has static storage duration. Try to be sparing with the size and use of these objects since they have the largest lifetime.\nstatic int x = 1; // x is allocated at program start and deallocated at program end. void Foo() { ++x; } Foo(); // x is incremented; x = 2. Foo(); // x is incremented; x = 3. An example of static local variables.\nvoid Foo() { static int y = 1; ++y; } Foo(); // y is initialized; y = 1. From now on, the declaration is skipped. Foo(); // y declaration is skipped and incremented; y = 2; Foo(); // y declaration is skipped and incremented; y = 3; Types # C# has a single type taxonomy where everything is derived / inherited from Object (example Int64). This taxonomy allows a common interface of methods available to all child objects like Equals and ToString. This is not the case for C++. It uses duck typing to resolve an operations validity at compile time. If the operands have the needed operation (e.g. equals) then it is considered valid.\nFundamental Types # C++ fundamental types are the same types as found in C# value types.\nObjects # Objects also encompass fundamental types as well as classes and structs. Without any additional modifiers, objects are automatically allocated.\n// Make a automatic copy and use inside Foo. void Foo(int x); // In terms of memory it is the same as Foo(int x) since x will be passed as a copy. // The only reason to keep as const would be to avoid modification inside the code block. // IMO, this is indicative of a leaky interface since it is implementation specific. void Foo(const int x) { x = 2; // ERROR: Cannot change a constant value. int y = x; // Make a copy of x and assign to y; x = 1, y = 1. const int z = x; // Make a copy of x and assign to z upon its initialization. } References \u0026amp; Pointers # Warning Don\u0026rsquo;t confuse C# reference types with C++ references. In computer science it is defined as a value that indirectly accesses a particular datum. C++ narrows this definition to a specific datatype implementation using the \u0026amp; operator, whereas the general definition could refer to pointers as well.\nvoid Foo() { int x = 1; int y = x; // Make a copy of x and assign to y; x = 1, y = 1 int* z = \u0026amp;x; *z = 2; // Reference x from z; x = 2 const int a = 1; int* b = \u0026amp;a; // ERROR: Cannot get a reference from a const value. } void Foo(int\u0026amp; x) { x = 2; } int x = 1; Foo(x); // x = 2 // Equivalent function but using a pointer. void Quux(int* x) { *x = 2; } int x = 1; Quux(\u0026amp;x); // x is now 2; Sharing a read only reference.\nclass LargeObject { public: explicit LargeObject(/*...*/) : /*...*/{}; int x; // Lots of members of large size. }; void Foo(const LargeObject\u0026amp; o) { // Read only and use data from o. } void Bar() { LargeObject o(...); Foo(o); } Note that LargeObject is specified for the sake of needing dynamic allocation. This can be due to: having a container with an unspecified size, lifetime is managed in threads, or is in a larger scope than the immediate function and dependent functions.\nSmart Pointers # Resource Acquisition Is Initialization (RAII) is a language idiom that essentially says that object creation is undone by destruction. In C++ this means that any object construction on the stack is also destructed when the object falls out of scope. Smart pointers are simply a wrapper around a pointer. It is allocated when the smart pointer is allocated and deallocated when the smart pointer is deallocated (see Resource Acquisition Is Initialization (RAII). This gets rid of the need to explicitely call a delete and ties the resource lifetime to that of the object wrapping it.\nSmart pointers have most of the common operations you would expect like *x and x-\u0026gt;y.\nA crude way to represent this, ignoring operations, would be:\ntemplate \u0026lt;class T\u0026gt; struct simple_smart_pointer { public: simple_smart_pointer() = delete; // Don\u0026#39;t allow copies or multiple assignments. // There should only ever be a single instance. simple_smart_pointer(const simple_smart_pointer\u0026amp;) = delete; simple_smart_pointer\u0026amp; operator=(const simple_smart_pointer\u0026amp;) = delete; template \u0026lt;class... TArgs\u0026gt; explicit simple_smart_pointer(...args) // Pass variadic args. { value_ = new T(args...); } ~simple_smart_pointer() { delete value_; } private: T* value_; }; Unique Pointer # An example of using a type of non-shared smart pointer called std::unique_ptr.\n#include \u0026lt;memory\u0026gt; class LargeObject { public: explicit LargeObject(/*...*/) : /* initialize members */ {}; int x; // Lots of members of large size. }; void Foo(std::unique_ptr\u0026lt;LargeObect\u0026gt; o) { // Take ownership and process o. } void Bar() { // Automatically allocate std::unique_ptr and dynamically allocate the wrapped value. auto o = std::make_unique\u0026lt;LargeObject\u0026gt;(/*...*/); // ... Foo(std::move(o)); } The std::move function is used to transfer the resource from Bar scope to Foo. This means that ownership can be passed to other scopes and objects. Although once moved, it can no longer be used in that same scope.\n#include \u0026lt;memory\u0026gt; void Baz() { std::unique_ptr\u0026lt;LargeObject\u0026gt; o(/*...*/); Foo(std::move(o)); int y = o-\u0026gt;x; // ERROR: Undefined behavior. } #include \u0026lt;memory\u0026gt; void Foo() { std::unique_ptr\u0026lt;LargeObject\u0026gt; o(/*...*/); LongLivedProcessing(std::move(o)); } So that in context of Foo it now owns the std::unique_ptr and its underlying destructions of the dynamically allocated LargeObject. Once it falls out of scope, the object is dynamically deallocated. This data type is important when applying to the concept of std::unique_ptr because it helps transfer ownership down stack.\nvoid Foo(std::unique_ptr\u0026lt;LargeObject\u0026gt; o) { // Acts on o-\u0026gt;x; } // o falls out of scope and it is dynamically destructed. Shared Pointer # Think of shared pointer as the simple_smart_pointer but with the copy operation allowed and holding an internal reference count.\nA crude way to represent this, ignoring operations, would be:\ntemplate \u0026lt;class T\u0026gt; struct simple_shared_pointer { public: simple_shared_pointer() = delete; simple_shared_pointer(const simple_smart_pointer\u0026amp; p) { // Increase reference count now that we have another automatically allocated instance. *counter_ = ++(*p.counter_); value_ = p.value_; } simple_shared_pointer\u0026amp; operator=(const simple_smart_pointer\u0026amp; p) { // Increase reference count now that we have another automatically allocated instance. *counter_ = ++(*p.counter_); value_ = p.value_; } template \u0026lt;class... TArgs\u0026gt; explicit simple_shared_pointer(...args) // Pass variadic args. { value_ = new T(args...); } ~simple_shared_pointer() { // Decrease reference count now that instance is automatically deallocated. int new_counter = --(*counter_); // If this is the last instance, dynamically deallocate the value. if (new_counter == 0) delete value_; } private: int* counter_; // Reference counter. T* value_; // Shared dynamically allocated value. }; An example of using a type of shared smart pointer called std::shared_ptr.\n// Construction and copy operations. void Bar(std::unique_ptr\u0026lt;LargeObject\u0026gt; o) { std::shared_ptr\u0026lt;LargeObject\u0026gt; ctor_shared = std::make_unique\u0026lt;LargeObject\u0026gt;(/*...*/); std::shared_ptr\u0026lt;LargeObject\u0026gt; moved_shared = std::move(o); // moved_shared takes ownership std::shared_ptr\u0026lt;LargeObject\u0026gt; copied_shared = moved_shared; // shared pointer reference count is 2 } void Bar(std::shared_ptr\u0026lt;LargeObject\u0026gt; o) { // Do stuff with o all in scope. No passing to objects that outlive Bar. } // Decrement reference count as std::shared_ptr is destructed. void Baz() { auto o = std::make_shared\u0026lt;LargeObject\u0026gt;(/*...*/); // Reference count starts at 1. Bar(o); // Increment reference count to 2 as std::shared_ptr is copied. Bar(o); // Increment reference count to 3 as std::shared_ptr is copied. } // o falls out of scope Weak Pointer # There is a similar concept in C# (i.e. WeakReference) that corresponds to (std::weak_ptr in C++. It is constructed from a std::shared_ptr and [de]allocated dynamically using reference count. If the std::shared_ptr is deallocated then the std::weak_ptr will return null.\nvoid Foo() { std::shared_ptr\u0026lt;int\u0026gt; shared = std::make_shared\u0026lt;int\u0026gt;(1); std::weak_ptr\u0026lt;int\u0026gt; weak = shared; // Creates a std::shared_ptr pointing to std::weak_ptr with reference count 1 std::shared_ptr\u0026lt;int\u0026gt; shared1 = weak.lock(); std::shared_ptr\u0026lt;int\u0026gt; shared2 = weak.lock(); // std::shared_ptr reference count 2 std::shared_ptr\u0026lt;int\u0026gt; shared3 = weak.lock(); // std::shared_ptr reference count 3 shared3.reset(); // std::shared_ptr reference count 2 shared2.reset(); // std::shared_ptr reference count 1 weak.expired(); // Check that there are still std::shared_ptr\u0026#39;s in memory. shared.reset(); // Only allowed if is the last std:shared_ptr // weak is expired and shared is null } Examples # Below are some different ways to think about how to provide your signatures and what they convey.\nWarning Note in each section how pointer can be ambiguous and possibly result in multiple ownership. Most of the time you never really want two objects holding on to the same pointer unless they are well coordinated. If one deletes the pointer you will get a segmentation fault upon further access.\nInputs # void Foo(int x); // Preferred for fundamental types as noted earlier. void Foo(const LargeObject\u0026amp; o); // Read-only reference to o. void Foo(LargeObject\u0026amp; o); // Read and write reference to o. void Foo(std::unique_ptr\u0026lt;LargeObject\u0026gt; o); // Transfer ownership to Foo block. // Copy pointer and allow read and write operations. // Could be owned by caller or being passed to caller. void Foo(LargeObject* o); Outputs # Outputs as Return Value # class Foo; int Bar(); // Return a copy of the function scoped return. const int Bar(); // Return a read-only copy of the function scoped return. // Return a reference to LargeObject held by Foo. // Foo must outlive caller. LargeObject\u0026amp; Foo::Bar(); // Return a read-only reference to LargeObject held by Foo. // Foo must outlive caller. const LargeObject\u0026amp; Foo::Bar(); std::unique_ptr\u0026lt;LargeObject\u0026gt; Bar(); // Transfer ownership to calling block. // Return a copy of a pointer. // Could be owned by Foo or being passed to caller. LargeObject* Foo::Bar(); Outputs as Arguments # void Foo(int\u0026amp; x); // Read and write reference to x. void Foo(int* x); // Read and write of x from pointer. Note There is nothing inherently bad with having outputs as arguments but whenever possible it is better to communicate it on the return. See the Readability section below for more information and cases.\nReadability # In functional programming there is a concept of referential transparency or pure functions, which forbids side effects. This makes programs easier to reason about because you don\u0026rsquo;t need to look at the implementation to see what they do. For example consider the signature:\nvoid DoStuff(int x, int* y); We can see that x is an input but y may or may not be an input or an output or both. As far as we know y could point to an unpopulated object that needs to be filled, or it could just be an input that is a pointer, or it could be an input transferring control of the pointer to something else (e.g. another thread or static member). Let\u0026rsquo;s narrow this and try to eliminate the ambiguity assuming that the input is a read and write value.\nvoid DoStuff(int x, int\u0026amp; y); Okay now we narrowed down the definition to allow for a variable that can be read from and written to. If we wanted to make it just a read only reference, we could const qualify it. Although lets assume it is read and write.\nvoid DoStuff(int x, int y); While it accomplishes the same thing as the signature before, it is now very clear that the value is both read and written / computed. The signature now communicates this very cleanly and in a transparent way. The same idea can be applied to multiple outputs too using std::tuple.\nstd::tuple\u0026lt;int, int\u0026gt; DoStuff(int x, int y); Although what do we do about LargeObject if we wanted to make a minor mutation, then return a copy of LargeObject along with the mutation?\nLargeObject DoStuff(const LargeObject\u0026amp; o); While this works from a functional point of view, it has horrible performance implications. How about:\nLargeObject\u0026amp; DoStuff(const LargeObject\u0026amp; o) { return o; } // Illegal This doesn\u0026rsquo;t work because you can\u0026rsquo;t make a const qualified input and be able to return a read/write reference to it. Although this will work:\nLargeObject\u0026amp; DoStuff(LargeObject\u0026amp; o); Isn\u0026rsquo;t this the same thing as just returning a void? Yes, and maybe it is best just to do so since \u0026amp; without const communicates that it is possibly read and definitely written. In these cases, it is hard to express the semantics cleanly due to performance.\nDependency Injection (DI) Container # DI containers can hold all the components necessary to maintain a service. If created at the code entry point it is best to share by reference in a top don way.\nclass Component1 { public: explicit Component1(bool production) : production_(production) {} // ... private: // ... bool production_; }; class Component2 { public: explicit Component2(bool production) : production_(production) {} // ... private: // ... bool production_; }; class Container { public: // Consume only unique pointers since no other object should hold the dependencies explicit Container( std::unique_ptr\u0026lt;Component1\u0026gt; component1, std::unique_ptr\u0026lt;Component2\u0026gt; component2) // Release ownership to the new Container. : component1_(std::move(component1)), component2_(std::move(component2)) {} static Container CreateNonProd() { return Container( std::make_unique\u0026lt;Component1\u0026gt;(/*production=*/false), std::make_unique\u0026lt;Component2\u0026gt;(/*production=*/false)); } static Container CreateProd() { return Container( std::make_unique\u0026lt;Component1\u0026gt;(/*production=*/true), std::make_unique\u0026lt;Component2\u0026gt;(/*production=*/true)); } // Never expose the pointer, just the value pointed to. Component1\u0026amp; component1() { return *component1_; } Component2\u0026amp; component2() { return *component2_; } private: std::unique_ptr\u0026lt;Component1\u0026gt; component1_; std::unique_ptr\u0026lt;Component2\u0026gt; component2_; }; void DoStuff(Component1\u0026amp; component) { /*...*/ } void main() { auto prod_container = Container::CreateProd(); DoStuff(prod_container.component1()); } If used in a separate thread or allocation is very large, prefer std::shared_ptr instead. Below is a modification of the previous class layout.\nclass Container { public: // Consume only unique pointers since no other object should hold the dependencies explicit Container( std::unique_ptr\u0026lt;Component1\u0026gt; component1, std::unique_ptr\u0026lt;Component2\u0026gt; component2) // Release ownership to the shared pointers. : component1_(std::move(component1)), component2_(std::move(component2)) {} // ... // Never expose the pointer, just the value pointed to. std::shared_ptr\u0026lt;Component1\u0026gt; component1() { return component1_; } std::shared_ptr\u0026lt;Component2\u0026gt; component2() { return component2_; } private: std::shared_ptr\u0026lt;Component1\u0026gt; component1_; std::shared_ptr\u0026lt;Component2\u0026gt; component2_; }; Rules of Thumb # Copies can be a faster operation for small objects. Only use smart pointers when the object is very large or needs to be shared outside the scope it was created in. Avoid raw pointers whenever possible. Prefer references over raw or smart pointers where available. Conclusion # Hopefully this gives you a sense now how storage works and how to communicate memory semantics in the best and most performant way possible. Please feel free to leave comments on errata or your own thoughts.\n","date":"21 September 2020","externalUrl":null,"permalink":"/cpp-memory-semantic-for-csharp-java-devs/","section":"Posts","summary":"Understanding how memory semantics works in C++ from the perspective of a C# or Java dev.","title":"C++ Memory Semantics for C# and Java Devs","type":"posts"},{"content":"","date":"24 April 2015","externalUrl":null,"permalink":"/tags/encoding/","section":"Tags","summary":"","title":"Encoding","type":"tags"},{"content":" Flashback # The last time I used Unix-to-Unix format (AKA UUEncoding) was when USENET was still the big thing and Mosaic web browser was just coming out. That was until recently, when I had a requirement to encode and decode this file type.\nSearching for an Implementation # Since Base64 has largely replaced this older format, it was hard to find a current implementation for the .NET platform. I did run across a port of KDE's kcodecs. Although the port wasn't a streaming solution in the context of implementing the Stream class. Also, it allocated a lot of one item byte arrays using the ReadByte call for each character.\nCreating an Implementation # Originally I tried to create my own solution by implementing the .NET Encoder class but the interface didn't fit the requirements of UUEncoding. For example, the GetBytes call works on a per character basis whereas UUEncoding takes 3 characters at a time. Also, a header and footer needs to be written, and the encoded payload is segmented into lines prefixed by encoded line lengths.\nI ended up creating my own encoder class that was scoped to only handle data line by line.\npublic static class UUEncoder { // Assumes the current position is at the start of a new line. public static byte[] DecodeLine(Stream buffer) { // ... } public static byte[] EncodeLine(byte[] buffer) { // ... } } I then created encode and decode Stream classes that depended on the encoder. Having the encoding and decoding happening in a Stream based way was critical for my requirements since I was lazily evaluating the data and wouldn't just read it all up front. This was important since some of the files tended to be Gigabytes in size and an in-memory solution would have created an unacceptable memory footprint. Along with the nastiness that potentially comes with it like thrashing.\nUsing the Code # You can find my implementation, with tests, on Github here.\nTo decode any stream:\nusing (Stream encodedStream = /* Any readable stream. */) using (Stream decodedStream = /* Any writeable stream. */) using (var decodeStream = new UUDecodeStream(encodedStream)) { decodeStream.CopyTo(decodedStream); // Decoded contents are now in decodedStream. } To encode any stream:\nbool unixLineEnding = // True if encoding with Unix line endings, otherwise false. using (Stream encodedStream = /* Any readable stream. */) using (Stream decodedStream = /* Any writeable stream. */) using (var encodeStream = new UUEncodeStream(encodedStream, unixLineEnding)) { decodedStream.CopyTo(encodeStream); // Encoded contents are now in encodedStream. } Note on Licensing # I published the code under version 2 (not 2.1) of the LGPL since I took the bit twiddling and encoder maps from KDE's implementation.\nMore Resources \u0026amp; Reading # Github: Awalsh128.Text Library (contains UUEncoder implementation) GNU Lesser General Public License, version 2.1 ","date":"24 April 2015","externalUrl":null,"permalink":"/streaming-uuencoder-in-net/","section":"Posts","summary":"Flashback # The last time I used Unix-to-Unix format (AKA UUEncoding) was when USENET was still the big thing and Mosaic web browser was just coming out. That was until recently, when I had a requirement to encode and decode this file type.\n","title":"Streaming UUEncoder in .NET.","type":"posts"},{"content":"I was recently putting some PowerShell scripts together for deployment and maintenance of software to our machine instances. One of the requirements was to be able to discover the service name from a Windows Service executable that uses ServiceInstaller. I needed to be able to extract this value in a generic way in order to query the service and stop it if running. Here is what I was able to put together.\nFunction Get-WindowsServiceName([string]$exePath) { $assembly = [System.Reflection.Assembly]::LoadFrom($exePath) $type = $assembly.GetTypes() | Where { $_.GetCustomAttributes([System.ComponentModel.RunInstallerAttribute], 0).Length -gt 0 } | Select -First 1 $installer = [System.Configuration.Install.Installer][Activator]::CreateInstance($type) $serviceInstaller = [System.ServiceProcess.ServiceInstaller]$installer.Installers[0] $serviceInstaller.ServiceName } Note that this doesn't support multiple installers in a single executable.\n","date":"20 April 2015","externalUrl":null,"permalink":"/get-windows-service-name-from/","section":"Posts","summary":"I was recently putting some PowerShell scripts together for deployment and maintenance of software to our machine instances. One of the requirements was to be able to discover the service name from a Windows Service executable that uses ServiceInstaller. I needed to be able to extract this value in a generic way in order to query the service and stop it if running. Here is what I was able to put together.\n","title":"Get Windows Service name from executable in PowerShell.","type":"posts"},{"content":"","date":"20 April 2015","externalUrl":null,"permalink":"/tags/powershell/","section":"Tags","summary":"","title":"PowerShell","type":"tags"},{"content":" Rookie Moves # Awhile ago, I had originally created a single logger for each service and shared it statically across the application.\npublic static class Log { private readonly static Lazy\u0026lt;Log\u0026gt; instance = new Lazy\u0026lt;Log\u0026gt;(() =\u0026gt; new Log(), true); public static Log Instance { get { return instance.Value; } } } It always had felt strange doing this since it violated encapsulation. I was referencing a static instance inside my objects without ensuring the existence of the log instance itself, while also making assumptions about the instance's state. Essentially reaching outside the class every time to the singleton, which is a global state instance in my case.\nBest Practices # As I researched more on best practices with logging, logger per class appeared to be the best pattern since it offered the most fine grain control with respect to filtering and configuration.\nWhen using logging in your class, you should separate the concern of how the log gets created from the use of the log. This can be achieved by having a factory accessor.\npublic class Log { // Set your factory property to the actual log implementation you wish to use. public static Func\u0026lt;Type, Log\u0026gt; Factory { get; set; } // Instance based properties and methods. } You can also use the abstract factory pattern if you have to wrap your logging implementations.\npublic abstract class LogBase { // Set your factory property to the actual log implementation you wish to use. public static Func\u0026lt;Type, LogBase\u0026gt; Factory { get; set; } // Abstract properties and methods. } Then just call the factory by passing in the class type it is for.\npublic class SomeObject { private static readonly LogBase log = Log.Factory(typeof(SomeObject)); } The Difference # It may not seem like much of a change but subtle differences are happening:\nCalling class can now communicate it's state to the log's creation. Log creation is no longer the responsibility, implicitly or otherwise, of the class. In my case, this was very liberating since we had several log implementations in our large codebase. I no longer needed to worry about which log implementation was being used by messing with the singleton construction, and could leverage filtering when I need to isolate a single component or service that was causing trouble. Something that I couldn't do before.\nMore Reading \u0026amp; Resources # C\u0026amp;C Inc: Logging Best Practices beefycode: Log4Net Recommended Practices pt 1: Your Code JayWay: A nice, basic log4net setup Stack Overflow: Why do loggers recommend using a logger per class? Stack Overflow: log4net: Configure to ignore messages from a specific class Rolf Engelhard: Logging Anti-Patterns, Part I ","date":"29 March 2015","externalUrl":null,"permalink":"/log-per-class-pattern/","section":"Posts","summary":"Rookie Moves # Awhile ago, I had originally created a single logger for each service and shared it statically across the application.\n","title":"Log per class pattern.","type":"posts"},{"content":" Motivation # I decided to use log4net for a recent project I had been working on. There is a concept of loggers and appenders. Loggers are composed of appenders and a log threshold. Appenders are consumers of logging information and provide specific implementations (eg. file, email, event log, database). You can configure the loggers and appenders either in the application configuration or at runtime.\nConfiguring logging in the application configuration provides the most flexibility. It is great being able to change settings on the fly, especially when it is running in a production environment and redeploying the build is out of the question. Although this approach comes at the expense of having a lot of information in your application configuration for the loggers and appenders. No big deal though if you just have to configure once.\nWhy Templates? # I had quite a bit of projects with a lot of redundant logging configuration information in each one's application configuration file. Much of the information had a standard form that we wanted uniform across our different projects too (eg. log file name conventions, event log setup, email format). Also, if we updated the logging appender configuration for a new standard, we would need to do it to every project's application configuration file. This is where templates came into play.\nWriting the Code # To cut down the amount of configuration information needed to start a new project with logging and make the configuration more uniform where needed, we offloaded it into the code and left the rest in the application configuration file like so.\n\u0026lt;log4net xsi:noNamespaceSchemaLocation=\u0026#34;log4net.xsd\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34;\u0026gt; \u0026lt;logger name=\u0026#34;LoggerTemplate\u0026#34;\u0026gt; \u0026lt;appender name=\u0026#34;SmtpAppenderTemplate\u0026#34; type=\u0026#34;log4net.Appender.SmtpAppender\u0026#34;\u0026gt; \u0026lt;to value=\u0026#34;peter@initech.com\u0026#34; /\u0026gt; \u0026lt;from value=\u0026#34;system@initech.com\u0026#34; /\u0026gt; \u0026lt;smtpHost value=\u0026#34;mail.initech.com\u0026#34; /\u0026gt; \u0026lt;username value=\u0026#34;peter\u0026#34; /\u0026gt; \u0026lt;password value=\u0026#34;abc123\u0026#34; /\u0026gt; \u0026lt;/appender\u0026gt; \u0026lt;/logger\u0026gt; \u0026lt;root\u0026gt; \u0026lt;level value=\u0026#34;INFO\u0026#34; /\u0026gt; \u0026lt;/root\u0026gt; \u0026lt;/log4net\u0026gt; We don't use the logger directly but rather as a template for our root logger. Now we just need to craft a method to consume the template and create the root appenders at runtime.\n/// \u0026lt;summary\u0026gt; /// Get appenders matching the logger template name and use them to populate the root appenders at runtime. /// \u0026lt;/summary\u0026gt; /// \u0026lt;param name=\u0026#34;loggerTemplateName\u0026#34;\u0026gt;The logger template name found in the application configuration.\u0026lt;/param\u0026gt; public static void ConfigureRootFromTemplate(string loggerTemplateName) { ILog logTemplate = LogManager.GetLogger(loggerTemplateName); if (logTemplate == null) { throw new ArgumentException( String.Format( \u0026#34;Logger template {0} not found in log4net configuration. Make sure there is an \u0026#34; + \u0026#34;logger in the log4net configuration with the name {0}.\u0026#34;, loggerTemplateName), \u0026#34;loggerTemplateName\u0026#34;); } IAppender[] appenderTemplates = logTemplate.Logger.Repository.GetAppenders(); var smtpAppenderTemplate = appenderTemplates.FirstOrDefault(a =\u0026gt; a is SmtpAppender) as SmtpAppender; if (smtpAppenderTemplate == null) { throw new ArgumentException( String.Format( \u0026#34;SmtpAppender template not found in log4net configuration. Make sure there is an \u0026#34; + \u0026#34;SmtpAppender in the log4net {0} logger.\u0026#34;, loggerTemplateName), \u0026#34;loggerTemplateName\u0026#34;); } // Can repeat the above pattern with other appenders as well. // Create appenders using the template information from above. AddAppendersToRootAndConfigure( new AppenderCollection { // Put your created appenders here. }); } private static void AddAppendersToRootAndConfigure(AppenderCollection appenders) { // Get the log repository. var hierarchy = (Hierarchy)log4net.LogManager.GetRepository(); // Get the root logger. Logger rootLogger = hierarchy.Root; foreach (var appender in appenders) { // Add all the appenders and activate. rootLogger.AddAppender(appender); ((AppenderSkeleton)appender).ActivateOptions(); } // Flag root logger as configured with new appender information. rootLogger.Hierarchy.Configured = true; } Then just call the configuration method at the application's startup.\nclass Program { /// /// The main entry point for the application. /// static void Main() { // Other startup configuration code. log4net.Config.XmlConfigurator.Configure(); // Load the application configuration information. Log.ConfigureRootFromTemplate(\u0026#34;LoggerTemplate\u0026#34;); // More startup configuration code. } Considerations # I don't recommend using this approach in all cases. It definitely cuts down the amount of application configuration needed but at the cost of information hiding, since it has been moved to code. Also, it may not be obvious to an uninitiated developer what your application configuration is doing, especially since this template approach is not encoded into the structure of log4net's XML. Although, if you have many projects and need to effect changes to logging across them, this may be a good solution for you.\nMore Reading \u0026amp; Resources # C\u0026amp;C Inc: Logging Best Practices beefycode: Log4Net Recommended Practices pt 1: Your Code JayWay: A nice, basic log4net setup Stack Overflow: Why do loggers recommend using a logger per class? Stack Overflow: log4net: Configure to ignore messages from a specific class Stack Overflow: Best way to dynamically set an appender file path ","date":"28 March 2015","externalUrl":null,"permalink":"/creating-xml-based-templates-in-log4net/","section":"Posts","summary":"Motivation # I decided to use log4net for a recent project I had been working on. There is a concept of loggers and appenders. Loggers are composed of appenders and a log threshold. Appenders are consumers of logging information and provide specific implementations (eg. file, email, event log, database). You can configure the loggers and appenders either in the application configuration or at runtime.\n","title":"Creating XML based templates in log4net.","type":"posts"},{"content":" Motivation # I had these flyweights that added a lot of overhead to the serialization process. They weren't really needed in the serialized payload either. In fact, I could recreate the flyweight in memory from just a single property on the object.\npublic class FlyweightObject { public string Key { get; private set; } public string AProperty { get; private set; } public string AnotherProperty { get; private set; } public string YetAnotherProperty { get; private set; } // ... Lots of properties. // Overridden GetHashCode and Equals methods to make equality by Key. } public class FlyweightObjectFactory { public static FlyweightObjectFactory Instance { get; private set; } // Singleton initialization code. public FlyweightObject GetObject(string key) { // Get from dictionary, or create object and add to dictionary. // Return object from dictionary. } } Nothing out of the ordinary here. Although, these flyweights were also used as keys in a dictionary I was serializing. This was a problem too because only scalars can be used when serializing dictionaries with JSON.NET. It does mention that I can use a type converter though.\nSerializing # Let focus on serializing this object to a scalar first. The JSON.NET serialization guide mentions that I can override the Object.ToString method. So let's do that.\npublic class FlyweightObject { public string Key { get; private set; } public string AProperty { get; private set; } public string AnotherProperty { get; private set; } public string YetAnotherProperty { get; private set; } // ... Lots of properties. // Overridden GetHashCode and Equals methods to make equality by Key. public override string ToString() { return this.Key; } } I'm done right? Who needs type converters? Well this doesn't help us when I need to deserialize. This is where type converters come into play.\nImplementing a Type Converter # First we have to provide the concrete implementation of our type converter class for the flyweight.\ninternal class FlyweightObjectConverter : TypeConverter { /// \u0026lt;summary\u0026gt; /// Returns whether this converter can convert an object of the given type to the type of this converter, using the specified context. /// \u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt; /// true if this converter can perform the conversion; otherwise, false. /// \u0026lt;/returns\u0026gt; /// \u0026lt;param name=\u0026#34;context\u0026#34;\u0026gt;An that provides a format context. \u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;sourceType\u0026#34;\u0026gt;A \u0026lt;see cref=\u0026#34;T:System.Type\u0026#34;/\u0026gt; that represents the type you want to convert from. \u0026lt;/param\u0026gt; public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType) { return sourceType == typeof(string) || base.CanConvertFrom(context, sourceType); } /// \u0026lt;summary\u0026gt; /// Converts the given object to the type of this converter, using the specified context and culture information. /// \u0026lt;/summary\u0026gt; /// \u0026lt;returns\u0026gt; /// An \u0026lt;see cref=\u0026#34;T:System.Object\u0026#34;/\u0026gt; that represents the converted value. /// \u0026lt;/returns\u0026gt; /// \u0026lt;param name=\u0026#34;context\u0026#34;\u0026gt;An \u0026lt;see cref=\u0026#34;T:System.ComponentModel.ITypeDescriptorContext\u0026#34;/\u0026gt; that provides a format context. \u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;culture\u0026#34;\u0026gt;The \u0026lt;see cref=\u0026#34;T:System.Globalization.CultureInfo\u0026#34;/\u0026gt; to use as the current culture. \u0026lt;/param\u0026gt; /// \u0026lt;param name=\u0026#34;value\u0026#34;\u0026gt;The \u0026lt;see cref=\u0026#34;T:System.Object\u0026#34;/\u0026gt; to convert. \u0026lt;/param\u0026gt;\u0026lt;exception cref=\u0026#34;T:System.NotSupportedException\u0026#34;\u0026gt;The conversion cannot be performed. \u0026lt;/exception\u0026gt; public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value) { return FlyweightObjectFactory.Instance.GetObject((string)value); } } Then attribute the convertible class with the implementation.\n[TypeConverter(typeof(FlyweightObjectConverter))] public class FlyweightObject { public string Key { get; private set; } public string AProperty { get; private set; } public string AnotherProperty { get; private set; } public string YetAnotherProperty { get; private set; } // ... Lots of properties. // Overridden GetHashCode and Equals methods to make equality by Key. } Good to go. Now I can deserialize the flyweights. The JSON.NET secret sauce can be found in the static method Newtonsoft.Json.Utilities.ConvertUtils.TryConvertInternal. This method is used by the internal reader class for its own deserialization method and consequently for the final Newtonsoft.Json.JsonSerializer class, which is the lynchpin for all the serialization helper methods in the JSON.NET ecosystem.\nConsiderations # Instead of overriding the Object.ToString method, I could implement the TypeConverter.CanConvertTo and TypeConverter.ConvertTo methods. It is really up to you. In my case, I already had the Object.ToString method overridden since the key property uniquely identifies the object without having to create an implicitly structured string (ie. comma delimited properties as a single string; \u0026quot;Prop1|Prop2|Prop3\u0026quot;).\nIf you only need a type conversion for JSON, you can also use JSON.NET's own custom type converters. Although this doesn't work in the case of dictionary keys.\nLastly, if you have a complex type as your dictionary key, you may want to consider just serializing it as a collection and then deserializing it using a JSON.NET custom converter.\nMore Reading \u0026amp; Resources # MSDN: Type Conversion in the .NET Framework MSDN: How to: Implement a Type Converter JSON.NET: Add support for IReadOnlyList and IReadOnlyDictionary - Provides JsonConverter implementations for read only collections. ","date":"28 March 2015","externalUrl":null,"permalink":"/using-type-converters-for-jsonnet/","section":"Posts","summary":"Motivation # I had these flyweights that added a lot of overhead to the serialization process. They weren't really needed in the serialized payload either. In fact, I could recreate the flyweight in memory from just a single property on the object.\n","title":"Using type converters for JSON.NET.","type":"posts"},{"content":" Pitfalls in Production # Being able to monitor production services for potential errors is critical. Especially if the services are dependant on external resources which may become unavailable or exhibit unexpected behavior. Even if you follow good software development discipline, this is always a source of concern. Think network outages, unannounced third party API changes, hosted services becoming unavailable, etc.\nFor large software projects, creating a testing strategy that involves unit and integration is helpful when managing complexity of the commit to deployment workflow. Functional/smoke tests are also good to ensure critical functionality works as expected. Although, in an environment where the running of your software is dependent on external resources, you need a system of continuous monitoring that run these smoke tests.\nMonitoring Confusion # At Xignite, we use Gallio for our testing framework and CC.NET for our continuous integration. I used these tools for my production smoke tests but soon realized that not all tests were equal. Getting paged at 2am for a test failure that is not mission critical sucks. Even worse, these lower priority failures can mask very high priority ones since they cause the entire test fixture to fail, and require scrutiny on behalf of the dev-ops person to ensure that a high priority failure doesn't fall through the cracks.\nConsider the following test fixture and how it lumps all the different tests together.\nnamespace MyServices.Tests.Smoke { using Gallio.Framework; using Gallio.Model; using MbUnit.Framework; [TestFixture] public class OneOfMyServicesTests { [SetUp] public void Setup() { // Setup an individual test to run. } [FixtureSetUp] public void TestFixtureSetUp() { // Configure fixture for testing. } [FixtureTearDown] public void TestFixtureTearDown() { if (TestContext.CurrentContext.Outcome == TestOutcome.Failed) { // Send signal to monitoring system that test fixture has failed. } else if (TestContext.CurrentContext.Outcome == TestOutcome.Passed) { // Send signal to monitoring system that test fixture has succeeded. } else { // Handle some other outcome. } } [Test] public void MissionCritical() { // ... } [Test] public void Important() { // ... } [Test] public void MinorFunctionality() { // ... } } } Any test failure will make the entire context outcome to fail whether it was the mission critical test or the test that affects minor functionality. I tried looking through the Gallio/MbUnit API, event its source, but couldn't find a way to find out which tests failed within a fixture. If anyone knows how to determine this, please let me know.\nPrioritized Testing # What I do know though is that you can inherit the TestAttribute class and override its Execute method. I created a required parameter to specify the priority of the test and then used a LoggedTestingContext class to store all the results.\nnamespace MyServices.Tests { using Gallio.Framework.Pattern; using MbUnit.Framework; public class LoggedTestAttribute : TestAttribute { public const int MinPriority = 1; public const int MaxPriority = 3; private readonly int priority; public int Priority { get { return this.priority; } } public LoggedTestAttribute(int priority) { if (priority \u0026lt; MinPriority || priority \u0026gt; MaxPriority) { throw new ArgumentException(\u0026#34;Priority must be 1, 2, or 3.\u0026#34;, \u0026#34;priority\u0026#34;); } this.priority = priority; } protected override void Execute(PatternTestInstanceState state) { try { base.Execute(state); LoggedTestingContext.AddTest(this, state.Test, true); } catch (Exception) { LoggedTestingContext.AddTest(this, state.Test, false); throw; } } } } namespace MyServices.Tests { using System; using System.Collections.Generic; using System.Linq; using Gallio.Framework; using Gallio.Model.Tree; public static class LoggedTestingContext { private class TestFailure { public string FullName { get; private set; } public LoggedTestAttribute TestAttribute { get; private set; } public TestFailure(LoggedTestAttribute testAttribute, Test test) { this.FullName = test.FullName; this.TestAttribute = testAttribute; } } private const int PriorityCount = LoggedTestAttribute.MaxPriority - LoggedTestAttribute.MinPriority + 1; private static readonly Dictionary nameToFailure = new Dictionary(); internal static void AddTest(LoggedTestAttribute testAttribute, Test test, bool passed) { if (passed) { return; } var failure = new TestFailure(testAttribute, test); if (!nameToFailure.ContainsKey(failure.FullName)) { nameToFailure.Add(failure.FullName, failure); } } private static bool HasFailed(Test fixtureTest, int priority) { return fixtureTest.Children .Any(c =\u0026gt; nameToFailure.ContainsKey(c.FullName) \u0026amp;\u0026amp; nameToFailure[c.FullName].TestAttribute.Priority == priority); } public static void LogSmokeTests(Test fixtureTest, string serviceName) { foreach (var priority in Enumerable.Range(LoggedTestAttribute.MinPriority, PriorityCount)) { if (HasFailed(fixtureTest, priority)) { // Send signal to monitoring system that test fixture has failed for priority # tests. } else { // Send signal to monitoring system that test fixture has succeeded for priority # tests. } } } } } Finally, putting it together, I replaced the TestAttribute with the new LoggedTestAttribute and then process the results in the test fixture teardown.\nnamespace MyServices.Tests.Smoke { using Gallio.Framework; using Gallio.Model; using MbUnit.Framework; [TestFixture] public class OneOfMyServicesTests { [SetUp] public void Setup() { // Setup an individual test to run. } [FixtureSetUp] public void TestFixtureSetUp() { // Configure fixture for testing. } [FixtureTearDown] public void TestFixtureTearDown() { LoggedTestingContext.LogSmokeTests(TestContext.CurrentContext.Test, \u0026#34;OneOfMyServices\u0026#34;); } [LoggedTest(1)] public void MissionCritical() { // ... } [LoggedTest(2)] public void Important() { // ... } [LoggedTest(3)] public void AffectsFunctionalityByDoesntRequireImmediateAttention() { // ... } } } More Reading \u0026amp; Resources # mb-unit: Downloads CruiseControl.NET ","date":"17 October 2014","externalUrl":null,"permalink":"/using-ccnet-gallio-for-priority-based/","section":"Posts","summary":"Pitfalls in Production # Being able to monitor production services for potential errors is critical. Especially if the services are dependant on external resources which may become unavailable or exhibit unexpected behavior. Even if you follow good software development discipline, this is always a source of concern. Think network outages, unannounced third party API changes, hosted services becoming unavailable, etc.\n","title":"Using CC.NET \u0026 Gallio for priority based smoke testing.","type":"posts"},{"content":"","date":"17 March 2014","externalUrl":null,"permalink":"/tags/.net/","section":"Tags","summary":"","title":".NET","type":"tags"},{"content":"","date":"17 March 2014","externalUrl":null,"permalink":"/tags/c%23/","section":"Tags","summary":"","title":"C#","type":"tags"},{"content":"","date":"17 March 2014","externalUrl":null,"permalink":"/tags/design-patterns/","section":"Tags","summary":"","title":"Design Patterns","type":"tags"},{"content":" The Creeping Problem # I recently found myself developing a request-response style system, where the lifetime of a request could be interrupted at any moment. For most same process execution, like your average desktop application, this is a concern, but arises more often when dealing with multiple coordinated processes. My case ended up being the latter.\nOne of the ways to ensure redundancy is to isolate the steps of the request-response workflow into isolated atomic units or states. This way, if it fails, it can always be re-executed without having to perform the work that came before it. It is especially helpful when the total resources required are large and there is a higher probability of failure. We can just divvy up the work into states that act like idempotent functions. Below is a great simplification of the actual project I worked on but I wanted to boil it down to its simplest form, eliminating excessive states that I have collapsed to the CreateResponse state.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} :::\nIn my original implementation, I modeled the requests as queued items (UserRequest) that I would dequeue and start work on.\nforeach (var request in requests) // as IEnumerable { switch (request.State) { case State.ReceiveRequest: if (TryReceiveRequest(request)) request.State = State.CreateResponse; break; case State.CreateResponse: if (TryCreateResponse(request)) request.State = State.SendResponse; break; case State.SendResponse: if (TrySendResponse(request)) request.State = State.ResponseSent; break; case State.ResponseSent: break; case State.Faulted: default: throw new ArgumentOutOfRangeException(\u0026#34;request.State\u0026#34;); } if (request.State != State.ResponseSent \u0026amp;\u0026amp; request.State != State.Faulted) requests.Enqueue(request); } Seems simple enough, but in my case the CreateResponse state ended being fairly computationally intensive and could take anywhere from a few seconds to several minutes. These long delays could be due to the workload of remote processes it was waiting on, temporal failure points like the network or even the system the process was running on. Another added complexity was that these requests were being serviced in parallel, by multiple processes that could be on the same system or not. Lastly, actual production level code never ends up being this simple; you quickly find yourself adding a lot of instrumentation and covering of edge cases.\nforeach (var request in requests) { log.LogDebug(\u0026#34;request.Id = {0}: Request dequeued in state {1}.\u0026#34;, request.Id, request.State); switch (request.State) { case State.ReceiveRequest: logger.LogDebug(\u0026#34;request.Id = {0}: Trying to receive request.\u0026#34;, request.Id); if (TryReceiveRequest(request)) request.State = State.CreateResponsePart1; break; case State.CreateResponsePart1: logger.LogDebug(\u0026#34;request.Id = {0}: Trying to create response for part 1.\u0026#34;, request.Id); if (TryCreateResponsePart1(request)) request.State = State.CreateResponsePart2; break; case State.CreateResponsePart2: logger.LogDebug(\u0026#34;request.Id = {0}: Trying to create response for part 2.\u0026#34;, request.Id); if (TryCreateResponsePart2(request)) { request.State = State.CreateResponsePart3; } else { request.State = State.CreateResponsePart1; ExecuteCreateResponsePart2Cleanup(); logger.LogError(\u0026#34;request.Id = {0}: Unexpected failure while evaluation create response part 2.\u0026#34;, request.Id); } break; case State.CreateResponsePart3: logger.LogDebug(\u0026#34;request.Id = {0}: Trying to create response for part 3.\u0026#34;, request.Id); bool unrecoverable; if (TryCreateResponsePart3(request, out unrecoverable)) { request.State = State.SendResponse; } else { if (unrecoverable) { logger.LogError(\u0026#34;request.Id = {0}: Failure is unrecoverable, faulting request.\u0026#34;, request.Id); request.State = State.Faulted; } else { request.State = State.CreateResponse2; } } break; case State.SendResponse: logger.LogDebug(\u0026#34;request.Id = {0}: Trying to send response.\u0026#34;, request.Id); if (TrySendResponse(request)) request.State = State.ResponseSent; break; case State.ResponseSent: break; case State.Faulted: logger.LogCritical(\u0026#34;request.Id = {0}: Request faulted.\u0026#34;, request.Id); break; default: throw new ArgumentOutOfRangeException(\u0026#34;request.State\u0026#34;); } log.LogDebug(\u0026#34;request.Id = {0}: Request transitioned to state {1}.\u0026#34;, request.Id, request.State); if (request.State != State.ResponseSent \u0026amp;\u0026amp; request.State != State.Faulted) { logger.LogDebug(\u0026#34;request.Id = {0}: Re-enqueuing request for further evaluation.\u0026#34;, request.Id); requests.Enqueue(request); } else { logger.LogDebug(\u0026#34;request.Id = {0}: Request evaluation is complete, not re-enqueuing.\u0026#34;, request.Id); } } What a mess! This code quickly starts getting bloated. In addition, not every state evaluation will be successful and be considered exceptional. Maybe it is polling another process and can't transition until that process is ready. As the result of each state evaluation changes beyond a simple yes/no (true/false), we end up with a state machine that could have multiple transitions. This makes for ugly code and too much coupling. All the state evaluation logic is in the same class and you have this huge switch statement. We could get around the extensive logging by using dependency injection but what do we inject? There is no consistent call site signature to inject to. The ever growing case statements could be extracted into their own method, but then readability suffers. This sucks.\nYou may be saying, \u0026quot;Well you obviously could solve it by ...\u0026quot; and I would agree with you. This code ugliness could be solved many different ways, and is intentionally crap for the purpose of this post. The major problems I faced was:\nA large state machine object and large code blocks. Lack of symmetry in state handling. Multiple method postconditions that couldn't be expressed by the boolean return result alone. Coupling of state transition logic, business logic and diagnostics. I knew something was wrong but I wasn't quite sure how to solve it without adding more complexity to the system and allowing readability to suffer. As someone who has had to spend hours reading other people's unreadable code, I didn't want to commit the same sin.\nLooking For A Solution # In university, they teach you how to be a good Computer Scientist; you learn complexity analysis, synthetic languages and the theoretical underpinning of computation. Although, none of this really prepares you to be a software engineer. I could concoct my own system, by why do this when I can stand on the shoulders of giants.\nI always read or heard references to the Gang of Four book, even listened to talks by the original authors and became familiar with some of the more famous patterns (Factory and Singleton come to mind). Maybe there was a solution in there. I can't be the first one to come across this simple design problem. So there I found it in the State design pattern.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} :::\nThe design is pretty simple. You have a context that is used by the end user, and the states themselves wrapped by the context. The context can have a range of methods that behave differently based on the concrete state type being used at that moment (eg. behavior of a cursor click in a graphics editor). I modified this design, using a single method to abstract workflow and act as a procedural agent for processing multiple state machines.\nThe Code # The first step was to construct a state object that will be the super type to all of my concrete states.\npublic abstract class StateBase { // Let the concrete type decide what the next transition state will be. protected abstract StateBase OnExecute(); public StateBase Execute() { // Can add diagnostic information here. return this.OnExecute(); } } Next I need a context class that can create and run the state machine.\npublic abstract class StateContextBase { private StateBase state; protected abstract StateBase OnCreate(); protected abstract StateBase OnExecuted(StateBase nextState); protected abstract bool OnIsRunning(StateBase state); public StateContextBase(StateBase state) { this.state = state; } public StateContextBase Execute() { // Need to create the state machine from something. if (this.state == null) { // We will get to this later. this.state = this.OnCreate(); } // Let the concrete context decide what to do after a state transition. this.state = this.OnExecuted(state.Execute()); return this; } public bool IsRunning() { // Have the concrete type tell us when it is in the final state. return this.OnIsRunning(this.state); } } While glossing over the details, what will this look like at the application's entry point.\nclass Program { static void Main(string[] args) { // Will need to get it from somewhere but won\u0026#39;t worry about this for now. var requests = Enumerable.Empty\u0026lt;StateContextBase\u0026gt;(); // Can be changed to false on an exit call. var running = true; while (running) { requests = requests .Where(r =\u0026gt; r.IsRunning()) .Select(r =\u0026gt; r.Execute()); } } } That is beautiful! All I see is the state machine decider logic and I don't even need to be concerned with what type of state machines are running.\nSo let's dive into the details. First, there is the creation of the state into memory. We have to get this from somewhere, so let's add another abstraction on top of our StateBase super type. Something that can be persisted in case the process crashes and can be accessed across many systems (eg. database).\nIn my case, I used the Entity Framework ORM, which is based off of the unit of work and repository design patterns. There is a context (DataContext) that I will get my model object (UserRequest) from to figure out the current state. A unique key (UserRequest.Id : Guid) will be used to identify the persisted object. We won't concern ourselves as to why this is just unique and not an identity key (that could be in another post) but it basically comes down to the object's initial creation at runtime not relying on any persistence store for uniqueness.\npublic class DataContext : System.Data.Entity.DbContext { public DbSet UserRequests { get; set; } public DataContext() : base(\u0026#34;name=DataContext\u0026#34;) { } } public abstract class PersistedStateBase\u0026lt;TEntity\u0026gt; : StateBase where TEntity : class { private Guid id; protected abstract StateBase OnExecuteCommit(DataContext context, Guid id, TEntity entity); protected abstract TEntity OnExecuteCreate(DataContext context, Guid id); protected abstract StateBase OnExecuteRollback(DataContext context, Guid id, TEntity entity); public PersistedStateBase(Guid id) { this.id = id; } protected override StateBase OnExecute() { // Also consider exceptions thrown by DataContext. StateBase nextState = this; using (var context = new DataContext()) { TEntity entity = null; try { entity = this.OnExecuteCreate(context, this.id); nextState = this.OnExecuteCommit(context, this.id, entity); context.SaveChanges(); } catch (Exception ex) { // Handle exception. nextState = this.OnExecuteRollback(context, this.id, entity); } } return nextState; } } The model object (UserRequest, our entity type) will hold the state as an enumeration (UserRequest.State) and contain all the data needed for processing through the state machine.\npublic enum UserRequestState { None = 0, Receive = 1, CreateResponse = 3, SendResponse = 4, ResponseSent = 5, Faulted = -1, } [DataContract] public class UserRequest { [DataMember] public Guid Id { get; private set; } [DataMember] public UserRequestState State { get; private set; } // Other properties here like the location of the user request and other metadata. private UserRequest() // Required by EF to create the POCO proxy. {} public UserRequest(Guid id, UserRequestState state) { this.Id = id; this.State = state; } } Now let's implement our first state using the types we have created.\npublic class ReceiveState : PersistedStateBase\u0026lt;UserRequest\u0026gt; { public ReceiveState(Guid id) : base(id) {} protected override StateBase OnExecuteCommit(DataContext context, Guid id, UserRequest entity) { var successful = false; var faulted = false; // Receive user request and decide whether successful, unsuccessful with retry or // unrecoverable/faulted. if (successful) { return new CreateResponseState(id); } else { return faulted ? new FaultedState(id) : this; } } protected override UserRequest OnExecuteCreate(DataContext context, Guid id) { // Get model object return context.UserRequests.Find(id); } protected override StateBase OnExecuteRollback(DataContext context, Guid id, UserRequest entity) { // Rollback any changes possibly made in the OnExecuteCommit method and attempt recovery, // if possible, in this method. For this example, we will just return the current state. return this; } } We need to also make our state context concrete with the type below. This tends to have more wiring since type per state doesn't really translate well in an ORM. This class could be greater simplified with attributes on the state types, designating the enumeration value they map to.\npublic class UserRequestContext : StateContextBase { private static Dictionary\u0026lt;Type, UserRequestState\u0026gt; typeToDbState; private static bool databaseRead = false; public Guid Id { get; private set; } static UserRequestContext() { databaseRead = false; typeToDbState = new Dictionary\u0026lt;Type, UserRequestState\u0026gt;() { { typeof(ReceiveState), UserRequestState.Receive }, { typeof(CreateResponseState), UserRequestState.CreateResponse}, { typeof(SendResponse), UserRequestState.SendResponse}, { typeof(ResponseSent), UserRequestState.ResponseSent}, { typeof(FaultedState), UserRequestState.Faulted }, }; } public UserRequestContext(Guid id) : base(null) { this.Id = id; } public static IEnumerable\u0026lt;Guid\u0026gt; GetRunningIds() { if (UserRequestContext.databaseRead) { var ids = Enumerable.Empty\u0026lt;Guid\u0026gt;(); // Get from message queue. return ids; } else { using (var dataContext = new DataContext()) { var ids = dataContext.UserRequests .Where(u =\u0026gt; u.State != UserRequestState.ResponseSent \u0026amp;\u0026amp; u.State != UserRequestState.Faulted) .Select(u =\u0026gt; u.Id) .ToArray(); // Force evaluation. UserRequestContext.databaseRead = true; return ids; } } } protected override bool OnIsRunning(StateBase state) { return !(state is CompleteState); } protected override StateBase OnCreate() { using (var dataContext = new DataContext()) { // Maps persisted state enumeration to runtime types. var entity = dataContext.UserRequests.Find(this.Id); switch (entity.State) { case UserRequestState.Receive: return new ReceiveState(this.Id); case UserRequestState.CreateResponse: return new CreateResponseState(this.Id); case UserRequestState.SendResponse: return new SendResponseState(this.Id); case UserRequestState.ResponseSent: return new ResponseSentState(this.Id); case UserRequestState.Faulted: return new FaultedState(this.Id); default: throw new ArgumentOutOfRangeException(); } } } protected override StateBase OnExecuted(StateBase nextState) { // Run any other deciding logic in here that is independent // of the states themselves (eg. logging, perf counters). return nextState; } } Finally, let's come full circle and show what the application entry point will look like once all is said and done.\nclass Program { static void Main(string[] args) { var requests = Enumerable.Empty\u0026lt;StateContextBase\u0026gt;(); var running = true; while (running) { requests = requests .Where(r =\u0026gt; r.IsRunning()) // Append new user requests found. .Concat((IEnumerable)UserRequestContext .GetRunningIds() .Select(i =\u0026gt; new UserRequestContext(i))); .Select(r =\u0026gt; r.Execute()); } } } Ahhhh Yeah... # After fleshing it all out, I really got that satisfying feeling you get as a software engineer when you know that you made the right design decisions. I isolated my business logic into their own types (eg. ReceiveRequestState), separated it from the state machine transition logic, added symmetrical handling of persistence logic by layering it on top of the state type (PersistedStateBase) and contained the persistence-runtime bridge (from UserRequest to PersistedStateBase subtypes) into its own type (UserRequestContext). If I want to add more states, I can simply add to the model's state enumeration (UserRequest.State) and update the state context (UserRequestContext). If I want to change the transition logic, all I need to do is go to the concrete state type itself (eg. ReceiveRequestState) and feel confident that my variables are all scoped correctly. No coupling, no excessive mutations and no excessive side effects.\nUsing The Right Tool # This design pattern isn't for every state machine problem. In simple cases, it can definitely be overkill; you can see a bit of starter code is needed. Although, if you find yourself designing a state machine with multiple outbound transitions and final states, this could be the right modified pattern for you.\nMore Reading \u0026amp; Resources # Amazon: Design Patterns Book Wikipedia: Design Patterns (AKA Gang of Four Book) Wikipedia: State Pattern Software Engineering Radio - Episode 81: Interview with one of the Gang of Four. Design Patterns Uncovered: The State Pattern State Design Pattern: Code examples in other languages. State Pattern: Really simple implementation in Java. ","date":"17 March 2014","externalUrl":null,"permalink":"/state-behavioral-pattern-to-rescue/","section":"Posts","summary":"The Creeping Problem # I recently found myself developing a request-response style system, where the lifetime of a request could be interrupted at any moment. For most same process execution, like your average desktop application, this is a concern, but arises more often when dealing with multiple coordinated processes. My case ended up being the latter.\n","title":"State behavioral pattern to the rescue.","type":"posts"},{"content":"","date":"14 April 2013","externalUrl":null,"permalink":"/tags/lambda-calculus/","section":"Tags","summary":"","title":"Lambda Calculus","type":"tags"},{"content":"","date":"14 April 2013","externalUrl":null,"permalink":"/tags/lambda-notation/","section":"Tags","summary":"","title":"Lambda Notation","type":"tags"},{"content":"","date":"14 April 2013","externalUrl":null,"permalink":"/tags/type-theory/","section":"Tags","summary":"","title":"Type Theory","type":"tags"},{"content":" Series Introduction # I initially was going to put this together as a single post but quickly realized that it would be a bit much to try and put all of the information into one post. To make it more digestable, I decided to break it up into separate parts. This is by no means a comprehensive study of lambda calculus or type theory, but rather a quick introductory series so you can can get started. My goal is that, by the end, you will be able to understand enough to start reading papers on your own and have some cursory knowledge of type systems. Also, I am targeting programmers that want to get a better understanding of the theory behind type systems in compiler design. If you are comfortable programming in functional languages, then you will start to see many parallels quickly. Although, I will give examples using an imperative approach to cover programmers who may not immediately appreciate these parallels.\nMathematical Formalisms # The mathematical underpinnings for describing computability is mostly famously attributed to Kurt Gödel, Alan Turing and Alonzo Church. Church and Turing took different routes to the same ending conclusion (ie. universal Turing machine, -calculus). What is now known as the Church-Turing thesis. In this post we will focus strictly on Church's work, more specifically his notation. By understanding the lambda calculus syntax, we will start to grasp an idealized model of a programming language and what it means later for learning type systems.\nNotation \u0026amp; Basic Structure # Let's look at a function that returns the square of its input:\n::: {align=\u0026ldquo;center\u0026rdquo;} Squared Function :::\n::: {align=\u0026ldquo;center\u0026rdquo;} int f(int x) { return x * x; } :::\nIn lambda calculus, we can represent it as:\n::: {align=\u0026ldquo;center\u0026rdquo;} Squared Function :::\nLet's break this down into its parts and start defining those pieces.\nTerm: Since x is a variable, it can be considered a term and the whole expression itself is also a term (specifically a lambda abstraction). Bound Variable: Any variable in the term that is a parameter of the function. In this case it is x. Free Variable: The term above has none since any argument to the term would be applied to all variables in it. If the term were to be instead, then y would be our free variable. We will give another example that demonstrates free variables later. Abstraction Symbol: The symbol indicates the beginning of the function, followed by the parameter. Function Body: Everything following the dot is part of the function body (ie. ). \\\nBinding \u0026amp; Applying Terms # Now the switch from our named function (f) to the lambda term was not entirely without loss. The symbol is not the function name but rather indicates that the following symbol (x) is a function parameter, just like the period represents the beginning of the function body. This is where the term anonymous function comes from. We took a named function (f) and anonymized it. Now let's try defining the function without loss of information.\n::: {align=\u0026ldquo;center\u0026rdquo;} Squared Function :::\nIt can sometimes be taken for granted in programming that a function name is just another type of variable. The above example makes that pretty clear. Much like a program function name directly references the code it needs to execute, we bound the function to a variable (f) that directly references the term. Although the lowercase letter name isn't considered good convention in lambda calculus. Terms are normally bound to a capitalized letter in these situations; it would be better form to call it F instead.\n::: {align=\u0026ldquo;center\u0026rdquo;} Squared Function :::\nThat's better. What about function pointers? How would they be expressed in the lambda calculus? Let's define some function pointer (G), our previous function (F) and bind them in the C language:\nint (*G)(int); int F(int x) { return x * x; } int main(void) { G = \u0026amp;F; (*G)(2); } This is generally equivalent to:\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nI could have written our term as but this is implicit in the lambda calculus. Terms are always applied from left to right; first F is applied to G and then 2 is applied to that. Having these semantics eliminates the need for excessive parentheses. Now let's break down the application of the terms in steps.\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nThe first line is like the declares in our C program (line 1 and 2). The second line is akin to when we assign the address of the function F to G (line 6) and dereference G below it (line 7). The third line is when we apply the value 2 to the function (line 7).\nMultiple Parameters # You may have noticed that all of our terms only accept a single parameter. This is called the curried form or Schönfinkeled form if you want to get all crazy about it. You may have heard this mentioned before when people talk about closures. Since we can't demonstrate currying in C (nested functions aren't supported), let's do it in Python:\n# uncurried form def f(x, y): return x # curried form using named functions def g(x): def h(y): return x return h # curried form using lambda functions h = lambda x: lambda y: x f(2,1) g(2)(1) h(2)(1) I took a lot of liberties comparing the lambda terms earlier against the C program and is why I wrote that they were generally equivalent. You may have noticed how line 12 looks almost exactly like the actual lambda calculus notation form of . This is why programmers who know functional programming have a much easier time learning lambda calculus and type systems. In Python, the type system is dynamic in the sense that a single variable can be rebound to a different type during its lifetime. Also, lambda functions are explicitly available, where they were not in C. These kind of features are what differentiates different languages in terms of expressibility with respect to the lambda calculus.\nBeta Reductions # You might not have known it, but by applying different terms, you were performing what is called a beta-reduction. This is the process of normalizing to its beta-redex term form. So in the case of FG, the beta-redex term is . Let's try something with more terms and reduce it.\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nThat normalized nicely. Let's do another one.\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nThis is a case where we could keep applying the terms ad-nauseum but that isn't needed because the repeating term is , which is also our beta-redex term. So there is no need to go any further. Now let's do another example using free variables.\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nI mentioned at the beginning I would give an example using free variables and here it is (ie. t). So what happened to 5? Well it was bound to w and w is not in the function body of the last lambda term ( ), so it didn't survive. Although t did and that is because t is not bound to anything, thus making it a free variable.\nBeta reductions are a lot like refactoring some complex code down to its simpler and functionally equivalent form.\nUntyped Lambda Calculus # So far we haven't really considered types. In the examples where multiplication was required, we went beyond the simple untyped system to demonstrate the similarities of lambda calculus with other more familiar systems of computation. In the untyped system, the only type is the function type. Our previous example could still be considered in the untyped system if we just interpret 5 as a free variable; attaching no meaning to the variable itself.\nAs developers, our building blocks are binary logical operations. Let's start by defining some of these essentials in the lambda calculus.\nLet T be true and F be false:\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nNow let's define some binary operations:\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nTo convince ourselves that these will give the desired results, let's start with AND:\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nThen for good measure, let's do OR as well:\n::: {align=\u0026ldquo;center\u0026rdquo;} :::\nLearning the untyped variant of lambda calculus is a good starting place, because we don't place any domain restrictions on our functions and it is fairly simple to get started defining particular behaviors.\nComing up next # Part two will cover type systems and get you familiar with the syntax. If you want to learn lambda calculus in more depth or go through some exercises, check out the links below. Also, please feel free to post comments with questions, suggestions or corrections.\nMore Reading \u0026amp; Resources # Notes on Untyped Lambda Calculus An Introduction to Lambda Calculi and Arithmetic A short introduction to the Lambda Calculus Lectures Notes on the Lambda Calculus A Tutorial Introduction to the Lambda Calculus Lambda Calculus Tutorial An Introduction to Type Theory A Short Introduction to Type Theory ","date":"14 April 2013","externalUrl":null,"permalink":"/understanding-type-theory-part-one/","section":"Posts","summary":"Series Introduction # I initially was going to put this together as a single post but quickly realized that it would be a bit much to try and put all of the information into one post. To make it more digestable, I decided to break it up into separate parts. This is by no means a comprehensive study of lambda calculus or type theory, but rather a quick introductory series so you can can get started. My goal is that, by the end, you will be able to understand enough to start reading papers on your own and have some cursory knowledge of type systems. Also, I am targeting programmers that want to get a better understanding of the theory behind type systems in compiler design. If you are comfortable programming in functional languages, then you will start to see many parallels quickly. Although, I will give examples using an imperative approach to cover programmers who may not immediately appreciate these parallels.\n","title":"Understanding type theory (Part one: Lambda Calculus).","type":"posts"},{"content":"","date":"10 April 2013","externalUrl":null,"permalink":"/tags/clr/","section":"Tags","summary":"","title":"CLR","type":"tags"},{"content":"","date":"10 April 2013","externalUrl":null,"permalink":"/tags/networking/","section":"Tags","summary":"","title":"Networking","type":"tags"},{"content":"","date":"10 April 2013","externalUrl":null,"permalink":"/tags/pcap-ng/","section":"Tags","summary":"","title":"PCAP-NG","type":"tags"},{"content":" Introduction \u0026amp; Scope # The PCAP-NG file format is used to store network captured data used by great programs like Wireshark. There are several API's available for both Linux and Windows but none that are ported to .NET. I decided to make one for my own use in building a network traffic replay agent.\nThis code demonstrates an approach to reading in PCAP-NG files. Most of the time I was concerned with the enhanced packet block type since it stored the TCP payloads I wanted to replay.\nDesign Considerations # All block types derive from a common abstract class (BlockBase). This is useful when you need a collection of blocks with different subtypes, as in the second code segments below. Internally, all classes populate themselves from the underlying binary reader.\nThe reader class can be used to iterate over a single block type.\nvar reader = new PcapngFile.Reader(\u0026#34;test.pcapng\u0026#34;); foreach (var packet in reader.EnhancedPackets) { byte[] payload = packet.Data; } ... or all of them, cast into their parent type.\nvar blocks = new List\u0026lt;PcapngFile.BlockBase\u0026gt;(); while (reader.PeekStoreType() != PcapngFile.BlockType.None) { blocks.Add(reader.ReadBlock()); } The Code and NuGet Package # Get access to the complete project here on my GitHub account. If you have any suggestions or comments, feel free to leave a comment. I also have the assembly available via NuGet here.\nMore Reading \u0026amp; Resources # WinPcap: PCAP-NG File Format Wireshark: Development - PcapNg Pcap.Net: Useful for reading transport payloads. ","date":"10 April 2013","externalUrl":null,"permalink":"/pcap-ng-reader-for-net/","section":"Posts","summary":"Introduction \u0026 Scope # The PCAP-NG file format is used to store network captured data used by great programs like Wireshark. There are several API's available for both Linux and Windows but none that are ported to .NET. I decided to make one for my own use in building a network traffic replay agent.\n","title":"PCAP-NG reader for .NET.","type":"posts"},{"content":" Introduction \u0026amp; Scope # When developing a new web service, you may need to do an initial spot test walk through the operations or try to break the service based on the permissible data type values. You could hand craft all the operations, with their respective arguments, into a handful of URLs but that would seem pretty inefficient. I found myself in a similar situation and wanted a programmatic way to walk through the service operations with a specific input data set. To be clear, I am not attempting to recreate a full web service testing application but rather a simple script to spot check responses and catch any bugs in a web service endpoint's handler.\nInvestigating the WSDL # We can automatically flesh out our operations and the message data types by leveraging the endpoint's WSDL. In my case, I was specifically interested in the request/response operations available under the HTTP GET verb. We can get the parameter names and argument types by looking at the types section and then the actual operations under a WSDL binding that includes an HTTP binding element with the verb attribute value of GET.\nFortunately, I am dealing with input messages using just the built in data type schema provided by the WSDL specification. This means that no complex types are involved and the testing input domain is somewhat simplified. Also, most of the web service operations use a standard set of parameter names such that I am able to create a smaller data set of inputs based around them.\nThe Code # Your modifications would be to the test_args dictionary. You can specify a wide range of different arguments to feed your operations by parameter names. The service I was testing against used many of the same parameter names and corresponding domain.\nConclusion # Thrown together, the script provides an easy way to spot test a web service by automatically processing the WSDL for a particular binding and running those bound operations against a provided set of input messages for it.\nMore Reading \u0026amp; Resources # Wikipedia: Web Services Description Language MSDN: Understanding WSDL ","date":"26 March 2013","externalUrl":null,"permalink":"/automated-combinatorial-and-spot/","section":"Posts","summary":"Introduction \u0026 Scope # When developing a new web service, you may need to do an initial spot test walk through the operations or try to break the service based on the permissible data type values. You could hand craft all the operations, with their respective arguments, into a handful of URLs but that would seem pretty inefficient. I found myself in a similar situation and wanted a programmatic way to walk through the service operations with a specific input data set. To be clear, I am not attempting to recreate a full web service testing application but rather a simple script to spot check responses and catch any bugs in a web service endpoint's handler.\n","title":"Automated combinatorial and spot testing of web service operations.","type":"posts"},{"content":"","date":"26 March 2013","externalUrl":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":"","date":"26 March 2013","externalUrl":null,"permalink":"/tags/rest/","section":"Tags","summary":"","title":"REST","type":"tags"},{"content":"","date":"26 March 2013","externalUrl":null,"permalink":"/tags/soap/","section":"Tags","summary":"","title":"SOAP","type":"tags"},{"content":"","date":"26 March 2013","externalUrl":null,"permalink":"/tags/web-services/","section":"Tags","summary":"","title":"Web Services","type":"tags"},{"content":"","date":"26 March 2013","externalUrl":null,"permalink":"/tags/wsdl/","section":"Tags","summary":"","title":"WSDL","type":"tags"},{"content":"","date":"4 January 2013","externalUrl":null,"permalink":"/tags/markov-chains/","section":"Tags","summary":"","title":"Markov Chains","type":"tags"},{"content":"","date":"4 January 2013","externalUrl":null,"permalink":"/tags/natural-language-processing/","section":"Tags","summary":"","title":"Natural Language Processing","type":"tags"},{"content":" Introduction # If you are already familiar with finite automata, then understanding a Markov chain will not be as hard. If you are not familiar with one, you can think of finite automata as a flowchart, with states and paths leading from one state to another based on some decision being made. An example could be a room with a light switch; there are the two states of having the light on or off. Using the light switch will transition you between the two states.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;400\u0026rdquo; height=\u0026ldquo;183\u0026rdquo;} :::\nThe difference between standard finite automata and Markov chains is that instead of having some definite outcome determining each path (eg. yes or no), they are probabilities. Take for example a coin flip, there is a 50% (0.5) chance that it could either be heads or tails.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;400\u0026rdquo; height=\u0026ldquo;259\u0026rdquo;} :::\nAn Example # Let's explore Markov chains a little deeper with respect to natural languages and take a look at some sample text that will form the corpus of our analysis.\n\u0026ldquo;I like turtles. I like rabbits. I don't like snails.\u0026rdquo;\nThis would render a Markov chain as follows:\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;400\u0026rdquo; height=\u0026ldquo;345\u0026rdquo;} :::\nThe above chain means that, among the sentences:\n100% (1.0) will start with I. This will be followed by like for 66% (0.66) of the time and don't for 33% (0.33) of the time. The word don't will always (100% / 1.0) be followed by like. Then lastly, turtles, rabbits and snails will all follow like 33% (0.33) of the time. The nice thing about Markov chains, with respect to natural language modelling, is that they form word dependencies without any knowledge of the language's syntax or semantics. The chain gets created purely based on statistical knowledge extracted from the corpus.\nText Generation # Random Selection # Using the Markov chain example from before and taking the weight of probabilities into account, we can randomly traverse the chain to generate new sentences like:\n\u0026ldquo;I don't like turtles. I like snails.\u0026rdquo;\nAlthough, our small corpus does not make for very interesting or new text. We could make it more interesting by adding to it or switching to a much larger corpus altogether. In my case, I decided to take all of Shakespear's sonnets and generate the text from that. You can see some example code that uses it as the corpus below.\nThe Code # The Markov chains will be built up for each new sentence it comes across. The critical parts to the MarkovChain object are its:\nCardinalities: The cardinality of the sample space for a given state. In our original example, the cardinality of like is 3 since snails, rabbits and turtles follow out of the like state. The cardinality of I would be 2 since don't and like follow. Occurrences: The edge valued occurrences between the different states. For example, a transition from the I to like state occurred twice. The probability can then be easily calculated by taking the number of occurrences on a transition and divide it by the cardinality of the state's sample space. Note, there is also a special start state transition that can be used at the beginning of each new sentence generation. Using these two parts, a random walk can be made across a chain path, starting from the special start state. Each walk will generate a new sentence based on probabilities of the transitions along the path.\n\\\n\\\nAdditional Resources # Wikipedia: Stochastic Grammar Wikipedia: Natural language generation Princeton CS206: Markov Model of Natural Language Markov-chain Text Generator ACL Special Interest Group on Natural Language Generation (SIGGEN) ","date":"4 January 2013","externalUrl":null,"permalink":"/text-generation-using-markov-chains/","section":"Posts","summary":"Introduction # If you are already familiar with finite automata, then understanding a Markov chain will not be as hard. If you are not familiar with one, you can think of finite automata as a flowchart, with states and paths leading from one state to another based on some decision being made. An example could be a room with a light switch; there are the two states of having the light on or off. Using the light switch will transition you between the two states.\n","title":"Text generation using Markov chains.","type":"posts"},{"content":"","date":"3 October 2012","externalUrl":null,"permalink":"/tags/ado.net/","section":"Tags","summary":"","title":"ADO.NET","type":"tags"},{"content":"","date":"3 October 2012","externalUrl":null,"permalink":"/tags/raw-text/","section":"Tags","summary":"","title":"Raw Text","type":"tags"},{"content":" Introduction # In some cases, you may want to read/write binary data from/to your relational database. This can be a problem when the data set size exceeds the memory address size (eg. 32-bit processes) or the managed large object heap cannot load it all without throwing an OutOfMemoryException. In those cases, you will need to load the data into memory in chunks. Fortunately, the T-SQL syntax supports this operation.\nLatest Standard # The old standard for reading and writing raw text data was by using the UPDATETEXT, READTEXT and TEXTPTR commands. Although, these are being obsoleted for the newer command SUBSTRING and the addition of the .WRITE argument to the UPDATE command. I found the newer syntax much easier to use and requiring less parameters. Below is an example schema and implementation.\nExample Schema # For this example, we will use a very simple table setup with a primary key and binary data column.\nCREATE TABLE [dbo].[DataTable] ( [Id] [int] IDENTITY(1,1) NOT NULL, [Data] [varbinary](max) NOT NULL ) Example Code # The Read method simply provides an offset into the data and the size of the chunk it wishes to read. The same applies for the Write method, except that a data argument is also supplied.\nprivate const int ReadChunkSize = 1 \u0026lt;\u0026lt; 21; // 2MB Chunks private const int WriteChunkSize = 1 \u0026lt;\u0026lt; 21; // 2MB Chunks ... public static void Read(string connectionString, string readPath) { using (var connection = new SqlConnection(connectionString)) { int offsetIndex = 0; var buffer = new byte[1]; var offset = new SqlParameter(\u0026#34;@Offset\u0026#34;, SqlDbType.Int, 32); var command = new SqlCommand(@\u0026#34;SELECT SUBSTRING([Data], @Offset, \u0026#34; + ReadChunkSize + \u0026#34;) FROM [dbo].[DataTable] WHERE [Id] = 1\u0026#34;, connection); command.Parameters.Add(offset); connection.Open(); using (var stream = File.Create(readPath)) { using (var writer = new BinaryWriter(stream)) { while (buffer.Length \u0026gt; 0) { offset.Value = offsetIndex; buffer = (byte[])command.ExecuteScalar(); if (buffer.Length \u0026gt; 0) { writer.Write(buffer); offsetIndex += ReadChunkSize; } } } } } } public static void Write(string connectionString, string writePath) { using (var connection = new SqlConnection(connectionString)) { int offsetIndex = 0; var buffer = new byte[1]; var offset = new SqlParameter(\u0026#34;@Offset\u0026#34;, SqlDbType.Int, 32); var data = new SqlParameter(\u0026#34;@Data\u0026#34;, SqlDbType.Binary, WriteChunkSize); var command = new SqlCommand(@\u0026#34;UPDATE [dbo].[DataTable] SET [Data] .WRITE(@Data, @Offset, \u0026#34; + WriteChunkSize + \u0026#34;) WHERE [Id] = 1\u0026#34;, connection); command.Parameters.Add(offset); command.Parameters.Add(data); connection.Open(); using (var stream = File.OpenRead(writePath)) { using (var reader = new BinaryReader(stream)) { while (buffer.Length \u0026gt; 0) { buffer = reader.ReadBytes(WriteChunkSize); if (buffer.Length \u0026gt; 0) { offset.Value = offsetIndex; data.Value = buffer; command.ExecuteNonQuery(); offsetIndex += WriteChunkSize; } } } } } } Performance Considerations # When benchmarking my code against the simple database schema, execution time was drastically different if the data was already read into memory. When a read was performed after a write, it took 2 seconds. A read without a write beforehand took about 10 seconds. This is also the same time that the write took before a read. The chunk size did not seems to have any significant impact when adjusted between 512K and 4MB sizes. Your results may vary though depending on your system's memory layout.\nThese time benchmarks are very informal and highly dependent on system architecture, network latency and hardware specifications. Although, the relative performance gain between the two benchmarks can be gleaned from these tests.\nA further performance gain would be to front load the data into memory on a separate thread and process the loaded data into and out of the database on another. This would require some tweaking and knowledge of the system's memory constraints to ensure it doesn't become a performance bottleneck. If properly done, a lot can be gained from not having to block on I/O between every chunk read and write.\n","date":"3 October 2012","externalUrl":null,"permalink":"/reading-and-writing-large-binary-data/","section":"Posts","summary":"Introduction # In some cases, you may want to read/write binary data from/to your relational database. This can be a problem when the data set size exceeds the memory address size (eg. 32-bit processes) or the managed large object heap cannot load it all without throwing an OutOfMemoryException. In those cases, you will need to load the data into memory in chunks. Fortunately, the T-SQL syntax supports this operation.\n","title":"Reading and writing large binary data in T-SQL and ADO.NET.","type":"posts"},{"content":"","date":"3 October 2012","externalUrl":null,"permalink":"/tags/t-sql/","section":"Tags","summary":"","title":"T-SQL","type":"tags"},{"content":"","date":"20 September 2012","externalUrl":null,"permalink":"/tags/algorithms/","section":"Tags","summary":"","title":"Algorithms","type":"tags"},{"content":"","date":"20 September 2012","externalUrl":null,"permalink":"/tags/c/","section":"Tags","summary":"","title":"C","type":"tags"},{"content":"","date":"20 September 2012","externalUrl":null,"permalink":"/tags/compression/","section":"Tags","summary":"","title":"Compression","type":"tags"},{"content":"","date":"20 September 2012","externalUrl":null,"permalink":"/tags/entropy-encoding/","section":"Tags","summary":"","title":"Entropy Encoding","type":"tags"},{"content":" Definitions # In a lot of texts regarding compression you will run across various conventions of syntax to describe the coding of data. Much of them are used in fields like automata theory, group theory and statistics. Although, the field of coding theory itself has a long and rich history in the 20st century; much of the terminology is borrowed from fields that gave rise to it.\nI find the best way to demonstrate the following definitions is by example. Let us take the text \u0026quot;abbcc\u0026quot; as a sample.\nAlphabet: The set of symbols that we will be encoding. Form: Sigma Example: Alphabet Example where Alphabet Example String: A permutation of symbols from our alphabet. From our example alphabet, strings like \u0026quot;a\u0026quot;, \u0026quot;aba\u0026quot;, \u0026quot;abc\u0026quot; or \u0026quot;cccb\u0026quot; could all be possible. Using regular expression syntax, we can describe this in our example. Form: String Example: String Example Probabilities: The set of probabilities for the alphabet. This is also known as their weights and can sometimes be denoted as w instead of p. Form: Probability Example: Weights Example where Probabilities Example since 'a' occurs once, 'b' occurs twice and 'c' occurs twice out of 5 symbols all together. Codewords: The set of codewords that we will create for every symbol in our alphabet.\\ Form: Codeword Example: Codeword Example where Codeword Example for symbols Codeword Example respectively. Another way to look at it is that we are providing a bijective function: Symbol to Codeword Mapping . Code Width # Standard coding of information for most computers today is byte aligned. For example, in character codes there is ASCII, which codes the entire English alpha-numeric system in a single byte. While UTF-8 has proven to be a much better standard due to better localization support and variable width codes, ASCII serves our purposes for this discussion. ASCII is considered a fixed width coding scheme:\n::: {align=\u0026ldquo;center\u0026rdquo;} ASCII Codeword :::\nWhen performance is at stake, it is best to code information atomically aligned to the computer architecture that it is being used for. This means allocating and defining information in terms of bytes (defined as 8-bits) for the majority of systems today. Although, if space is the priority, performance can take a hit to shift around bits along the byte boundaries and create variable width codes that do not byte align.\n::: {align=\u0026ldquo;center\u0026rdquo;} Variable Width Codewords :::\nNote: The ellipses in the alphabet denote the control characters (eg. CR, LB) and grammatical structure symbols (eg. #, $, %) contained in the ASCII code page. The alphabets will reflect this in the ellipses.\nData Compression # Variable Width Savings # Coming back to the example of ASCII, if we wish to encode a string, w, like \u0026quot;aaabb\u0026quot;, it will take up 5 bytes. Although we only really need one bit to encode our information if we assign 0 to 'a' and 1 to 'b'.\n::: {align=\u0026ldquo;center\u0026rdquo;} String Example 1 :::\nWe just took a 5 byte ASCII encoded string and compressed it down to under a byte. Now let us take another string, w, and give it \u0026quot;aaabbcc\u0026quot;. Three alphabet symbols need to be coded; it will take at least 2 bits. At first glance, this assignment would make sense:\n::: {align=\u0026ldquo;center\u0026rdquo;} String Example 2 :::\nAlthough this gives us an ambiguous string since it could either be \u0026quot;aaabbcc\u0026quot;, \u0026quot;aaabbabab\u0026quot;, \u0026quot;aacbcc\u0026quot; or any other permutation of our codeword set when we attempt to decompress it. The reason for this ambiguity is due to the fact that the codeword for 'a' (0), is used to start the codeword for 'c' (01).\nPrefix Codes # The way to get around this ambiguity is to ensure that all of your codes are prefix type. Something like this would work:\n::: {align=\u0026ldquo;center\u0026rdquo;} String Example 2 :::\nThis ensures that we maintain the one-to-one mapping between our alphabet and codewords; we can translate alphabet symbols to codewords and codewords back to alphabet symbols without confusion.\nCodeword Assignment # So how do you decide which symbols, get which codewords? Also, since the size of the codeword is critical to the compression ratio (average codeword length to average symbol length), designing an algorithm to weigh the more frequently used symbols is important. In our previous example, it would be poor design to give 'a' the longest codeword because it is the most used symbol in the string. This is where the Huffman Coding algorithm comes into play.\nHuffman Coding Algorithm # Count the number of times each symbol occurs in the data source.\nPut [symbol:count] pairing as nodes in a minimum priority queue based on the symbol's count.\nEvaluate number of queue nodes.\nIf queue has at least two nodes: Dequeue the two nodes. Create a new node with a null symbol and set the count equal to the sum of the two dequeued node's counts. Attach the two nodes as children to the new node. Put the newly created node into the queue. Go to step 3. If queue has only one node, dequeue and set as the root of the tree. Assign each left branch a 0 and each right branch a 1, or vice versa if you choose.\nThe symbol nodes will be the children of the tree and their codeword will be the bits that are traversed to get there.\nAlgorithm Example # Given our previous example string of \u0026quot;aaaabbcc\u0026quot;, let us perform the algorithm on it:\nInitial queue state with only symbol nodes in it.\n{width=\u0026ldquo;267\u0026rdquo; height=\u0026ldquo;53\u0026rdquo;}\n\\ New node created from first two symbol nodes and requeued.\n{width=\u0026ldquo;237\u0026rdquo; height=\u0026ldquo;127\u0026rdquo;}\n\\ New node created from non-symbol and symbol node, becoming the tree's root node.\n{width=\u0026ldquo;280\u0026rdquo; height=\u0026ldquo;204\u0026rdquo;}\n\\ The final alphabet to codeword mapping is:\nAlgorithm Results If you need further examples, you can look here, here and here. Once the you get a grasp of the algorithm, you will see the simplicity and beauty of it.\nWorking Code Example # If you want to see a naive implementation of the Huffman Coding algorithm, I posted some source code on my GitHub account. Below is the central code to the algorithm execution.\nMore Reading \u0026amp; Resources # If you are interested in coding algorithms, these links might be of interest to you.\nWikipedia: Arithmetic Coding Code Project: Arithmetic Compression With C# LZW Compression Mohamed F. Mansour: Efficient Huffman Decoding With Table Lookup Wikipedia: Golomb-Rice Coding Zlib Homepage ","date":"20 September 2012","externalUrl":null,"permalink":"/understanding-compression-and-huffman/","section":"Posts","summary":"Definitions # In a lot of texts regarding compression you will run across various conventions of syntax to describe the coding of data. Much of them are used in fields like automata theory, group theory and statistics. Although, the field of coding theory itself has a long and rich history in the 20st century; much of the terminology is borrowed from fields that gave rise to it.\n","title":"Understanding compression and Huffman Coding.","type":"posts"},{"content":"","date":"18 September 2012","externalUrl":null,"permalink":"/tags/imapi/","section":"Tags","summary":"","title":"IMAPI","type":"tags"},{"content":" Introduction # Microsoft introduced native image mastering support in Vista, going forward. The library itself is not too complex but the latest version of the image mastering API (IMAPI) has some quirks. This is well documented by Eric Haddan in his article describing how to build an interop file and use it from .NET.\nImplementation # I wanted to take it a step further and provide a library wrapper that was CLI compliant. The end result is a class that provides 3 methods, called in order, to burn an image. The project can be found on my GitHub account.\nProperties # Media: A read only property that populates after the LoadMedia call and reports what type of physical media is connected (eg. CD-R, DVD-R, DVD+RW, etc.). MediaStates: A read only collection of states in which the media is under, that populates after the LoadMedia call (eg. is blank, overwriteable, unsupported). MediaCapacity: A read only property that populates after the LoadMedia call and reports the size capacity of the media. Nodes: A root list of file and directory nodes that are to be written to the media. Recorders: A read only selectable collection. A recorder must be selected for the LoadRecorder call. VolumeLabel: A property that sets the volume label of the media to be written to. Methods # LoadRecorder: Verifies a recorder is selected, loads the recording drive resources and verifies that it is working. LoadMedia: Loads the media resource and verifies that it is valid for the recorder. An exception will occur if the media isn't supported (eg. closed session disc, no disc present). FormatMedia: Formats the media of any previous data. WriteImage: The final step to write your files to disc. This is a blocking call. Events # FormatEraseUpdate: Reports progress of FormatMedia call. FormatWriteUpdate: Reports progress of files being written to the media. ImageUpdate: Reports progress of files being added to the media image. Caveats # I use this library for the simple single purpose of writing finalized images to a blank CD. It has not been fully tested to accomodate all media and recorder state scenarios; I haven't tried it yet with DVD's or multi-session discs. If you would like to extend the library, feel free to push some changes to the GitHub project and I will wrap them in.\nI decided to leave the method calls as blocking (synchronous) since there could be many different synchronization models that may use this library. For my purposes, I used this in an WPF application that would call the WriteImage method from a Task and then post back the progress events of the write using the Dispatcher from my WPF window. Below is a code snippet demonstrating this.\nprivate ImageMaster _burner; ... private void _CreateCd() { // files added to _burner and _burner.WriteImage called } private void _burner_FormatWriteUpdate(IMAPI2.ImageMaster sender, IMAPI2.FormatWriteUpdateEventArgs e) { this.Dispatcher.Invoke(() =\u0026gt; this.statusProgressBar.Value == (e.ElapsedTime / e.TotalTime) * 100); } private void CreateCdWindow_Loaded(object sender, System.Windows.RoutedEventArgs e) { System.Threading.Tasks.Task.Factory.StartNew(_CreateCd); } ","date":"18 September 2012","externalUrl":null,"permalink":"/imapi-version-2-managed-wrapper/","section":"Posts","summary":"Introduction # Microsoft introduced native image mastering support in Vista, going forward. The library itself is not too complex but the latest version of the image mastering API (IMAPI) has some quirks. This is well documented by Eric Haddan in his article describing how to build an interop file and use it from .NET.\n","title":"IMAPI version 2 managed wrapper.","type":"posts"},{"content":"","date":"13 September 2012","externalUrl":null,"permalink":"/tags/bit-twiddling/","section":"Tags","summary":"","title":"Bit Twiddling","type":"tags"},{"content":" Introduction # I was working on implementing a simple compression tool based on naive Huffman coding and initially assumed all of my prefix codes would be at or under a byte long. Although I failed to account for the worst case of Huffman coding, which can yield codes up to 255 bits in length.\nThis required a bit reading and writing function that would allow for variable length codes. I could have used other methods to balance worst case Huffman trees, but this issue presented a unique challenge. How to write a bit reader/writer in C.\nBit Reader/Writer Specifications # Read and write using reader and writer ADT's. Handle any possible bit length given. Remember bits read from file but are outside the requested bit length (eg. 2 bytes read in but only 12 bits requested, leaving 4 bits in the buffer). Flush a partially written block to disk on a free of the bit writer ADT. Return number of bits read and written on the read and write functions. Design # The underlying structure to the ADT will contain fields for the: file pointer, the byte long buffer and the buffer bit length. The examples given for every case are in steps; what is originally in the buffer, what bits are requested, what bits are read from or written to the file, and finally what bits are left over and buffered.\nReader # There are two main cases to consider when designing the read function.\nEmpty Buffer: The is simple since the file read bits will already be byte aligned to your output bits. You just need to simply mask out the unneeded bits and buffer them for the next read (eg. no bits buffered, 12 bits requested, 16 bits read in, 4 bits buffered).\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;400\u0026rdquo; height=\u0026ldquo;42\u0026rdquo;} :::\nPopulated Buffer: The can get a little trickier, especially in the last two cases of this.\nRequested bits already buffered. This is very easy since it doesn't require any file read and the buffer just needs to shift out the read bits (eg. 7 bits in the buffer and 3 are requested; buffer serves out 3 and truncates to 4).\nNot all requested bits buffered; end of read bits not in same byte block as end of output bits. The last output byte block must mask out the unneeded bits, put them in the buffer and then append the additional unneeded read bits from the next byte block (eg. 4 bits buffered, 13 bits requested, 16 bits read in, 8 bits buffered). ::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} :::\nNot all requested bits buffered; end of read bits in same byte block as end of output bits. The last output byte block must mask out the unneeded bits and put them in the buffer (eg. 4 bits buffered, 10 bits requested, 8 bits read in, 2 bits buffered). ::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;400\u0026rdquo; height=\u0026ldquo;42\u0026rdquo;} :::\nWriter # There are only two cases to consider when designing the write function.\nFull Buffer: Shift in write bits, commit to disk and repeat until all bits are processed. If the last bit to write is not on the buffer byte boundary, it will wait until the next write (eg. 6 bits buffered, 12 bits to write, 16 bits written out, 2 bits buffered). Partial Buffer: Shift in write bits (eg. 2 bits buffered, 4 bits to write, no bits written out, 6 bits buffered). Caveats # The code below is purely a demonstration of how a bit reader/writer works in a generic sense. There is room for plenty of optimization if you know what type of architecture your targeting and the domain of your reader/write arguments.\nThe Code # Header File # Source File # ","date":"13 September 2012","externalUrl":null,"permalink":"/implementing-bit-readerwriter-in-c/","section":"Posts","summary":"Introduction # I was working on implementing a simple compression tool based on naive Huffman coding and initially assumed all of my prefix codes would be at or under a byte long. Although I failed to account for the worst case of Huffman coding, which can yield codes up to 255 bits in length.\n","title":"Implementing a bit reader/writer in C.","type":"posts"},{"content":" Motivation # One of the main concerns in any licensing implementation is that authored license files are coming from the trusted authority (ie. your company) and not some third party. This can be accomplished in software by encrypting and decrypting the license with a single key, but symmetric key storage can be cumbersome and have added complexity.\nAssymetric cryptography, on the other hand, allows for two keys: a private one to author files with and a public one to verify its authenticity. This is well suited for licensing, where we want to ensure that the authored license has not been altered in anyway and the content can be relied on.\nImplementation # I ran across a great article that details the use of XML digital signatures in the .NET framework. It describes the type of digital signatures that can be implemented and lays out the classes and methods needed to author and verify license files.\nThis specific implementation allows for an: authentication key to compare against some unique value tying the license to the system, expiration date to set a time span on the software use and set of available features the user is authorized for.\nBase Class # This forms the runtime object representation of the license and will be inherited by the license reader and writer.\nusing System; using System.Collections.Generic; using System.Collections.ObjectModel; using System.Text; using System.Security; using System.Security.Cryptography; using System.Security.Cryptography.Xml; using System.Xml; namespace Licensing { public abstract class LicenseBase { private byte[] _authenticationKey; private DateTime _expiration; private List _features; private int _id; public byte[] AuthenticationKey { get { return _authenticationKey; } protected set { _authenticationKey = value; } } public DateTime Expiration { get { return _expiration; } protected set { _expiration = value; } } public int Id { get { return _id; } protected set { _id = value; } } protected List Features { get { return _features; } } public LicenseBase() { _features = new List(); } public void Clear() { _id = 0; _authenticationKey = null; _features.Clear(); } } } Writer Class # This class will be used internally to author the license files that will be distributed with the released software. Note that the private key is added as an embedded resource in the project and should not be distributed since the public key can be derived from it.\nusing System; using System.IO; using System.Security; using System.Security.Cryptography; using System.Security.Cryptography.Xml; using System.Text; using System.Xml; namespace Licensing { class License : LicenseBase { License(int id, byte[] authenticationKey, DateTime expiration, string[] features) { this.AuthenticationKey = authenticationKey; this.Expiration = expiration; this.Id = id; this.Features.AddRange(features); } /// /// Generates a key pair for digital signing and verification. /// /// /// static internal void GenerateAssymetricKeys() { string datestamp = null; StreamWriter output = null; RSA key = RSA.Create(); key.KeySize = 1024; datestamp = DateTime.UtcNow.ToString(\u0026#34;yyyyMMdd\u0026#34;); // Generate private key to only be used internally (DO NOT DISTRIBUTE). output = File.CreateText(\u0026#34;private-\u0026#34; + datestamp + \u0026#34;.key\u0026#34;); output.Write(key.ToXmlString(true)); output.Close(); // Generate public key to be used by customers (distribute). output = File.CreateText(\u0026#34;public-\u0026#34; + datestamp + \u0026#34;.key\u0026#34;); output.Write(key.ToXmlString(false)); output.Close(); } /// /// Digitally sign an XML document. /// /// The XML document to sign. /// The private key to sign it with. /// private static void _SignXmlDocument(System.Xml.XmlDocument document, RSA privateKey) { SignedXml signedDocument = new SignedXml(document); signedDocument.SigningKey = privateKey; signedDocument.SignedInfo.CanonicalizationMethod = SignedXml.XmlDsigCanonicalizationUrl; // Add reference to XML data Reference @ref = new Reference(\u0026#34;\u0026#34;); @ref.AddTransform(new XmlDsigEnvelopedSignatureTransform(false)); signedDocument.AddReference(@ref); // Build the signature. signedDocument.ComputeSignature(); // Attach the signature to the XML document. XmlElement signature = signedDocument.GetXml(); document.DocumentElement.AppendChild(signature); } /// /// Write the contents and digitally sign the document. /// /// The file path to the digitally signed document. public void WriteDocument(string filepath) { XmlDocument document = new XmlDocument(); document.AppendChild(document.CreateXmlDeclaration(\u0026#34;1.0\u0026#34;, \u0026#34;UTF-8\u0026#34;, null)); XmlElement root = document.CreateElement(\u0026#34;License\u0026#34;); document.AppendChild(root); XmlElement id = document.CreateElement(\u0026#34;Id\u0026#34;); id.InnerText = this.Id.ToString(); root.AppendChild(id); XmlElement authenticationKey = document.CreateElement(\u0026#34;AuthenticationKey\u0026#34;); authenticationKey.InnerText = Convert.ToBase64String(this.AuthenticationKey); root.AppendChild(authenticationKey); XmlElement expiration = document.CreateElement(\u0026#34;Expiration\u0026#34;); expiration.InnerText = this.Expiration.ToString(); root.AppendChild(expiration); XmlElement options = document.CreateElement(\u0026#34;Features\u0026#34;); XmlElement featureItem = null; foreach (string feature in this.Features) { featureItem = document.CreateElement(\u0026#34;Feature\u0026#34;); featureItem.InnerText = feature; options.AppendChild(featureItem); } root.AppendChild(options); XmlElement publickey = document.CreateElement(\u0026#34;Certificate\u0026#34;); publickey.SetAttribute(\u0026#34;DateCode\u0026#34;, KeyCreationDateCode); publickey.InnerXml = File.ReadAllText(PublicKeyFilename); root.AppendChild(publickey); RSA privateKey = RSA.Create(); privateKey.FromXmlString(Licensing.Properties.Resources.privatekey); _SignXmlDocument(document, privateKey); File.WriteAllText(filepath, document.InnerXml); } } } Reader Class # This class will be used in the field to verify the license files that will be distributed with the released software. It is best to wrap the license read and authenticate methods in try-catch blocks since they represent exceptional behavior that fall outside the post-conditions of the method. Handling the exceptions will allow for better error reporting in the field so you can decide what information you will share with the end user (eg. bad authentication key, malformed/tampered licenses). Note again that the public key is an embedded resource for the project. It should never be distributed with the license since an attacker could generate their own keypair and envelope their own public key, with signature, in the license.\nusing System; using System.Collections.Generic; using System.Collections.ObjectModel; using System.Text; using System.Security; using System.Security.Cryptography; using System.Security.Cryptography.Xml; using System.Xml; namespace Licensing { class LicenseReader : LicenseBase { LicenseReader() { } public void Authenticate() { // Check the hardware key value against the AuthenticationKey if (!_AuthenticationKeyMatches(hardwareKey)) throw new LicenseAuthenticationException(\u0026#34;The license failed to authenticate the hardware key.\u0026#34;); // ... or check some node locking ID (eg. MAC ID, hard drive serial number) against the AuthenticationKey. if (!_AuthenticationKeyMatches(nodeLockedId)) throw new LicenseAuthenticationException(\u0026#34;The license failed to authenticate the node locked ID.\u0026#34;); if (this.Expiration \u0026gt; DateTime.UtcNow) throw new LicenseAuthenticationException(\u0026#34;The license has expired.\u0026#34;); } /// /// Compare byte array against authentication key byte array. /// /// Byte array to compare against. /// True if a match, else false. private bool _AuthenticationKeyMatches(byte[] compare) { if (this.AuthenticationKey == null) return false; int upperBound = Math.Min(this.AuthenticationKey.Length, compare.Length); for (int i = 0; i \u0026lt; upperBound; i++) { if (this.AuthenticationKey[i] != compare[i]) return false; } return true; } public bool IsFeature(string featureName) { return this.Features.Contains(featureName); } /// /// Read the digitally signed document and load its contents. /// /// The file path to the digitally signed document. protected virtual void ReadDocument(string filepath) { this.Clear(); XmlDocument document = new XmlDocument(); document.Load(filepath); RSA publicKey = RSA.Create(); publicKey.FromXmlString(Licensing.Properties.Resources.publickey); if (_VerifyXmlDocument(document, publicKey)) { this.Id = int.Parse(document.SelectSingleNode(\u0026#34;//License/Id\u0026#34;).InnerText); this.AuthenticationKey = Convert.FromBase64String(document.SelectSingleNode(\u0026#34;//License/AuthenticationKey\u0026#34;).InnerText); this.Expiration = DateTime.Parse(document.SelectSingleNode(\u0026#34;//License/Expiration\u0026#34;).InnerText); XmlNodeList features = document.SelectNodes(\u0026#34;//License/Features/Feature\u0026#34;); foreach (XmlNode feature in features) { this.Features.Add(feature.InnerText); } } } /// /// Verify the digital signature of an XML document. /// /// The XML document containing the signature. /// The public key to verify signature authenticity. /// True if the signature is authentic, else false. /// private bool _VerifyXmlDocument(XmlDocument document, RSA publicKey) { SignedXml signedDocument = new SignedXml(document); try { XmlNode signature = document.GetElementsByTagName(\u0026#34;Signature\u0026#34;, SignedXml.XmlDsigNamespaceUrl)[0]; signedDocument.LoadXml((XmlElement)(signature)); } catch { return false; } return signedDocument.CheckSignature(publicKey); } } } Additions \u0026amp; Caveats # Symmetric encryption could also be used to hinder casual viewing of the license file. The issue of symmetric key storage must be revisited but will no longer be critical to maintaining the integrity of the license files.\nStrong naming of the license reading assembly is encouraged to prevent dissasembly and re-embedding of an attacker's own public key. Although, this is by no means an attack-proof method since strong naming can be removed by changing information in the CLI header of the portable executable (PE). Even if the application was able to secure the public key outside the application in a PKI, there is still the issue of code injection or recompilation once strong naming is stripped. Indirection of the public key or obfuscation of the assembly can also help but is not a reliable security measure.\n","date":"14 August 2012","externalUrl":null,"permalink":"/licensing-using-xml-digital-signing/","section":"Posts","summary":"Motivation # One of the main concerns in any licensing implementation is that authored license files are coming from the trusted authority (ie. your company) and not some third party. This can be accomplished in software by encrypting and decrypting the license with a single key, but symmetric key storage can be cumbersome and have added complexity.\n","title":"Licensing using XML digital signing.","type":"posts"},{"content":"","date":"14 August 2012","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"14 August 2012","externalUrl":null,"permalink":"/tags/xml/","section":"Tags","summary":"","title":"XML","type":"tags"},{"content":" Background # There are plenty of sites that will tell you how to enable IIS through the Control Panel. Although, if you are tasked with enabling IIS automatically (eg. for an installer), it can be a nightmare. This is only really needed though in special cases where the installer has more control over the hardware specifications. Determining an upgrade or uninstall scenario with IIS can be very tough since you are not installing your own software but rather enabling a feature of Windows. With that in mind, we can propose a solution for automated enabling of IIS.\nRequirements # Windows Editions # The edition of Windows 7 for IIS must be either Ultimate or Professional if you plan on using ASP.NET. You can try this on other editions, or even Vista, but your results may differ since this is only scenario I have tested against. More information can be found at the IIS TechNet site.\nDISM Tool # To enable IIS features through the command line, DISM can be used. This can be reliably found on Windows 7 but is inconsistent on Vista, where the older Package Manager is used instead. This was one of the reasons I decided to constrain the specification to Windows 7.\nThe first step in using DISM, is to find out which features need to be enabled, in order for IIS to work completely. I ran across this MSDN blog entry that lists all the needed features. Next task is to script the DISM commands.\nScripting # Caveats # Iterating over each feature and enabling it with DISM can be very time consuming. The faster approach is to evaluate the enabled state of each feature and turn them all on in one command. PowerShell can be used but the presence of it and its needed modules can be inconsistent on Windows 7. I had to fall back on batch scripting, which is the part that can cause nightmares due to its ugly syntax.\nArrays are need to store the enabled states of each feature and the feature names themselves. In my case, I needed an array of pairs. I was able to fake it using a suggestion on Stack Overflow. Also, special syntax is needed to reference variables inside a loop that are scoped outside the loop.\nLast, but not least, remember to register the ASP.NET framework version being used with IIS. This will need to be done once IIS is enabled (reboot is required). Adjust the framework bitness and version below according to the one you are using.\n[WindowsFolder]\\Microsoft.NET\\Framework64\\v4.0.30319\\aspnet_regiis.exe -i The Code # The following batch script will discover which features are disabled, concatenate them into the enable command and then execute. You may need to replace SysNative with system32 or the Windows system folder being used by your target system if this is being ran outside the installer.\nsetlocal enableextensions enabledelayedexpansion set features=IIS-ApplicationDevelopment IIS-ASPNET IIS-CommonHttpFeatures IIS-HostableWebCore IIS-HttpCompressionDynamic IIS-HttpCompressionStatic IIS-HttpErrors IIS-HttpLogging IIS-HttpRedirect IIS-HttpTracing IIS-Metabase IIS-NetFxExtensibility IIS-Performance IIS-StaticContent IIS-URLAuthorization IIS-WebServerRole WAS-ConfigurationAPI WAS-NetFxEnvironment WAS-ProcessModel WAS-WindowsActivationService set dism_params= set n=0 for %%f in (%features%) do ( set /a n+=1 set dism_params=!dism_params! /FeatureName:%%f set features[!n!]=%%f ) set n=0 for /f \u0026#34;tokens=3*\u0026#34; %%i in (\u0026#39;%WINDIR%\\SysNative\\Dism.exe /Online /Get-FeatureInfo %dism_params% ^| findstr /b /r \u0026#34;^State : \u0026#34;\u0026#39;) do ( set /a n+=1 set states[!n!]=%%i ) set dism_params= for /l %%i in (1,1,%n%) do ( if \u0026#34;!states[%%i]!\u0026#34; == \u0026#34;Disabled\u0026#34; set dism_params=!dism_params! /FeatureName:!features[%%i]! ) if \u0026#34;%dism_params%\u0026#34; neq \u0026#34;\u0026#34; %WINDIR%\\SysNative\\Dism.exe /Online /Enable-Feature %dism_params% ","date":"30 July 2012","externalUrl":null,"permalink":"/automate-iis-enable-on-windows-7/","section":"Posts","summary":"Background # There are plenty of sites that will tell you how to enable IIS through the Control Panel. Although, if you are tasked with enabling IIS automatically (eg. for an installer), it can be a nightmare. This is only really needed though in special cases where the installer has more control over the hardware specifications. Determining an upgrade or uninstall scenario with IIS can be very tough since you are not installing your own software but rather enabling a feature of Windows. With that in mind, we can propose a solution for automated enabling of IIS.\n","title":"Automate IIS enable on Windows 7.","type":"posts"},{"content":"","date":"30 July 2012","externalUrl":null,"permalink":"/tags/batch/","section":"Tags","summary":"","title":"Batch","type":"tags"},{"content":"","date":"30 July 2012","externalUrl":null,"permalink":"/tags/iis/","section":"Tags","summary":"","title":"IIS","type":"tags"},{"content":" Background # Unlike most sensors used in data acquisition, thermocouples do not have a linear mapping between their voltage and celcius values. This means that instead of factoring some scalar value against the data, in order to scale it to engineering units (eg. volts to acceleration), an experimentally derived equation must be created to approximate the temperature at specific voltages. This non-linearity is primarily due to the materials being used to discover the temperature difference.\nThe science and engineering behind thermocouples goes beyond the scope of this post but more information can be found below:\nWikipedia: Thermoelectric Effect Wikipedia: Thermocouple Reference Temperatures Simple Demonstration of the Seebeck Effect K-Type Thermocouples # One of the most general purpose thermocouple types used in data acquisition are K types since their are relatively inexpensive to make. This will also be the thermocouple that I will focus on.\nLuckily, NIST provides a table of discrete values, correlating voltages to celcius and vice-versa. In addition, they also provide the exact polynomial equation to use underneath the given tables, depending on the voltage range the value falls in. Following is the exact functions to do the conversion for K-type thermocouples.\nVoltage To Celcius # Given some ordered set of coefficients, D, and a voltages value, v, let the voltage to celcius conversion be defined as a function of v as follows.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;56\u0026rdquo; height=\u0026ldquo;56\u0026rdquo;} :::\nD is dependent on the voltage value being evaluated:\nvalues greater than 206.44 mV will have D = {-131.8058, 48.30222, -1.646031, 0.05464731, -0.0009650715, 0.000008802193, -0.0000000311081, 0.0, 0.0, 0.0 }; values geater than 0.0 V will have D = { 0.0, 25.08355, 0.07860106, -0.2503131, 0.0831527, -0.01228034, 0.0009804036, -0.0000441303, 0.000001057734,-0.00000001052755}; and values less than 0.0 V will have D = { 0.0, 25.173462, -1.1662878, -1.0833638, -0.8977354, -0.37342377, -0.086632643, -0.010450598, -0.00051920577, 0.0 }. Celcius To Voltage # Given some ordered set of coefficients, C, an ordered set of exponentials, A, and a celcius value, x, let the celcius to voltage conversion be defined as a function of x as follows.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;182\u0026rdquo; height=\u0026ldquo;56\u0026rdquo;} :::\nC is dependent on the celius value being evaluated:\npositive and zero values will have C = { -0.017600413686, 0.038921204975, 0.000018558770032, -0.000000099457592874, 0.00000000031840945719, -0.00000000000056072844889, 0.00000000000000056075059059, -3.2020720003E-19, 9.7151147152E-23, -1.2104721275E-26}; negative values will have C = { 0.0, 0.039450128025, 0.000023622373598, -0.00000032858906784, -0.0000000049904828777, -0.000000000067509059173, -0.00000000000057410327428, -0.0000000000000031088872894, -1.0451609365E-17, -1.9889266878E-20, -1.6322697486E-23}; A is a fixed ordered set of exponentials; A = {0.1185976, -0.0001183432, 126.9686}.\nPerformance # Since the polynomial equation's number of terms are bounded by a constant (ie. |D| or |C|) the performance hit will be a constant factor to the number of voltage or celcius values to calculate. Also, temperature is sampled at a very low rate, yielding small data sets that usually have redundant values in them. Further optimization can be made by caching previous computations and reusing their results if the same voltage or celcius value is encountered again.\nThe Code # private static double[] _thermocoupleCoefficientsTypeKneg = { 0.0, 0.039450128025, 2.3622373598E-05, -3.2858906784E-07, -4.9904828777E-09, -6.7509059173E-11, -5.7410327428E-13, -3.1088872894E-15, -1.0451609365E-17, -1.9889266878E-20, -1.6322697486E-23 }; private static double[] _thermocoupleCoefficientsTypeKpos = { -0.017600413686, 0.038921204975, 1.8558770032E-05, -9.9457592874E-08, 3.1840945719E-10, -5.6072844889E-13, 5.6075059059E-16, -3.2020720003E-19, 9.7151147152E-23, -1.2104721275E-26 }; private static double[] _thermocoupleExponentialsTypeK = { 0.1185976, -0.0001183432, 126.9686 }; /// /// Type K thermocouple inverse coefficient values for voltage to celcius conversions. /// /// Valid values for -200C - 0C / -5.891mV - 0mV. private static double[] _thermocoupleInverseCoefficientsTypeK0 = { 0.0, 25.173462, -1.1662878, -1.0833638, -0.8977354, -0.37342377, -0.086632643, -0.010450598, -0.00051920577, 0.0 }; /// /// Type K thermocouple inverse coefficient values for voltage to celcius conversions. /// /// Valid values for 0C - 500C / 0mV - 20.644mV. private static double[] _thermocoupleInverseCoefficientsTypeK1 = { 0.0, 25.08355, 0.07860106, -0.2503131, 0.0831527, -0.01228034, 0.0009804036, -4.41303E-05, 1.057734E-06, -1.052755E-08 }; /// /// Type K thermocouple inverse coefficient values for voltage to celcius conversions. /// /// Valid values for 500C - 1372C / 20.644mV - 54.886mV. private static double[] _thermocoupleInverseCoefficientsTypeK2 = { -131.8058, 48.30222, -1.646031, 0.05464731, -0.0009650715, 8.802193E-06, -3.11081E-08, 0.0, 0.0, 0.0 }; public static double CelciusToVoltageTypeK(double value) { double[] a = _thermocoupleExponentialsTypeK; double[] c = null; double result = 0.0; if ((value \u0026gt;= 0.0)) { c = _thermocoupleCoefficientsTypeKpos; } else { c = _thermocoupleCoefficientsTypeKneg; } for (int index = 0; index \u0026lt;= c.Length - 1; index++) { result += (c[index] * Math.Pow(value, index)) + (a[0] * Math.Exp(a[1] * Math.Pow(value - a[2], 2))); } return result; } public static double VoltageToCelciusTypeK(double value) { double[] d = null; double result = 0.0; if ((value \u0026gt; 0.020644)) { d = _thermocoupleInverseCoefficientsTypeK2; } else if ((value \u0026gt;= 0.0)) { d = _thermocoupleInverseCoefficientsTypeK1; } else { d = _thermocoupleInverseCoefficientsTypeK0; } for (int index = 0; index \u0026lt;= d.Length - 1; index++) { result += d[index] * Math.Pow(value, index); } return result; } ","date":"25 July 2012","externalUrl":null,"permalink":"/converting-thermocouple-voltages-to/","section":"Posts","summary":"Background # Unlike most sensors used in data acquisition, thermocouples do not have a linear mapping between their voltage and celcius values. This means that instead of factoring some scalar value against the data, in order to scale it to engineering units (eg. volts to acceleration), an experimentally derived equation must be created to approximate the temperature at specific voltages. This non-linearity is primarily due to the materials being used to discover the temperature difference.\n","title":"Converting thermocouple voltages to temperature.","type":"posts"},{"content":"","date":"25 July 2012","externalUrl":null,"permalink":"/tags/data-acquisition/","section":"Tags","summary":"","title":"Data Acquisition","type":"tags"},{"content":" Using Conditioned Tags # The Visual Studio 2010 UI does not have options to handle conditional references based on what kind of configuration and platform you are running. For example, you may have a project reference that needs to change depending on whether you are doing a 32 or 64 bit build.\nFortunately, the project file syntax does support conditional references. I was able to find this out on stack overflow.\nIn our case, we have our DLL's in source control under the project's bin directory. For some reason I kept getting the 64-bit version of the DLL and then found out it was because the Content tags also needed to be conditioned.\nExample # Lets say you have a project (MyProject) with two DLL's, 32bit.dll and 64bit.dll, that are referenced by the project and kept under the bin directory for the project:\nMyProject\\bin\\x86\\32bit.dll MyProject\\bin\\x64\\64bit.dll You will need to hand edit the MyProject\\MyProject.vbproj file and add these entries to it (or change it if they are already in there). The first two entries are for the project references and the last two are for their inclusion as a content item in the project.\n\u0026lt;Reference Include=\u0026#34;32bit\u0026#34; Condition=\u0026#34;\u0026#39;$(Platform)\u0026#39; == \u0026#39;x86\u0026#39;\u0026#34;\u0026gt; \u0026lt;HintPath\u0026gt;bin\\x86\\32bit.dll\u0026lt;/HintPath\u0026gt; \u0026lt;/Reference\u0026gt; \u0026lt;Reference Include=\u0026#34;64bit\u0026#34; Condition=\u0026#34;\u0026#39;$(Platform)\u0026#39; == \u0026#39;x64\u0026#39;\u0026#34;\u0026gt; \u0026lt;HintPath\u0026gt;bin\\x64\\64bit.dll\u0026lt;/HintPath\u0026gt; \u0026lt;/Reference\u0026gt; ... \u0026lt;ItemGroup\u0026gt; \u0026lt;Content Include=\u0026#34;bin\\x86\\32bit.dll\u0026#34; Condition=\u0026#34;\u0026#39;$(Platform)\u0026#39; == \u0026#39;x86\u0026#39;\u0026#34;/\u0026gt; \u0026lt;Content Include=\u0026#34;bin\\x64\\64bit.dll\u0026#34; Condition=\u0026#34;\u0026#39;$(Platform)\u0026#39; == \u0026#39;x64\u0026#39;\u0026#34;/\u0026gt; \u0026lt;/ItemGroup\u0026gt; Now, whenever the platform is changed to x86, it will only use the x86 conditioned tags and likewise when changed to x64. This can also be applied to other conditions too, check the links below for a complete breakdown of the MSBuild schema and conditions.\nMSBuild Project File Schema Reference MSBuild Conditions ","date":"23 April 2012","externalUrl":null,"permalink":"/bit-specific-builds-in-visual-studio/","section":"Posts","summary":"Using Conditioned Tags # The Visual Studio 2010 UI does not have options to handle conditional references based on what kind of configuration and platform you are running. For example, you may have a project reference that needs to change depending on whether you are doing a 32 or 64 bit build.\n","title":"Bit specific builds in Visual Studio 2010.","type":"posts"},{"content":"","date":"23 April 2012","externalUrl":null,"permalink":"/tags/visual-studio/","section":"Tags","summary":"","title":"Visual Studio","type":"tags"},{"content":"","date":"21 December 2011","externalUrl":null,"permalink":"/tags/multithreading/","section":"Tags","summary":"","title":"Multithreading","type":"tags"},{"content":"","date":"21 December 2011","externalUrl":null,"permalink":"/tags/wpf/","section":"Tags","summary":"","title":"WPF","type":"tags"},{"content":" WPF \u0026amp; Multithreading # In designing applications, it may be advantageous to run data processing methods on a separate thread from the UI. This allows greater responsiveness in the application and takes advantage of concurrent computation.\nThe cornerstone interface to any binding in WPF is INotifyPropertyChanged, which is implemented on a data structure's different properties. Although this can be the source of deadlocks if the data structure is being processed on a separate thread where locks are explicitly implemented by the developer.\nA deadlock could occur if some UI event was being triggered on a data structure (eg. expanding a view, changing a text box entry) while the data thread was processing it with an acquired lock. Internally, the PropertyChanged event will call a Dispatcher.Invoke on the UI. Although the UI could be waiting for the lock to release on that same property that the data thread already has, creating a cyclic dependency or deadlock.\nThe call stack for the UI thread. Note the wait call.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} :::\nThe call stack for the data thread.\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} :::\nDeadlock Scenario # As an example, suppose you have your standard WPF application UI thread and a separate data thread. The data thread polls a time stamp from a remote database and updates a class property called DataObject.Timestamp{.inlinecode}, which raises the PropertyChanged{.inlinecode} event. The property is also bound to a text box in our WPF application window, which is made visible in the window based on the user's preference (ie. collapsing and expanding a view).\nIn our deadlock example, the data thread puts a lock on the DataObject.Timestamp{.inlinecode} property while it updates it and other data structures. While this is happening, the view is expanded, causing the text box to start binding to the DataObject.Timestamp{.inlinecode} property and wait for an acquire of the lock that the data thread is holding. The data thread then sets the DataObject.Timestamp{.inlinecode} property inside the critical region code and the PropertyChanged{.inlinecode} event is raised. This causes the new time stamp value to be marshaled back onto the Dispatcher{.inlinecode} with a blocking/synchronous Dispatcher.Invoke{.inlinecode} call. Although the data thread now deadlocks in the critical region, waiting for the UI Dispatcher.Invoke{.inlinecode} to return, which is waiting on the data thread to release the lock.\nSolution # To prevent the data thread's deadlock critical region from blocking on the Dispatcher.Invoke{.inlinecode}, override the OnPropertyChanged{.inlinecode} method to marshal the value asynchronously to the dispatcher.\nprotected override void OnPropertyChanged(string propertyName) { Application.Current.Dispatcher.BeginInvoke(new Action(() =\u0026gt; { base.OnPropertyChanged(propertyName); })); } This will ensure that the critical region doesn't block and the property changes are still serviced by the dispatcher.\n","date":"21 December 2011","externalUrl":null,"permalink":"/wpf-and-data-multithreading-deadlocks/","section":"Posts","summary":"WPF \u0026 Multithreading # In designing applications, it may be advantageous to run data processing methods on a separate thread from the UI. This allows greater responsiveness in the application and takes advantage of concurrent computation.\n","title":"WPF and data multithreading deadlocks.","type":"posts"},{"content":"","date":"16 December 2011","externalUrl":null,"permalink":"/tags/android/","section":"Tags","summary":"","title":"Android","type":"tags"},{"content":"","date":"16 December 2011","externalUrl":null,"permalink":"/tags/java/","section":"Tags","summary":"","title":"Java","type":"tags"},{"content":"","date":"16 December 2011","externalUrl":null,"permalink":"/tags/performance/","section":"Tags","summary":"","title":"Performance","type":"tags"},{"content":"While Java may be a higher level language that makes code writeability easier, conventions must be broken when developing on Android to reap performance gains. This is especially true for games where logic and rendering must be done on a tight loop.\nThe Android developer page provides helpful tips for optimizing your Java code. Below are the optimizations that I have found to improve performance.\nFront load memory allocation at startup instead of dynamically.\nWe saw the largest performance gain here. The GC can kill performance in a CPU intensive application or game. If you pool your instances in an array instead of dynamically allocating it, you can save the GC calls from happening. This will result in a slightly larger memory footprint if you tweak it right.\nBelow is code that provides a fixed-size generic list implementation; the internal array never grows/shrinks. The executePendingRemoves routine is needed since object removal must sometimes be put into a critical region with a synchronized block. You may want to queue up removals but still make them available to other threads up until the moment they are removed in bulk.\npublic class List\u0026amp;ltT\u0026gt; { private int mCount; private T mItems[]; private int mRemoveCount; private int[] mRemoveIndices; @SuppressWarnings(\u0026#34;unchecked\u0026#34;) public List(int capacity) { mItems = (T[]) new Object[capacity]; mRemoveIndices = new int[capacity]; mCount = 0; } public void add(T item){ mItems[mCount] = item; mCount++; } public void clear(){ for(int index=0; index \u0026lt; mCount; index++) { mItems[index] = null; } mCount = 0; } public void executePendingRemoves() { for (int i=0;i \u0026lt; mRemoveCount; i++) { mItems[mRemoveIndices[i]] = null; } int newCount=0; for (int i=0;i \u0026lt; mCount; i++) { if (mItems[i] != null) { mItems[newCount] = mItems[i]; newCount++; } } mCount = newCount; mRemoveCount = 0; } public T get(int index) { return mItems[index];} public int getCount() { return mCount;} public int getIndex(T item) { for (int i=0; i \u0026lt; mCount; i++) { if (mItems[i] == item) return i; } return -1; } public int getPendingRemoveCount() { return mRemoveCount; } public int[] getPendingRemoves() { return mRemoveIndices; } public Object[] getItems() { return mItems;} public void pendingRemoveAt(int index) { for (int i=0; i\u0026lt; mRemoveCount; i++) { if (index == mRemoveIndices[i]) return; else if (index \u0026lt; mRemoveIndices[i]) { int swapIndex = mRemoveIndices[i]; mRemoveIndices[i] = index; index = swapIndex; } } mRemoveIndices[mRemoveCount] = index; mRemoveCount++; } public void removeAt(int index) { if (index \u0026lt; mCount) { for (int i = index; i \u0026lt; mCount; i++) { if (i + 1 \u0026lt; mItems.length \u0026amp;\u0026amp; i + 1 \u0026lt; mCount) { mItems[i] = mItems[i + 1]; } else { mItems[i] = null; } } mCount--; } } public T removeLast() { mCount--; final T item = mItems[mCount]; mItems[mCount] = null; return item; } } Lastly is the pooling object that can be inherited by any application level instance (eg. FlyingWizardsPool extends ObjectPool). When you want to instantiate a flying wizard object, just type in FlyingWizard wizard = wizardPool.checkOut();.\npublic abstract class ObjectPool\u0026amp;ltT\u0026gt; { private int mCapacity; private List\u0026amp;ltT\u0026gt; mPool; @SuppressWarnings({ \u0026#34;unchecked\u0026#34;, \u0026#34;rawtypes\u0026#34; }) public ObjectPool(int capacity) { mPool = new List(capacity); mCapacity = capacity; this.fill(); } public void checkIn(T poolItem) { mPool.add(poolItem); } public T checkOut() { return mPool.removeLast(); } public abstract void fill(); public int getCount() { return mPool.getCount(); } public int getCapacity() { return mCapacity; } } Below is an example implementation of the ObjectPool abstract class. Note how the fill routine is implemented. Also, front loading memory like this means that you will need an initialize on your instances so that the class fields can be put back to a \u0026quot;just instantiated\u0026quot; state. Normally we would use a constructor but to keep the allocation static, an initialize call is necessary.\npublic static class FlyingWizardPool extends ObjectPool { public FlyingWizardPool(int capacity) { super(capacity); } public FlyingWizard checkOut(float x, float y, float destinationX, float destinationY) { FlyingWizard wizard = this.checkOut(); wizard.initialize(x, y); return wizard; } @Override public void fill() { for (int i=0; i \u0026lt; this.getCapacity(); i++) { this.checkIn(new FlyingWizard()); } } } Choose larger block routines instead of calling out to helper subroutines.\nSince the Dalvik is a register based architecture, every time a routine is called from another, the register values must be stored to memory before the call and loaded back in after the routine has returned. Avoiding helper routine inside commonly called routines will reduce this load/store overhead.\nRemove accessors and allow direct access to class fields.\nAccessors are usually good form and observe the ADT. Although, to avoid the overhead involved in a routine call and additional allocation of memory for the copied out value, it is best to allow direct access to the class field. This makes it more difficult to protect the conditions for the field but is a worthy sacrifice if it is called often.\nMake routines static if they don't access class fields.\nThe Android team recommends using this for an approximate 15-20% performance gain. This means less work for the Dalvik during run-time optimization and the routine can be inlined for faster execution.\nUse constants instead of enums.\nThis is more of a negligible optimization; it was delisted from Android's performance page. Instead of having an enum, you can keep the enumeration as an integer constant.\nIt is important to follow the ADT for good code writeability/readability and bug minimization. Use these optimizations only on code that really needs it. This means it should apply to routines and fields that are called frequently by your application.\n","date":"16 December 2011","externalUrl":null,"permalink":"/tips-for-java-performance-on-android/","section":"Posts","summary":"While Java may be a higher level language that makes code writeability easier, conventions must be broken when developing on Android to reap performance gains. This is especially true for games where logic and rendering must be done on a tight loop.\n","title":"Tips for Java performance on Android.","type":"posts"},{"content":"","date":"7 November 2011","externalUrl":null,"permalink":"/tags/entity-framework/","section":"Tags","summary":"","title":"Entity Framework","type":"tags"},{"content":"A POCO (Plain Old CLR Object) must be transformed into a POCO proxy\n[MyNamespace.MyClass -\u0026gt;{System.Data.Entity.DynamicProxies.MyClass_0A943C2FC37D33304CEB497A9210C754E3F553B50315F8E6F0F7A6FAF43777F2}]{.Apple-style-span style=\u0026ldquo;font-size: x-small;\u0026rdquo;})\nin order for features like lazy loading and change tracking to work.\nMSDN has a great article that explains how to prepare a POCO to be transformed into a proxy. If you find that an object retrieved or attached to the context does not show itself in the DynamicProxies namespace (eg. MyNamespace.MyClass instead of ...DynamicProxies.MyClass), then you probably don't have a compliant class. You may have noticed this already when hovering your mouse cursor over an instance and seeing the runtime information on it.\nThe distilled points are that your POCO must:\nbe public, not abstract and not sealed; have a public or protected parameterless constructor; have public overridable properties that are read/write if they are navigation properties; and implement ICollection if they are a one-to-many navigation property. ","date":"7 November 2011","externalUrl":null,"permalink":"/poco-not-transforming-into-poco-proxy/","section":"Posts","summary":"A POCO (Plain Old CLR Object) must be transformed into a POCO proxy\n[MyNamespace.MyClass -\u003e{System.Data.Entity.DynamicProxies.MyClass_0A943C2FC37D33304CEB497A9210C754E3F553B50315F8E6F0F7A6FAF43777F2}]{.Apple-style-span style=“font-size: x-small;”})\nin order for features like lazy loading and change tracking to work.\nMSDN has a great article that explains how to prepare a POCO to be transformed into a proxy. If you find that an object retrieved or attached to the context does not show itself in the DynamicProxies namespace (eg. MyNamespace.MyClass instead of ...DynamicProxies.MyClass), then you probably don't have a compliant class. You may have noticed this already when hovering your mouse cursor over an instance and seeing the runtime information on it.\nThe distilled points are that your POCO must:\n","title":"POCO not transforming into a POCO proxy.","type":"posts"},{"content":"I was recently working on solving problem 64 for project Euler and one of the needed algorithms was a detection of cycles in a list. This routine works for detecting cycles in lists of cardinality greater than 2.\n\\\ndef get_cycle_len(N): i = k = 0 for j in range(2, len(N)): if N[i] == N[j]: if k == 0: k = j i = 1 else: if i == k: return k i += 1 else: i = 0 k = 0 return -1 ","date":"16 October 2011","externalUrl":null,"permalink":"/detecting-cycles-in-decimal-digits-and/","section":"Posts","summary":"I was recently working on solving problem 64 for project Euler and one of the needed algorithms was a detection of cycles in a list. This routine works for detecting cycles in lists of cardinality greater than 2.\n\\\n","title":"Detecting cycles in decimal digits and lists.","type":"posts"},{"content":"","date":"8 October 2011","externalUrl":null,"permalink":"/tags/bugs/","section":"Tags","summary":"","title":"Bugs","type":"tags"},{"content":"I recently installed the Entity Framework June 2011 CTP to preview the enum support but ran into a bug along the way that reported an error while processing the template.\n\\\n::: {.separator style=\u0026ldquo;clear: both; text-align: center;\u0026rdquo;} {width=\u0026ldquo;400\u0026rdquo; height=\u0026ldquo;390\u0026rdquo;} :::\nI encountered this error while trying to run an Entity based project using the \u0026quot;Database \u0026amp; Model First\u0026quot; paradigm. Uninstalling the preview removed the problem. Not sure why this error occured since the preview doesn't modify the assembly it is complaining about not finding an entry point in.\n","date":"8 October 2011","externalUrl":null,"permalink":"/entity-framework-june-2011-ctp-bug/","section":"Posts","summary":"I recently installed the Entity Framework June 2011 CTP to preview the enum support but ran into a bug along the way that reported an error while processing the template.\n\\\n","title":"Entity Framework June 2011 CTP Bug","type":"posts"},{"content":"I just started using the Entity Framework recently to employ persistence for an already established code base. Initially I was using the \u0026quot;Model \u0026amp; Database First\u0026quot; paradigm where a designer is used to layout the data model. Although, this ended up being tedious since my POCO instance had to exactly match the POCO proxy instance in my model designer view. After upgrading to EF 4.1, I realized the \u0026quot;Code First\u0026quot; paradigm was the way to go since I had to adapt the persistence model to my already existing code.\nOne of the earliest problems that I ran into was having more than one property on a class that referenced the same class type. This example was taken from a great site that explored the different EF properties.\n[]{.Apple-style-span style=\u0026ldquo;background-color: #f8f8f8; color: #2e2e2e; font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, mono; font-size: 12px;\u0026rdquo;}\\\npublic class Customer { public int Id { get; set; } public int Age { get; set; } public Address BillingAddress { get; set; } public Address DeliveryAddress { get; set; } } Notice how both the [BillingAddress ]{.Apple-style-span style=\u0026ldquo;background-color: #f8f8f8; color: #2e2e2e; font-family: monospace; font-size: 12px; white-space: pre;\u0026quot;}and [DeliveryAddress ]{.Apple-style-span style=\u0026ldquo;background-color: #f8f8f8; color: #2e2e2e; font-family: monospace; font-size: 12px; white-space: pre;\u0026quot;}properties references the same class type. I kept trying to establish two 1-to-1 relationships on the class but this caused a problem with cascading deletes and foreign key constraints. I found a solution but it still required exceptional behavior on the part of my context and the database schema.\nThe most elegant solution was to just treat both of the properties as complex types. The only thing I had to do was remove the [Id ]{.Apple-style-span style=\u0026ldquo;background-color: #f8f8f8; color: #2e2e2e; font-family: monospace; font-size: 12px; white-space: pre;\u0026quot;}property from the properties' class and the context took care of the rest. So instead of two tables in the database (one for [Customer ]{.Apple-style-span style=\u0026ldquo;font-family: monospace; font-size: 12px; white-space: pre;\u0026quot;}and the other for [Address]{.Apple-style-span style=\u0026ldquo;font-family: monospace; font-size: 12px; white-space: pre;\u0026rdquo;}), there would be only one. Below is a layout of what the table would look like.\n\\\nCustomer { Id Age BillingAddress_City BillingAddress_Street BillingAddress_Zipcode DeliveryAddress_City DeliveryAddress_Street DeliveryAddress_Zipcode } \\\n","date":"5 October 2011","externalUrl":null,"permalink":"/ef-41-code-first-many-one-to-one/","section":"Posts","summary":"I just started using the Entity Framework recently to employ persistence for an already established code base. Initially I was using the \"Model \u0026 Database First\" paradigm where a designer is used to layout the data model. Although, this ended up being tedious since my POCO instance had to exactly match the POCO proxy instance in my model designer view. After upgrading to EF 4.1, I realized the \"Code First\" paradigm was the way to go since I had to adapt the persistence model to my already existing code.\nOne of the earliest problems that I ran into was having more than one property on a class that referenced the same class type. This example was taken from a great site that explored the different EF properties.\n[]{.Apple-style-span style=“background-color: #f8f8f8; color: #2e2e2e; font-family: ‘Courier New’, Courier, mono; font-size: 12px;”}\\\n","title":"EF 4.1 Code First: Many one-to-one properties for same class type on an object.","type":"posts"},{"content":"I have been working on a game where enemy characters must select a target seemingly at random but such that the distribution of selected targets is balanced. Initially I had come up with a randomization algorithm with weighted values for each target but this became cumbersome and much too complex.\nThen I remembered the scheduling algorithms from my Operating Systems class. In particular, the lottery scheduling algorithm.\n\\\n// assume a queue of targets int total = 0; for (Target t : targets) { t.tickets = 5; total += t.tickets; } // generate a random number from 0 to total for (Target t : targets) { randNum -= t.tickets; if (randNum \u0026lt;= 0) { targets.reenqueue(t); return t; } } The queue structure is critical to ensuring proportionality in target selection. Also, targets can be given a higher amount of tickets if bias is needed.\n","date":"2 September 2011","externalUrl":null,"permalink":"/pseudo-random-proportional-scheduling/","section":"Posts","summary":"I have been working on a game where enemy characters must select a target seemingly at random but such that the distribution of selected targets is balanced. Initially I had come up with a randomization algorithm with weighted values for each target but this became cumbersome and much too complex.\nThen I remembered the scheduling algorithms from my Operating Systems class. In particular, the lottery scheduling algorithm.\n\\\n","title":"Pseudo random proportional scheduling in games.","type":"posts"},{"content":"I was tasked recently with writing a parser for TEDS bit stream data. There is an IEEE published paper with an overview of the standard but it doesn't give any examples for implementation. I was finally able to find a paper that gave examples but still had trouble parsing the data. I finally realized there is not only 2 selector bits after the basic TEDS block but selector bits inside the standard template (2 in the case of template 25). Here are my main findings:\n\\\nIt depends on the software interface to get the bit-stream, but bits usually start their indexing from left to right. If you are using a language that assumes right to left bit encoding, the stream indices will need to be reversed (eg. 01011000[2]{.Apple-style-span style=\u0026ldquo;font-size: xx-small;\u0026rdquo;}[ ]{.Apple-style-span style=\u0026ldquo;font-size: xx-small;\u0026quot;}translates to 00011010[2 ]{.Apple-style-span style=\u0026ldquo;font-size: xx-small;\u0026rdquo;}= 26[10]{.Apple-style-span style=\u0026ldquo;font-size: xx-small;\u0026rdquo;}). If you are using an little-endian architecture (eg. x86), then the byte order will need to be converted from big-endian (reverse the byte indices). It is big-endian since the standard is defined for networks and thus has a network byte order. To calculate [ConRelRes ]{.Apple-style-span style=\u0026ldquo;font-family: Arial, Helvetica, sans-serif;\u0026quot;}values, use the formula...\n[\u0026lt;start_value\u0026gt; * [1 + 2 * \u0026lt;tolerance\u0026gt;] ^ \u0026lt;teds_value\u0026gt;]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026rdquo;}\nIn the published paper, you will see an entry like [ConRelRes(5E-7 to 172, +- 0.15%)]{.Apple-style-span style=\u0026ldquo;font-family: Arial, Helvetica, sans-serif;\u0026rdquo;} in the \u0026quot;[Data Type (and Range)]{.Apple-style-span style=\u0026ldquo;font-family: Arial, Helvetica, sans-serif;\u0026rdquo;}\u0026quot; field. 5E-7 is the start value, 0.15 is the tolerance and the TEDS value is whatever value gets parsed from the bit stream. Here are some helper functions to parse TEDS correctly on the .NET platform (x86 systems).\\\nint _streamOffset = 0; private System.Collections.BitArray _GetBits(System.Collections.BitArray bits, int length) { System.Collections.BitArray result = new System.Collections.BitArray(length); for (int index = 0; index \u0026lt;= length - 1; index++) { result[index] = bits[_streamOffset + index]; } return result; } private byte[] _GetBytes(System.Collections.BitArray bits) { int byteCount = Convert.ToInt32(Math.Ceiling(bits.Count / 8)); int padCount = (byteCount * 8) - bits.Count; byte[] bytes = new byte[byteCount]; int byteIndex = 0; int bitIndex = 0; System.Collections.BitArray reverseBits = new System.Collections.BitArray(bits.Count + padCount); //byte align new bit array for (int index = 0; index \u0026lt;= padCount - 1; index++) { reverseBits[index] = false; } //reverse bits to read from right to left for (int index = 0; index \u0026lt;= bits.Length - 1; index++) { reverseBits[padCount + index] = bits[bits.Length - 1 - index]; } //create byte array from byte aligned right to left bits for (int index = 0; index \u0026lt;= reverseBits.Count - 1; index++) { if ((reverseBits[index])) { bytes[byteIndex] = bytes[byteIndex] | Convert.ToByte(1 \u0026lt;\u0026lt; (7 - bitIndex)); } bitIndex += 1; if ((bitIndex == 8)) { bitIndex = 0; byteIndex += 1; } } return bytes; } private bool _GetBoolean(System.Collections.BitArray bits) { bool result = bits[_streamOffset]; _streamOffset += 1; return result; } private char _GetChar(System.Collections.BitArray bits, int length) { char result = BitConverter.ToChar(new byte[] { 0, _GetBytes(_GetBits(bits, length))[0] }, 0); _streamOffset += length; return result; } private int _GetInteger(System.Collections.BitArray bits, int length) { byte[] intBytes = new byte[] { 0, 0, 0, 0 }; byte[] bytes = _GetBytes(_GetBits(bits, length)); int result = 0; //pad out rest of bytes so it is int32 aligned and convert endianess for (int index = 0; index \u0026lt;= bytes.Length - 1; index++) { intBytes[index] = bytes[bytes.Length - 1 - index]; } result = Convert.ToInt32(BitConverter.ToUInt32(intBytes, 0)); _streamOffset += length; return result; } ","date":"17 August 2011","externalUrl":null,"permalink":"/writing-teds-ieee-14514-parser/","section":"Posts","summary":"I was tasked recently with writing a parser for TEDS bit stream data. There is an IEEE published paper with an overview of the standard but it doesn't give any examples for implementation. I was finally able to find a paper that gave examples but still had trouble parsing the data. I finally realized there is not only 2 selector bits after the basic TEDS block but selector bits inside the standard template (2 in the case of template 25). Here are my main findings:\n\\\n","title":"Writing a TEDS (IEEE 1451.4) parser.","type":"posts"},{"content":"I always like when a post gets to the point (especially after searching in Google). In short, if you perform an operation that waits to acquire a lock, the thread doesn't actually stop.\nIn concrete terms of .NET coding, if you call the WaitOne method on a thread, it may still process events that the CLR deems \u0026quot;high priority\u0026quot;. This can be a big problem if the event handlers go into the critical region you are trying to protect.\n\\\nint criticalRegion = 0; private void criticalRegionWait() { AutoResetEvent are = new AutoResetEvent(false); Thread thread = new Thread(() =\u0026gt; { criticalRegion = -1 Debug.WriteLine(criticalRegion); are.Set(); }); thread.SetApartmentState(ApartmentState.STA); thread.IsBackground = true; thread.Start(); } private void criticalRegionUpdateEvent(object sender, EventArgs e) { criticalRegion = 1; } In the above example, if there is a high priority event that has [criticalRegionUpdateEvent ]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026quot;}in its call path a race condition is possible. While [criticalRegionWait ]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026quot;}waits for the thread to finish, the [criticalRegionUpdateEvent ]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026quot;}has a chance of setting [criticalRegion = 1]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026rdquo;} so that [WriteLine ]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026quot;}ends up printing 1 instead of -1.\n","date":"29 July 2011","externalUrl":null,"permalink":"/reentrant-lock-acquires-in-clr/","section":"Posts","summary":"I always like when a post gets to the point (especially after searching in Google). In short, if you perform an operation that waits to acquire a lock, the thread doesn't actually stop.\nIn concrete terms of .NET coding, if you call the WaitOne method on a thread, it may still process events that the CLR deems \"high priority\". This can be a big problem if the event handlers go into the critical region you are trying to protect.\n\\\n","title":"Reentrant lock acquires in CLR.","type":"posts"},{"content":"","date":"22 July 2011","externalUrl":null,"permalink":"/tags/class-namespace/","section":"Tags","summary":"","title":"Class Namespace","type":"tags"},{"content":"When designing classes for outside use (eg. API) or as a library component for multiple applications, readability and easy element (field/function) selection is important. No developer likes hunting around for fields or functions; a legible and hierarchical organization of elements aids quicker turnaround time in code delivery.\nIn some cases, a large class is needed that can possibly contain dozens or hundreds of elements. Naming convention can aid in this organization (ie. using common prefixes in field/function names) but are not inherent to the grammar of languages. Therefore, no semantic advantage can be taken from using these conventions.\nclass Store ItemList clothingMensShirts ItemList clothingMensShoes ... ItemList clothingWomensShirts ItemList clothingWomensShoes ... ItemList hardwareToolsWrenches ... ItemList hardwareMaterialsWood ... ItemList electronicsGamesPCFallout3Copies ... ItemList electronicsAudioHeadphones ...\nThe current solution to the management of large classes is by using publicly exposed nested classes.\nclass Clothing inherits Department ClothingSection mens ...class Mens inherits ClothingSection ItemList shirt ItemList shoes ...abstract class Store ... Department clothing ...class IKEA inherits Store ...IKEA newYorkStore = new IKEA();newYorkStore.buy(newYorkStore.clothing.mens.shirts[0]);\nThis is a possible solution but also comes with it's own problems. Encapsulation of the nested class will not only hide information from the public user but also from the parent class as well. This can be problematic when designing an abstract class where the functions, that must be implemented, must also access non-public elements inside the nested. For example, in the store example, any data the top level class (Store) may want to access from the lower level ones (ClothingSection, ItemList) will be unavailable if that data must also be hidden from the public user.\nAlso, the nested classes cannot be abstracted to require implementation from any inheriting class. Revisting the store example, this means that abstracting the ClothingSection class would require two new classes for one implementation. Creating not only a child Store but also a child ClothingSection class.\nNested classes are helpful for creating logical hierarchies within a class or providing abstract functionality privately inside the class. Although, I would propose the use of namespacing inside a class instead. It would provide the logical hierarchy needed in a very simple way and allow for information sharing across the entire domain of the class. It would need a separate keyword though to prevent use outside of classes.\nclass Storeclassnamespace clothing classnamespace mens ItemList shirts ItemList shoes ... ... classnamespace womens ItemList shirts ItemList shoes ... ... classnamespace hardware classnamespace tools ItemList wrenches ... ... classnamespace materials ItenList Wood ... ... ...\n","date":"22 July 2011","externalUrl":null,"permalink":"/namespace-within-classes/","section":"Posts","summary":"When designing classes for outside use (eg. API) or as a library component for multiple applications, readability and easy element (field/function) selection is important. No developer likes hunting around for fields or functions; a legible and hierarchical organization of elements aids quicker turnaround time in code delivery.\nIn some cases, a large class is needed that can possibly contain dozens or hundreds of elements. Naming convention can aid in this organization (ie. using common prefixes in field/function names) but are not inherent to the grammar of languages. Therefore, no semantic advantage can be taken from using these conventions.\nclass Store ItemList clothingMensShirts ItemList clothingMensShoes ... ItemList clothingWomensShirts ItemList clothingWomensShoes ... ItemList hardwareToolsWrenches ... ItemList hardwareMaterialsWood ... ItemList electronicsGamesPCFallout3Copies ... ItemList electronicsAudioHeadphones ...\nThe current solution to the management of large classes is by using publicly exposed nested classes.\nclass Clothing inherits Department ClothingSection mens ...class Mens inherits ClothingSection ItemList shirt ItemList shoes ...abstract class Store ... Department clothing ...class IKEA inherits Store ...IKEA newYorkStore = new IKEA();newYorkStore.buy(newYorkStore.clothing.mens.shirts[0]);\nThis is a possible solution but also comes with it's own problems. Encapsulation of the nested class will not only hide information from the public user but also from the parent class as well. This can be problematic when designing an abstract class where the functions, that must be implemented, must also access non-public elements inside the nested. For example, in the store example, any data the top level class (Store) may want to access from the lower level ones (ClothingSection, ItemList) will be unavailable if that data must also be hidden from the public user.\nAlso, the nested classes cannot be abstracted to require implementation from any inheriting class. Revisting the store example, this means that abstracting the ClothingSection class would require two new classes for one implementation. Creating not only a child Store but also a child ClothingSection class.\nNested classes are helpful for creating logical hierarchies within a class or providing abstract functionality privately inside the class. Although, I would propose the use of namespacing inside a class instead. It would provide the logical hierarchy needed in a very simple way and allow for information sharing across the entire domain of the class. It would need a separate keyword though to prevent use outside of classes.\nclass Storeclassnamespace clothing classnamespace mens ItemList shirts ItemList shoes ... ... classnamespace womens ItemList shirts ItemList shoes ... ... classnamespace hardware classnamespace tools ItemList wrenches ... ... classnamespace materials ItenList Wood ... ... ...\n","title":"Namespaces for classes.","type":"posts"},{"content":"I made the stupid mistake of importing an Eclipse project into my workspace when the import location was from the workspace itself. I went to delete the recursive folder when I encountered an error that the file path name was too long to delete.\nThe best manual solution that I found was to go as deep as possible in the recursive folder and then map it to a virtual drive.\n[\n]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026rdquo;}\n[subst x: c:\\recursiveFolder\\]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026rdquo;}[recursiveFolder\\]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026rdquo;}[recursiveFolder\\...]{.Apple-style-span style=\u0026ldquo;font-family: \u0026lsquo;Courier New\u0026rsquo;, Courier, monospace;\u0026rdquo;}\n[\n]{.Apple-style-span style=\u0026ldquo;font-family: inherit;\u0026rdquo;}\n[This can be quite an arduous process; I found myself creating up to 9 virtual drives. It eventually worked though and I was able to delete the recursive folder.]{.Apple-style-span style=\u0026ldquo;font-family: inherit;\u0026rdquo;}\n[\n]{.Apple-style-span style=\u0026ldquo;font-family: inherit;\u0026rdquo;}\n[I also found a site with a Perl script that deletes long file paths but haven't tested it myself.]{.Apple-style-span style=\u0026ldquo;font-family: inherit;\u0026rdquo;}\n","date":"19 July 2011","externalUrl":null,"permalink":"/delete-long-file-path-names/","section":"Posts","summary":"I made the stupid mistake of importing an Eclipse project into my workspace when the import location was from the workspace itself. I went to delete the recursive folder when I encountered an error that the file path name was too long to delete.\nThe best manual solution that I found was to go as deep as possible in the recursive folder and then map it to a virtual drive.\n[\n]{.Apple-style-span style=“font-family: ‘Courier New’, Courier, monospace;”}\n[subst x: c:\\recursiveFolder\\]{.Apple-style-span style=“font-family: ‘Courier New’, Courier, monospace;”}[recursiveFolder\\]{.Apple-style-span style=“font-family: ‘Courier New’, Courier, monospace;”}[recursiveFolder\\...]{.Apple-style-span style=“font-family: ‘Courier New’, Courier, monospace;”}\n[\n]{.Apple-style-span style=“font-family: inherit;”}\n[This can be quite an arduous process; I found myself creating up to 9 virtual drives. It eventually worked though and I was able to delete the recursive folder.]{.Apple-style-span style=“font-family: inherit;”}\n[\n]{.Apple-style-span style=“font-family: inherit;”}\n[I also found a site with a Perl script that deletes long file paths but haven't tested it myself.]{.Apple-style-span style=“font-family: inherit;”}\n","title":"Delete long file path names.","type":"posts"},{"content":"","date":"13 July 2011","externalUrl":null,"permalink":"/tags/apache/","section":"Tags","summary":"","title":"Apache","type":"tags"},{"content":"I have been working on implementing a stream reader for IEEE 1451.4 TEDS and came across this interesting paper regarding representation of scientific units in code. It provides solutions to determining unit equivalency (eg. V/(m•kg/s²) = m/(s•A)). Also discovered some indivisible base units that can represent any unit (amp, candela, kelvin, gram, meter, mole, radian, second). With a bit to represent the placement (high for numerator, low for denominator) and a bit for presence, a unit storage type can be derived that only occupies 2 bytes.\n","date":"13 July 2011","externalUrl":null,"permalink":"/representing-scientific-units-in-code/","section":"Posts","summary":"I have been working on implementing a stream reader for IEEE 1451.4 TEDS and came across this interesting paper regarding representation of scientific units in code. It provides solutions to determining unit equivalency (eg. V/(m•kg/s²) = m/(s•A)). Also discovered some indivisible base units that can represent any unit (amp, candela, kelvin, gram, meter, mole, radian, second). With a bit to represent the placement (high for numerator, low for denominator) and a bit for presence, a unit storage type can be derived that only occupies 2 bytes.\n","title":"Representing scientific units in code.","type":"posts"},{"content":"","date":"13 July 2011","externalUrl":null,"permalink":"/tags/scientific-units/","section":"Tags","summary":"","title":"Scientific Units","type":"tags"},{"content":"","date":"13 July 2011","externalUrl":null,"permalink":"/tags/svn/","section":"Tags","summary":"","title":"SVN","type":"tags"},{"content":"I setup a server recently using Ubuntu 10.04. LAMP was easy to configure, as was Subversion. Although making Subversion and Apache2 play nice together with DAV SVN was a nightmare. I followed the guide provided but the website location just kept reprompting for my login.\nAfter hours of running in circles, I was able to find the solution. The Require valid-user directive in /etc/apache2/mods-available/dav_svn.conf required the Limit directive to be around it.\n\\\n\u0026lt;Limit GET POST PUT DELETE CONNECT OPTIONS PATCH PROPFIND PROPPATCH MKCOL COPY MOVE LOCK UNLOCK\u0026gt;\nRequire valid-user\n\u0026lt;/Limit\u0026gt;\nThe LimitExcept directive can also be used as long as the limitations are specified.\n","date":"13 July 2011","externalUrl":null,"permalink":"/using-ubuntu-1004-with-svnhttp/","section":"Posts","summary":"I setup a server recently using Ubuntu 10.04. LAMP was easy to configure, as was Subversion. Although making Subversion and Apache2 play nice together with DAV SVN was a nightmare. I followed the guide provided but the website location just kept reprompting for my login.\nAfter hours of running in circles, I was able to find the solution. The Require valid-user directive in /etc/apache2/mods-available/dav_svn.conf required the Limit directive to be around it.\n\\\n","title":"Using Ubuntu 10.04 with SVN+HTTP.","type":"posts"},{"content":"Senior Software Engineer · Software Architecture · Agentic AI Tooling · Infrastructure · Privacy Engineering\nSanta Cruz, CA (Remote / Hybrid) | awalsh128@gmail.com | linkedin.com/in/andrew-walsh-a235787 | github.com/awalsh128\nSummary # Senior software engineer with 18 years of experience (9 at Google) building reliable, secure, and compliant production systems. Specializes in software architecture, agentic AI developer tooling, CI/CD and code health, and privacy engineering (GDPR, US state law, data residency). Leads cross-functional teams and sets engineering standards. Motivated by technology that puts people first: protecting them, respecting their privacy, and improving their lives.\nCore Skills # Software Architecture: Distributed systems, system and API design, service decomposition, platform integration, design docs and review, scalability and performance, modernizing legacy codebases Agentic AI Tooling: Agent skill and workflow configuration, development automation, failure-mode debugging, performance tuning Productionization: CI/CD (GitHub Actions), test-driven stabilization, code health and security audits, tech-debt remediation, monitoring, requirements gap analysis Privacy \u0026amp; Compliance: GDPR, US state privacy law, data retention enforcement and verification, data classification, auditing, data portability, identity handling, data residency Languages: Python, C++, Java, Kotlin, TypeScript, Go Infrastructure \u0026amp; Delivery: Google Cloud, container orchestration (Borg), Bazel, Git Data \u0026amp; APIs: Spanner, BigQuery, Pub/Sub, SQL, gRPC, Protocol Buffers, REST, data processor orchestration Leadership: Tech lead for cross-functional teams, engineering standards, mentorship, technical interviewing Professional Experience # Lykotek Consulting — Senior Software Engineer # Remote / Bay Area, CA | Sep 2026 – Present\nHands-on senior engineering that hardens and scales production systems for startups with AI-built PoCs\nLeading productionization for OrderlyAI, an AI-powered contract-to-close platform for real estate transaction coordinators, including CI/CD, a code health and security audit, and test-driven stabilization of critical paths. Guiding software architecture, infrastructure, and platform integration; defining coding standards and introducing agentic AI tooling to raise development velocity. Advising on privacy compliance architecture (GDPR, US state law, data residency) and internal data-handling policy, working with founders, investors, and partner consultants. Google — Senior Software Engineer # Bay Area, CA | 2017 – Aug 2026\nFive teams spanning infrastructure, privacy, and trust \u0026amp; safety.\nGoogle-wide Impact # Helped redesign Google\u0026rsquo;s company-wide SWE interview pipeline to assess agentic AI skills. As a Java and Kotlin readability mentor, reviewed 500+ code changes and certified engineers across Google on code quality standards. Interviewed 120+ software engineering candidates for engineering proficiency, team fit, and role fit. Cloud Data Center Design Engineering # 2025 – 2026\nBuilt agentic AI workflows that automated troubleshooting, knowledge discovery, and development tasks, increasing team delivery velocity by 22%. Developed data center modeling features that scaled with the growth of deployments across Google\u0026rsquo;s software-defined network and hardware fleet. Partnered across teams to deliver and support systems throughout the network planning lifecycle. Corporate Engineering Support # 2023 – 2025\nDeveloped internal support systems for Google employees, with a primary focus on UX analytics. As tech lead, built a UX analytics reporting platform measuring user experience across 5 products, integrating multiple backend and frontend technology stacks. Owned requirements, design, and delivery timelines, coordinating work across 4 engineering teams. Corporate Engineering Privacy # 2021 – 2023\nDesigned and built a self-service privacy platform governing third-party tools used across Google. Extended GDPR controls to third-party tools: retention enforcement, auditing, and data classification. Led the project from requirements through launch, working with product, privacy, and legal partners. Google Ads Privacy Engineering — Software Engineer # 2019 – 2021\nBuilt GDPR retention enforcement and verification pipelines for Ads, applied to all Google users globally. Implemented data portability, identity handling, and law-enforcement request compliance. Served as a privacy engineering resource for adjacent teams, raising compliance across products. YouTube Trust \u0026amp; Safety Engineering — Software Engineer # 2017 – 2019\nBuilt and maintained a self-service anti-abuse platform that fraud, copyright, legal, and regulatory compliance teams used to act on all YouTube content. Partnered with policy stakeholders to translate abuse policy into enforcement workflows. Xignite (financial market data APIs) — Senior Software Engineer # San Mateo, CA | 2012 – 2017\nLed engineering for private label offerings serving clients including NASDAQ and CME. Cut platform operating costs by 66% through infrastructure and engineering improvements. Built a white label solution that saved Bloomberg users over 20% on data licensing. Received an award for the BMO Skyway offering. Designed a REST API framework that decoupled front-end and back-end development, shortening release cycles. Vibrant Technology — Software Engineer # Scotts Valley, CA | 2008 – 2012\nPerformed a security audit and redesigned security protocols, reducing the attack surface by 99%. Refactored a 500,000+ line monolithic codebase into a modular object-oriented library system. Architected a machine monitoring system and data acquisition framework that expanded product capabilities. Earlier # Web Design Consultant (2006 – 2008) Senior Systems Analyst, Yum! Brands (2001 – 2006): automated help desk troubleshooting; named Analyst of the Year. References # Open Source # Cache APT Packages Action (maintainer): caches APT packages for GitHub Actions CI pipelines; 10k+ usages. TypeScript APT Library: programmatic APT package management for DevOps automation. Keep Sorted VS Code Extension: native editor support for Google\u0026rsquo;s keep-sorted linter. PCAP-NG Reader (C#, RFC implementation) and Fluent C++ (functional expression library). Education # B.S. Computer Science, University of California, Santa Cruz # 2012\nSumma Cum Laude (3.98 GPA); High Honors, Baskin School of Engineering David S. Lee and Campus Merit Scholarships ","externalUrl":null,"permalink":"/assets/text/resume.html","section":"","summary":"Senior Software Engineer · Software Architecture · Agentic AI Tooling · Infrastructure · Privacy Engineering\nSanta Cruz, CA (Remote / Hybrid) | awalsh128@gmail.com | linkedin.com/in/andrew-walsh-a235787 | github.com/awalsh128\nSummary # Senior software engineer with 18 years of experience (9 at Google) building reliable, secure, and compliant production systems. Specializes in software architecture, agentic AI developer tooling, CI/CD and code health, and privacy engineering (GDPR, US state law, data residency). Leads cross-functional teams and sets engineering standards. Motivated by technology that puts people first: protecting them, respecting their privacy, and improving their lives.\n","title":"Andrew Walsh - Resume","type":"page"}]