Top 6 Gotchas with Node.JS

Node.js has many advantages. Many developers know Javascript at least a little bit. Node offers great performance for applications that wait on I/O (like web services). There’s a vibrant open source community creating reusable node modules for almost every task (and a handy package manager to bundle them together). Node is cross platform – the same code runs on Windows, OSX and Linux. And of course, there’s the promise of re-using Javascript patterns on both the client side and the server side.

I love node, but I’ve run into several gotchas with node in production the last few years. Here are the top 6.

1. The website (server process) is fragile. With node, the process restarts on every unhandled exception. This is different from Ruby on Rails, where you have a separate server process like Unicorn, or PHP where it runs as an Apache module.

With one client that was transitioning from PHP to node, our first deployments felt like we were “plugging the dike” – there were so many server crashes. Until we stabilized things, this sort of gave node a bad name (the perception was that PHP was more stable – but in fact the exceptions were just hidden).

Handling all potential errors, as well as unplanned exceptions is impossible. Most deployments have another process to restart the primary node server process when it bombs. If you’re using a PAAS provider, this will be provided for you, but if you want fine grain control of your server (most web application companies will), hosting and scaling (keeping processes up and enough process around) is a challenge.

Common server-restarting libraries for node are: Upstart, Forever, or God.

2. Writing and maintaining asynchronous code. Callbacks are a simple mechanism starting out, but as your server code grows in complexity, they become a hairball. Crazy callback chains (AKA “callback hell”) make code unreadable. Even simple code that could’ve been procedural is forced to use callbacks due to the libraries you’re using.

Mixing synchronous and asynchronous code makes it difficult to handle every error. Furthermore, you have to understand the event loop to know when execution actually switches contexts. As you’re running your code – if you forget to call a callback, you’ll be left waiting to find out what happened. And, asynchronicity makes even your tests become more complicated.

The Async library exists to handle common asynchronous patterns. Personally, I prefer the “promise” (or “deferred”) pattern. It seems like a hack to send a function as a parameter, instead the promise pattern returns an object representing the state of the asynchronous call instead. jQuery has a promise library for the client side. When is the promise library we selected for a recent node-based web site (comprehensive API and more standards based).

3. Too much choice.  The flip side of all those great NPM packages is that there is no clear winning web framework (think Ruby on Rails) for node – in fact, there are several competing server side frameworks (express, hapi, sails, kraken). Each with their own conventions for:

a) File structure
b) Deploying code
c) Building assets
d) Automated testing
e) ORM

And of course, the best choices for each of these changes monthly. This, combined with #2 is why Node is not yet a great choice for simple human facing web sites.

Of course, it’s not necessary to use node on the server.  On one team we used it to great success for building thousands of lines of Javascript and templates to be hosted inside a Python Django website. The problem of too much choice exists for client side frameworks (and build tools) as for the server side (you’ve heard of Backbone, Angular, Ember). Node (with the help of tools like grunt or gulp) is essential to building a modern front-end, just prepared for the evaluation time and conflicting opinions on frameworks.

Screen Shot 2014-12-04 at 10.55.39 AM

Visualization of NPM Packages


4. Javascript the language. Many developers think they “know” javascript from having used it in client side code. In fact, they haven’t learned about things like prototypal inheritance, function binding (what is this pointing to?), or variable scoping that are unique to Javascript. Even once you’ve chosen Javascript, there are still a lot of decisions to be hammered out amongst your team. Code conventions (4 spaces or two?) Camelcase or snake case? Jshint is a huge help with this. But, be ready for…

5. Functional versus object oriented programming. There are two very distinct paths any Javascript project can take. Functional style favors dependency injection for testability and clear interfaces (that are theoretically more re-usable). Object oriented favors composition and inheritance for more readable code. The best choice is probably a mixture, as both are silly at the extremes. My current team is finding compromises, but please let me know if you’ve got any guidelines for when to be functional and when to be OO.

6. What about Coffeescript? I hate semicolons. Compared to other languages that give white space meaning (Python and Ruby), Javascript feels heavy and hard to read. I love Coffeescript because it lets you write clear (information dense) code while avoiding common Javascript pitfalls (see #4 above). Node supports Coffeescript natively, but unfortunately, transpiling from Coffeescript means you lose a debugger (and sometimes line numbers).

You also lose the benefit of going with a “lowest common denominator” language (everyone knows Javascript, C, or Java). And, you still need to know javascript deeply (#4). Worse, it can create division on your team. Make sure if you switch to coffeescript, you switch all the way.

Thanks to Scott Nonnenberg, author of thehelp-cluster (a solution to #1) for feedback on this post.

Please leave a comment and tell me: what problems are you having with Node.js?



10 habits of 10x developers

Last week, I discovered The Rise of Developeronomics via Brad Feld. As long as I’ve lived, it has always been a great time to be a software developer. The economics keep getting better for us. My favorite quote:

The one absolutely solid place to store your capital today — if you know how to do it –  is in software developers’ wallets. If the world survives looming financial apocalypse dangers at all, this is the one investment that will weather the storms. It doesn’t matter whether you are an individual or a corporation, or what corner of the world you inhabit. You need to find a way to invest in software developers.

The article also introduced me to the concept of a “10x developer,” and linked to a thoughtful explanation.

The 10x effect is the anecdotal observation that great programmers aren’t just a little more productive than average ones (like 15-20%). They tend to be 10 times more productive.

The second blog post is a fascinating analysis of 10x engineering in terms of thrust (high value flow state work) and drag (administrative necessities). All this got me thinking, what do I do that makes me a 10x developer? I quizzed Ben Sharpe, Collin Watson, and Jonathan Locke, author of Coding – On Software Design Process. Here is what we came up with.

1. Only do the work that needs to be done
. Use the agile process. Involve yourself in UX design. Communicate first. Coding might not be the solution. Premature optimization is the root of all evil. Choose the simplest solution that solves the problem.

2. Build on the shoulders of giants
. Use open source frameworks. Use shorthand languages (HAML, Jade, Coffeescript) to go faster. Don’t re-invent the wheel. Leverage package managers for public and private code distribution. Don’t be at the mercy of Central Command (i.e. Microsoft) to fix a bug in a library. And don’t wait for your employer to let you learn it. Learn it on your own, and get a new job that pays twice as much.

3. Know your data structures and algorithms.
Our profession is open to self-taught tradesmen, but you can’t be a 10x developer if you don’t know when to use quick sort, identify an O(n2) routine, or write a recursive function. Be multilingual – so you can see how different frameworks solve the same problems over and over again. Know enough about the plumbing to be able to make informed decisions (how is this Web framework storing the session state? What is a cookie actually).

4. Don’t be afraid of buying tools that save you time. Ben says: “I bought a bitmap font tool yesterday for $50. It easily saved me more than $200 of my time.”

5. Find focus. Don’t keep your email open all day. Same for Twitter, Facebook, HackerNews, and Techememe. Minimize interruptions during productive moments. Close email, turn off chat, put headphones on… whatever it takes to not be interrupted. Tiny hack: I wear headphones at work even though I don’t listen to music because people don’t interrupt me.

6. Refactor early and often. Kill your darlings. Sometimes you have to throw out clever code to do what’s right by the project, but that’s OK. If you have to touch code on an existing project, always leave it better than you found it.

7. Just do it. Think small. Do your weekend project, participate in Startup Weekend. I bought a Mac and worked in a Windows Virtual Machine on .NET projects for a year before I was able to fully leap to Unix and Ruby on Rails.

8. Pick an editor and PWN it. Some 10x devs prefer text editors to IDE’s because they find they learn more without the crutches. In any case, learn your keyboard shortcuts. There’s no faster way to look something up than to know it. When
picking an editor — seriously consider one of the greats (emacs or vim), they’re universal. After that, pick the best/most supported editor on your preferred platform.

Use macros. Write code that writes code. Use TextExpander on the Mac to create shortcuts for entire paragraphs. Use auto-completion in Visual Studio or SublimeText. Use an editor that supports splitting your window into columns or rows. This way you can see your both a unit test and the code it is testing (or a model and view) at the same time.But think before you type. A friend in a large group said that the most productive coder in their group was a paraplegic who used a mouth stick to type. He had to really think things out before he started typing, he only rarely made mistakes.

9. Clearness beats Cleverness. Write your code so that the next person can read it, not so it uses the fewest characters possible. Strive to adhere to the DRY (Don’t repeat your self) principle. Re-use clearly defined objects and libraries. Break work into clean interfaces between components and developers. Break the problem into small simple tasks.
10. Your sub-conscience is a powerful tool. Stepping away for 10 minutes is often all that is required to solve a problem. Treat your self to a diverse life and limit
your coding time. You will be able to work more effectively during your coding time because it will be more enjoyable. Of course, us older developers also know that a minimum effective dose of physical fitness is also a prerequisite to being a 10x developer. Human networking is more important than computer networking. Practice your pong. I often feel the best thing I ever did for my career as a programmer was to step away from the computer and go meet some movers and shakers.
11. Refine your process as a team. One of the biggest things is to really pay attention and to be wide open to criticism and changing and refining your process. You can’t become a 10x anything without that foundation because it will take you too long to get better at it. A wise man once said “A smart man learns from his mistakes, but the truly clever man learns from the mistakes of others.”

84 Quotes from the Four Hour Workweek

The 4-Hour Workweek by Tim Ferriss is a book about entrepreneurship and productivity. Ferriss invites you to re-prioritize your life so you can earn money ultra-efficiently. The book is full of inspirational tips, but also many great quotes. I went through the book and grabbed all the quotes in the order they appear. These are not quotes of the author, but the quotes from other accomplished people that the author cites. I wanted to do this so I could print out a few for my wall, and turn some into tweets for Twitter (if you haven’t heard, good quotes are supposedly one of the best ways to get re-tweeted). Enjoy, and please leave a comment to tell me which is your favorite!

  1. Whenever you find yourself on the side of the majority, it is time to pause and reflect. – Mark Twain
  2. Anyone who lives within their means suffers from a lack of imagination. – Oscar Wilde
  3. An expert is a person who has made all the mistakes that can be made in a very narrow field. – Niels Bohr, Danish physicist and Nobel prize winner
  4. Ordinarily he was insane but he had lucid moments when he was merely stupid. – Heinrich Heine, German critic and poet.
  5. Reality is merely an illusion, albeit a very persistent one. – Albert Einstein
  6. These individuals have riches just as we say that we “have a fever,” when really the fever has us. – Seneca (4 B.C – 65 A.D.)
  7. I also have in mind that seemingly wealthy, but most terribly impoverished class of all, who have accumulated dross, but know not how to use it, or get rid of it, and thus have forged their own golden or silver fetters. – Henry David Thoreau (1817-1862)
  8. I can’t give you a surefire formula for success, but I can give you a formula for failure: try to please everybody all the time. – Herbert Bayard Swope, American editor and journalist; first recipient of the Pulitzer Prize
  9. Everything popular is wrong. – Oscar Wilde, the importance of being Earnest
  10. Many a false step was made by standing still. – Fortune cookie
  11. Named must your fear be before banish it you can. – Yoda, from Star Wars: the Empire Strikes Back
  12. Action may not always bring happiness, but there is no happiness without action. – Benjamin Disraeli, former British Prime Minister
  13. Set aside a certain number of days, during which you shall be content with the scantiest and cheapest fare, with coarse and rough dress, saying to yourself the while: “Is this the condition that I feared?” – Seneca
  14. There is no difference between a pessimist who says, “oh, it’s hopeless, so don’t bother doing anything,” and an optimist who says, “don’t bother doing anything, it’s going to turn out fine anyway.” Either way, nothing happens. – Yvon Choinard, founder of Patagonia
  15. I am an old man and have known a great many troubles, but most of them never happened. – Mark Twain
  16. “Would you tell me please, which way I ought to go from here?” “That depends a good deal on where you want to get to,” said the cat. “I don’t much care where…” Said Alice. “Then it doesn’t matter which way you go,” said the cat. – Lewis Carroll, Alice in Wonderland
  17. The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. – George Bernard Shaw, Maxims for Revolutionists
  18. The existential vacuum manifests itself mainly in the state of boredom. – Victor Frankl Auschwitz survivor and founder of Logotherapy, Man’s Search for Meaning
  19. One does not accumulate but eliminate. It is not daily increase but daily decrease. The height of cultivation always runs to simplicity. – Bruce Lee
  20. Perfection is not when there is no more to add, but no more to take away. – Antoine de Saint-Exupery, where international postal flight and author of Le Petit Prince (The Little Prince)
  21. It is vain to do with more what can be done with less. – William of Occam, (1300-1350), originator of “Occam’s Razor”
  22. What gets measured gets managed. – Peter Drucker, management theorist, author of 31 books, recipient of Presidential Medal of Freedom
  23. I saw a bank that said “24 hour banking,” but I don’t have that much time. – Steven Wright, comedian
  24. Love of bustle is not industry. – Seneca
  25. We create stress for ourselves because you feel like you have to do it. You have to. I don’t feel that anymore. – Oprah Winfrey
  26. What information consumes is rather obvious: it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention and the need to allocate that attention efficiently among the overabundance of information sources that might consume it. – Herbert Simon, recipient of the Nobel Memorial prize in economics and the A.M. Turing award, “Nobel Prize of computer science”
  27. Reading, after a certain age, diverts the mind too much from its creative pursuits. Any man who reads too much and uses his own brain too little falls into lazy habits of thinking. – Albert Einstein
  28. There are many things of which a wise man might wish to be ignorant. – Ralph Waldo Emerson
  29. Learning to ignore things is one of the great paths to inner peace. – Robert J. Sawyer, Calculating God
  30. Do your own thinking independently. Be the chess player, not the chess piece. – Ralph Charell
  31. Meetings are an addictive, highly self-indulgent activity that corporations and other organizations habitually engage in only because they cannot actually masturbate. – Dave Barry, Pulitzer prize-winning American humorist
  32. The best defense is a good offense. – Dan Gable, Olympic gold medalist in wrestling and the most successful coach in history; personal record 299-6-3, with 182 pins
  33. A schedule defends from chaos and whim. – Annie Dillard, winner of the Pulitzer Prize in nonfiction, 1975
  34. People think it must be fun to be a super genius, but they don’t realize how hard it is to put up with all the idiots in the world. – Calvin, from Calvin and Hobbes
  35. Scotty: she’s all yours, sir. All systems automated and ready. A chimpanzee and two trainees could run her! Captain Kirk: thank you, Mr. Scott. I’ll try not to take that personally. – Star Trek
  36. A man is rich in proportion to the number of things he can afford to let alone. – Henry David Thoreau, naturalist
  37. The future is here. It’s just not widely distributed yet. – William Gibson, author of Neuromancer; termed “cyberspace” in 1984
  38. Nobody can give you freedom. Nobody can give you equality or justice or anything. If you’re a man, you take it. – Malcolm X., Malcolm X. Speaks
  39. The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applies to an inefficient operation will magnify the inefficiency. – Bill Gates
  40. I am not interested in picking up crumbs of compassion thrown from the table of someone who considers himself a master. I want the full menu of rights. – Bishop Desmond Tutu, South African cleric and activist
  41. Just set it and forget it! – Ron Popeil, founder of Ronco; responsible for more than $1 billion in sales of rotisserie chicken roasters.
  42. As to methods there may be a million and then some, but principles are few. The man who grasps principles can successfully select his own methods. The man who tries methods, ignoring principles, is sure to have trouble. – Ralph Waldo Emerson
  43. When I was younger… I [didn’t] want to be pigeonholed. Basically, now you want to be pigeonholed. It’s your niche. – Joan Chen, actress; appeared in The last Emperor and Twin Peaks
  44. Some people are just into lavish dwarf entertainment. – Danny Black (4’2″) part owner of
  45. Genius is only a superior power of seeing. – John Ruskin, famed art and social critic
  46. I not only use all the dreams that I have, but all that I can borrow. – Woodrow Wilson
  47. Creation is a better means of self-expression and possession; it is through creating, not possessing, that life is revealed. – Vida D. Scudder, The Life of the Spirit in the Modern English Poets
  48. Many of these theories have been killed off only when some decisive experience exposed their incorrectness… Thus the yeoman work in any science… Is done by the experimentalist, who must keep the theoreticians honest. – Michio Kaku, theoretical physicist and cocreator of String Field Theory, Hyperspace
  49. The factory of the future will have only two employees, a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment. – Warren G. Bennis, University of Southern California professor of business administration; adviser to Ronald Reagan and John F. Kennedy
  50. The power of hiding ourselves from one another is mercifully given, more men are wild beasts and would never devour one another but for this protection. – Henry Ward Beecher, US abolitionist and clergyman “Proverbs from Plymouth Pulpit”
  51. A company is stronger if it is bound by love rather than by fear…. If the employees come first, then they’re happy. – Herb Kelleher, cofounder of Southwest airlines
  52. Look, kiddie. I built this business by being a bastard. I run it be by being a bastard. I’ll always be a bastard, and don’t you ever try to change me. – Charles Revson, founder of Revlon, to a senior executive within his company
  53. Orders are nobody can see the great Oz! Not nobody, not nohow! – Guardian of the emerald city gates, the Wizard of Oz
  54. The system is the solution. – AT&T
  55. Companies go out of business when they make the wrong decisions or, just as important, make too many decisions. The latter creates complexity.- Mike Maples, cofounder of Motive Communications (IPO to $260 million market cap), founding executive of Tivoli (sold to IBM for $750 million), and investor in companies such as
  56. By working faithfully eight hours a day, you may eventually get to be a boss and work 12 hours a day. – Robert Frost, American poet and winner of four Pulitzer prizes
  57. On this path, it is only the first step that counts. – St. Jean-Baptiste-Marie Vianney, Catholic saint, “Cure d’Ars”
  58. I was asked if I was going to fire an employee w forho made a mistake that cost the company $600,000. Now, I replied, I just spent $600,000 training him. – Thomas J. Watson, founder of IBM
  59. Liberty means responsibility. That is why most men dread it. – George Bernard Shaw
  60. All courses of action are risky, so prudence is not in avoiding danger (it’s impossible), but calculating risk and acting decisively. Make mistakes of ambition and not mistakes of sloth. Develop the strength to do bold things, not the strength to suffer . – Niccolo Machiavelli, The Prince
  61. If you must play, decide on three things at the start: the rules of the game, the stakes, and the quitting time. – Chinese proverb
  62. The average man is a conformist, accepting miseries and disasters the stoicism of a cow standing in the rain. – Colin Wilson, British author of the Outsider; New Existentialist
  63. Only those who are asleep make no mistakes.- Ingvar Kamprad, founder of IKEA, worlds largest furniture brand
  64. Before the development of tourism, travel was conceived to be like study, and its fruits were considered to be the adornment of the mind and the formation of the judgment.- Paul Fussel, Abroad
  65. The simple willingness to improvise is more vital, in the long run, then research.- Rolf Potts, Vagabonding
  66. There is more to life than increasing its speed.- Mohandas Gandhi
  67. Thanks to the interstate highway system, it is now possible to travel from coast to coast without seeing anything. – Charles Kuralt, CBS news reporter
  68. This is the very perfection of a man to find out his own imperfection. – St. Augustine (354 AD – 430 AD)
  69. Traveling is the ruler of all happiness! There’s no looking at a building here after seeing Italy. – Fanny Burney (1752-1840), English novelist
  70. It is fatal to know too much at the outcome: boredom comes as quickly to the traveler who knows his route as the novelist who is over certain of his plot. – Paul Thoreau, To the Ends of the Earth
  71. To being grossed by something outside ourselves as a powerful antidote for the rational mind, the mind that so frequently has its head up its own ass. – Anne Lamott, Bird by Bird
  72. There is not enough time to do all the nothing we want to do. – Bill Watterson, creator of the Calvin and Hobbes cartoon strip
  73. Man I was so made that he can only find relaxation from one kind of labor by taking up another. – Anatole France, author of The Crime of Sylvestre Bonnard
  74. People say that what we’re all seeking is a meaning for life. I don’t think that’s what we’re really seeking. I think that what we’re seeking is an experience of being alive. – Joseph Campbell, The Power of Myth
  75. What man actually needs is not a tensionless state but rather the striving and struggling for a worthwhile goal, a freely chosen task. – Viktor E. Frankl, Holocaust survivor; author of Man’s Search for Meaning
  76. Americans who travel abroad for the first time are often shocked to discover that, despite all the progress that has been made in the last 30 years, many people still speak in foreign languages. – Dave Barry
  77. Morality is simply the attitude we adopt towards people we personally dislike. – Oscar Wilde
  78. Adults are always asking kids what they want to be when they grow up because they are looking for ideas. – Paula Poundstone
  79. The miracle is not to walk on water. The miracle is to walk on the green earth, dwelling deeply in the present moment and feeling truly alive. – Thich Nhat Hanh
  80. If you don’t make mistakes, you’re not working on hard enough problems. And that’s a big mistake. – Frank Wilczek, 2004 Nobel Prize winner in physics
  81. Ho imparato che niente e impossibile, e anche che quasi niente e facile…(I’ve learned that nothing is impossible, and that almost nothing is easy…) – Articolo 31 (Italian rap group), “Un Urlo”
  82. There is nothing that the busy man is less busy with then living; there is nothing harder to learn. – Seneca
  83. For the past 33 years, I have looked in the mirror every morning and ask myself: “if today was the last day of my life, would I want to do what I am about to do today?” And whenever the answer has been “No” for too many days in a row, I know I need to change something… Almost everything–all external expectations, all pride, all fear of embarrassment or failure–these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. – Steve Jobs, college dropout and CEO of Apple Computer, Stanford University commencement, 2005
  84. The hypocrite is a person who–but who isn’t? – Don Marquis