After solving the problem, I created this Node Stream Inheritance sample project with code to share. The project contains sample code showing how to subclass node’s Readable, Writeable, Transform, and Duplex streams as they exist in node v0.12.2.
The node streaming API has a long history. For the purposes of this project, I wanted to make my class support streaming in the simplest way possible without any external dependencies.
Specific node stream inheritance questions
I implemented this project to demonstrate the following:
What event should I wait for to know stream is complete? This depends on the stream you are piping to (the sink). In the case of a FileStream, handle the 'close' event.
What events should I implement (emit) to indicate my writable stream has received everything it can handle? In this case I opted to implement my own 'full' event – indicating that my custom stream had no room for further input.
How do I handle errors?_write and _transform provide callbacks. If you pass your error out, the underlying implementation will automatically emit an ‘error’ event. In the case of _read, you need to use this.emit('error', err);
Here is some sample code from the project.
Duplex stream subclass test
Duplex stream subclass example
If you do want to use a library for a simpler API (that is more consistent across node versions), look into through2 and event-stream. Thanks @collinwat for the code review and recommendations!
Recently, Numenta Founder Jeff Hawkins made a claim that we’d have the technology necessary for “intelligent machines” in just five years. Numenta is working to model the neocortex in the hopes of combining machine learning with self directed action. Wow. I’d love that. But I think most normal people are terrified.
The impressive part was that the algorithm predicted “fox eats rodent” – without having seen the word “fox” before in the input.
The code actually does sort of “know” what a fox is, though. It queries Cortical.io (formerly CEPT) for a Sparse Distributed Representation (SDR) of each word it sees. The SDR is a numerical representation of the word, derived from “reading” wikipedia. Still, this is an impressive generalization – it is effectively understanding that foxes is like other animals, and it knows that those other animals eat rodents, so it appears to guess that foxes must eat rodents.
There is a ton of interesting stuff going on here, including real-time learning that I won’t even attempt to explain. In the videos above, Hawkins explains how this is different from a traditional neural network.
But, in some ways this demo is misleading. It is not showing how the neocortex works (or how the brain reads, interprets, and generalizes between words), it is only showing how the building blocks we’ve got so far can be hacked to do interesting things.
The experiment only shows how an upper layer of the brain might work. This demo (unless I’m misunderstanding) is showing how one layer CLA/OPF magic behaves when fed a sequence of richly derived word meanings (which in a multi-layer model would be learned and stored by a different level of the brain).
What I wanted to test was how robust this prediction is. Did Ahmad just get lucky with his 38 lines of input?
After a couple hours twidling my laptop to get things to run, I did reproduce the result with the same input file. Aside: it is wonderful that Numenta and the hackers have open sourced their code and examples so the rest of us can play with it!
However, I also gave it a bunch more input, and got different – sometimes less or more logical results. With this data set, I get:
I also got “leaves” or “mice” with other variations of the input (I didn’t change much related to animals). It seemed kind of random.
But, I also get these great results (starting with grandma the first time it sees any of these terms in the input file)…
“Release” and “artists” don’t exist anywhere in the input. WTF? To be sure, I’m not training it on the best data set, and it is coming up with reasonable predictions. Here’s the full input and output.
I tried a bunch of much more abstract terms to see if we could use this at Haiku Deck, where we’ve got some interesting ways of guessing the best background image for a slide in your presentation. While the algorithm is clearly learning, it leaves a lot of mental jumping to decide if its predictions are correct.
I have no idea how Numenta is regarded by other AI or neuroscience researchers. But Numenta’s advances in modeling the brain have definitely re-awakened my dormant interest in Artificial Intelligence. Next, I want to try simpler input like images or even X/Y data (bitcoin prices, anyone?).
Another good point that Scott makes: Taking the time to do tests right is hard. In my opinion writing automated tests is necessary for shipping good node code. As a developer, I try to include it in my estimate whenever possible. It’s been hard figuring out how to test all the asynchronous code, but worth it to make sure code works fully before shipping it.
If you’re just getting started with Node, see Scott’s post for some helpful links.
For me, Scott’s #3 (async by default) is the big gotcha. I just don’t think 80% of server code needs to be asynchronous. It’s not worth the complexity it adds to writing and testing your code. I always think code should be simple, not clever, and written for non-geniuses to understand. Computers are cheap, and brain power is expensive (at least for now) – so if you have a server-side function that retrieves data from a database, does a few other things, and then returns it to the client – it doesn’t need to be asynchronous to have sub 100ms response times.
I’ve been playing minecraft to understand what my nephews like about it. The biggest problem is I keep getting stuck in the dark (in a pitch black hole). Either I dig straight down, or night comes, or I fall into a lake.
The key to getting out was to go to the options menu and turn up the brightness. I’m playing using the Mac desktop version of the game. Other articles I found told me to dig stairs to get out, but I couldn’t even see which way was up or down.
Apparently there used to be some kind of /home or /kill command but they don’t work for me in the current version. The only solution was to increase the brightness of the screen and dig stairs (in an upwards diagonal direction) to climb out.
As a developer, your Github profile is quickly becoming your new resume. Hiring companies look at your Github activity as a measure of your skill and experience.
How did I become a contributor to the Ember.js website even though I’ve never used the framework, and without writing a line of code? Here are the 3 easy steps you can follow to become an open source superstar.
Find a prominent open source library. Or, just pay attention the next time you’re following one of those “How to install this” README.md guides.
Read the readme and look for typos, spelling errors, or confusing grammar. Or, follow the instructions for installing something. Find confusing parts of the documentation where you can add more detail.
Submit a pull request! Github makes this extremely easy for text files (like README.MD). Just click “edit” and Github will fork the repo and guide you through creating a pull request. Fix up the documentation, commit it, and submit your pull request. Here’s one I did to make the Ember.js documentation more clear.
The steps above are good for open source in general, and also make you look good. If you want to make yourself look good in a more superficial way, how about keeping a public repo and contributing something daily. Or, write a robot to do it!
Five times now I’ve worked for a startup as it went through a growth phase (major round of funding). When it worked well, each new team member made the team stronger. When it didn’t work, the company was a revolving door. For development teams, it’s a tricky time.
Early developers enjoy high individual productivity as they work with the technology they know (or have pioneered). Adding more developers does not mean an immediate increase in productivity. More team members requires more interaction, planning, and code interfaces.
Developers are a quirky bunch. There are geniuses that come to work when they want (or not at all). There are verbally challenged code generators that turn out code faster than the team can agree what to build. Lone wolves that go off on all nighters and don’t come back with ship-able code. Work-from-homers that need to be “skyped in.” And the loyal guys that do what the boss wants today without finishing what he wanted yesterday. Not to mention the “leader” that rarely takes his headphones off.
For the people I currently work with, don’t worry – I’m not thinking about you ;-).
In this storming, forming, and norming process the team needs to set guidelines on how to work together. I’ve written before about 10x developers – a similar concept applies to productive teams. I’ve never been trained as a manager, but there are a couple things that keep coming up. It is critical to establish a team agreement centered around communication. What kills startups are the things that left unsaid, so nail down a few specifics with a “team agreement” document.
Example agile team agreement
Work on the right things for the business
Increase leverage by improving our skills and using the right tools
Ship code that works
Have unit tests and be able to ship often with confidence
Stand-up meeting at 10AM M-F: 1 minute to report on what you did the previous day, what you are doing doing, and what you are blocked on
If you can’t attend, send in your status update via email
Be available on Skype when you’re working
Sprint planning and process
A weekly sprint to complete top priority items from the backlog
Tasks recorded in Trello (or sticky notes, or whatever works)
Task complexity discussed prior to or during planning
Stick to your assigned tasks during sprint
Any time something gets brought into the sprint, notify the team and create a task to track it
There’s many other things to go into with team-building, but a couple other tangible things keep coming up.
I’ve been working on a web site that is often viewed from iPads. iPads have a retina display (higher pixel density), which makes low-resolution images look grainy. Here’s an example, zoomed in to exaggerate the effect.
The problem is the images are raster (bitmap or pixel) based, not vector (line art) based. Now that I recognize the problem, I see it everywhere. Even the Google search looking glass.
Anti-aliased pixel-based PNGs are the standard. For web developers, the problem is, once they flatten an image to a pixelated PNG, the browser can’t enlarge it again. This is a problem with the widening array of mobile display resolutions.
Unfortunately, a common fix is to swap in a 2x resolution image. This is a dumb approach for icons, logos, and buttons – anything that is not a photograph. All you are doing is downloading a second set of larger files to handle another fixed resolution.
One consideration is the file size. Especially when developing for mobile web pages, for complex icons, a larger-dimensioned PNG may be smaller in file size than a vector representation. For most icons, logos, and buttons, SVG files will be smaller than the PNG. PNGs are actually not that small, since it is a lossless format.
The second consideration is CPU time to render and composite the image (often on top of something else). I haven’t done any performance tests. For maximum performance, scaling and compositing bitmap based icons will be the best. However, my guess is that unless your vector is crazy complicated, or your phone a few years old, using vectors will be fine.
One great trick I wish I knew before converting a batch of PNG icons last month, was that algorithmic conversion (aka image tracing) from pixel to vector is pretty common. There are several services like http://vectormagic.com/home that will automate your conversion. Vector creation is even built in to Adobe Illustrator’s with the “Live Trace” function.