Where Does Node.js And Require() Look For Modules?
In the last week or so, I've started to experiment with building server-side Javascript code for Node.js. When you build a Node.js application, you can use the global require() method to load and cache Javascript modules. These modules provide localized memory spaces that can house singletons and class definitions. When you invoke the require() method, Node.js searches a number of system paths looking for the module in question. This morning, I was brought to the edge of sanity trying to figure out what that searching algorithm was. After failing to find anything useful in the Node.js docs regarding the require() method, Jeremy Martin came to my rescue, pointing me to the documentation on Modules.
If you want to load a local, relative Javascript module into a Node.js application, you can simply use the require() method in conjunction with relative (or absolute) file paths:
var moduleA = require( "./module-a.js" );
var moduleB = require( "../../module-b.js" );
var moduleC = require( "/my-library/module-c.js" );
NOTE: The (.js) file extension is optional. Node.js will add it for you when attempting to load the file. It will also attempt to load the given module as a directory with an index file (index.js) or a package description (package.json).
Ok, so that stuff makes sense - relative file paths are something that I can wrap my head around. But what about when no file path is used? Where does Node.js look if you attempt to include a module by name alone:
var moduleA = require( "module-a" );
var moduleB = require( "module-b.js" );
This is where I really started to lose my sanity this morning! When you refer to a Node.js module by name, Node.js uses a complex searching algorithm to locate the requested source code.
First, Node.js looks to see if the given module is a core module - Node.js comes with many modules compiled directly into the executable binary (ex. http, fs, sys, events, path, etc.). These core modules will always take precedence in the loading algorithm.
If the given module is not a core module, Node.js will then begin to search for a directory named, "node_modules". It will start in the current directory (relative to the currently-executing Javascript file in Node) and then work its way up the folder hierarchy, checking each level for a node_modules folder.
Once Node.js finds the node_modules folder, it will then attempt to load the given module either as a (.js) Javascript file or as a named sub-directory. And, if it finds the named sub-directory, it will then attempt to load the file in various ways. So, for example, if you make a request to load the module, "utils":
var utils = require( "utils" );
... Node.js will perform a hierarchical directory search for "node_modules" and "utils" in the following ways:
- ./node_modules/utils.js
- ./node_modules/utils/index.js
- ./node_modules/utils/package.json
If it still can't find the file in this directory-spidering manner, Node.js will then proceed to look at the directory paths outlined in the "require.paths" array. The paths values in this array default to the paths defined, in part, by the environmental variable, NODE_PATH; but, they can be updated programmatically within a Node.js application.
NOTE: From what I have read, programmatically updating the "require.paths" value is considered bad practice.
If I log my default require.paths value to the console, here is what I get:
[
'/usr/local/lib/node',
'/Users/ben/.node_modules',
'/Users/ben/.node_libraries',
'/usr/local/Cellar/node/0.4.5/lib/node'
]
I write all of this in an attempt to hammer this Node.js module search overview into my head. If you want to know more about Node.js's module search algorithm, check out the documentation on Modules. There, you can find a much more in-depth explanation of the search algorithm including module path caching and package descriptions.
Ok - now I can start to breathe a bit easier. I'm serious - I almost lost it a few times trying to get to the bottom of this.
Want to use code from this post? Check out the license.
Reader Comments
hey ben,
while you're exploring node.js, you should consider using Node Package Manager (NPM) (http://npmjs.org/) ... it's a great utility for managing additional modules, and can help remove some of the guesswork.
nice writeup!
@Keegan,
I actually started off this morning by installing NPM - I wanted to play with NowJS. The one-line installer for NPM (curl command) didn't work; had to download and build the make file.
I finally did get it installed and then, subsequently, installed NowJS. But, I couldn't for the life of me figure out the directory structure that NowJS was using; now, could I figure out how it was loading dependencies. That's how I got lost down the module-location algorithm rabbit hole :)
Now, I'm feeling a *little* bit more in control. Thanks!
Thanks for posting this. I spent a good chunk of last weekend trying to figure out where to put my modules and ultimately never found the answer.
NodeJS is still so new that it's not as easy to find answers as it for most other server-side languages.
export NODE_DEBUG=module is your friend...
@martin cleaver THANK YOU :)
there's a node module called "rekiure"
it allows you to "require" without using of relative paths
its a big time saver when it comes to testing/refactoring
https://npmjs.org/package/rekuire
super easy to use
This is helpful but is there any chance you can point me towards a discussion of the same methodology applied to client side includes? It would seem like the same methodology would be followed but it does not seem to be!?!?!
thnkz for the post! i came to kno bout NODE_PATH env variable - set that variable and cld install modules globally now!
Ah thanks Ben - I'm a noob, and after mastering the "Hello World" bit I thought, ok I'm going to write a bit of code, where should I put it? 48 hours later I find this - saved my sanity! - Terry.
This also saved me from losing my mind, so thanks! total noob to nodejs and have spent all day trying to figure out how the hell it wasn't finding any modules. a new folder named node_modules in my projects directory and voila! thanks again!
"If the given module is not a core module, Node.js will then begin to search for a directory named, "node_modules". It will start in the current directory (relative to the currently-executing Javascript file in Node)"
Thanks!
That was the clue for my case. I had to
cp -r node_modules ~/Demo...
This was while in ~/local/node. My index.js was in ~/Demo.
just do a strace or dtruss when running your code you will surely see how the path to your script is resolved.
e.g.
$> strace node myapp.js
Grateful for the post, I was deep in the woods last night over this question
But, but....what is then the point of doing a global install ?
I installed the express module globally and yet I get this error message. A global installation should allow accessing modules from any where.
No ?
Installation command:
sudo npm install -g express
(sudo because -g fiddles with /usr )
Nagesh, the node.js philosophy (which I don't necessarily agree with) is that global install is only for modules that add commands to your system, not for modules that act as libraries for your own projects. For libraries, you are encouraged to install the libraries locally, within each project you develop.
Take the express module. When you install express globally it creates an 'express' command that can be accessed from any shell. (Actually these days you might have to also install the 'express-generator' module as well but that's beside the point of this discussion.) If somebody has created a complete node.js application that you want to add to your system, 'npm install -g' is probably the right way to install it.
I can see where the node.js philosophy comes from. All these open source modules are developing rapidly and may have very specific version dependencies on other modules. Maintaining this on a global basis on your system could result in the kind of problem that's called 'DLL hell' on Windows systems. node.js modules are relatively small and it's easy to keep local copies within each project.
However, coming from a C++ background, where the 3rd party libraries you use can be quite enormous (I'm thinking off Qt as a prime example), the idea of every program I write having its own copy of those libraries is dreadful. When you install something like Qt for development it usually puts its libraries (and DLLs on Windows) in one specific place and you reference -that copy- in your projects.
The people working on node.js -have- provided a 'preferred' way to work with globally installed modules if that's your choice. For example, if you have installed express globally, you can go into the main directory of your project and enter this command:
npm link express
This will create a node_modules directory in your project, but then under there it will create a soft link to the globally installed express module (wherever that happens to be on your system). At this point this even works on Windows, using 'junctions' which is the Windows equivalent of a soft link. You can then write 'require('express')' and it will find the global module through the soft link.