Nested paths as an object instead of strings with slashes

[daedalus28]

Great work on this - I'm integrating this now into a project and it's going to save a ton of time over the manually created files we had before. With that said, it would be great if nested paths could be represented as an object instead of a string with slashes (behind a configuration flag or otherwise).

Right now the output looks something like this
{ 'a/b/c': arguments[0], ... }

I'd like to be able to support behavior like the excellent node module include-all with something like this:

{ a: { b: { c: arguments[0] } }, ... }

For now, I am just doing something like this to make it happen when I require in anything generated by this tool:

var objectifyPaths = function(pathsObject) {
    return _.reduce(pathsObject, function(result, value, key) {
        return _.set(result, key.replace(/\//g, '.'), value);
    }, {});
}
var all = require('./generatedFromMetagen');
return objectifyPaths(all);

Of course, that assumes lodash as a dependency so I understand why you wouldn't want that in the generated output - that's why it'd be great if this tool could generate this kind of output without needing something at run time. If that's too complex, I'd at least like to see an extension point where it's either really easy for me to either swap out the template text or just be able to post process the generated js file. I was going to add support myself, but since your arguments can come from either the parameters object or as positional arguments, I wasn't sure how that would impact how you use this library.

[ORESoftware]

Thanks for this feature request. Let me think about it.
For now, it looks like you have it right - you can manually map any '/' character to a nested object.

What I did in my project, was simply to refer to objects using:

const foo = x['a/b/c'];

I think it would be feasible to do what you're talking about, so I will get to this soon. This project also needs a code facelift. Glad you got it working and let me know if you have any other questions.

[ORESoftware]

by the way, you may also be interested in this article I wrote:

https://medium.com/@the1mills/hot-reloading-with-react-requirejs-7b2aa6cb06e1

One thing I discovered was that using this module (requirejs-metagen) did not always jive super well with "multiple-entry-point SPAs" and with the requirejs optimizer (r.js). The reason is that the requirejs optimizer will utilize bundles, and those bundles will most likely be a group of .js/.css files, but that group of files will not correspond to the grouping of files produced by this project. This project was originally designed with grouping all your views together, all your models together, etc. But this, again, does not jive with the best way to use RequireJS. But let me know if you find it useful and how!

[daedalus28]

I have a marginally better approach now - I'm doing string replacements on the generated files so I don't need the files that reference it to be aware of the function, but it would still be good to have this built and (and also drop this function).

For reference this is a slightly modified snippet from my gulp file (which assumes things are in /public). I'd love to see this cleaned up (any suggestions you have are most welcome!).

// Metagen
var _       = require('lodash'),
    gulp    = require('gulp'),
    Promise = require('bluebird'),
    rmg     = Promise.promisify(require('requirejs-metagen')),
    fs      = Promise.promisifyAll(require('fs'));

var metagenPaths = [/*PATHS GO HERE*/];

function metagen(path) {
    var output = __dirname + '/public/' + path + '/all.js';
    console.log('prepping to metgen path: ', output);
    var controllerOpts =  {
        inputFolder: __dirname + '/public/' + path,
        appendThisToDependencies: '../',
        appendThisToReturnedItems: '',
        eliminateSharedFolder: true,
        output: output
    };
    // Postprocess output and wrap in objectifyPaths until https://github.com/ORESoftware/requirejs-metagen/issues/1
    return rmg(controllerOpts).then(function() {
        return fs.readFileAsync(output, 'utf-8');
    }).then(function(data) {
        var newValue = data
                .replace(/'\n    ]/, "',\n        'objectifyPaths'\n    ]")
                .replace(/return/, 'return arguments[arguments.length - 1](')
                .replace(/}/, '})');
        return fs.writeFileAsync(output, newValue, 'utf-8');
    });
}
gulp.task('metagen', function() {
    return Promise.map(metagenPaths, metagen);
});
gulp.task('metagen-watch', function() {
    var paths = _.map(metagenPaths, path => __dirname + '/public/' +  path);
    // Needs to watch for new files added and removed
    require('chokidar').watch(paths)
        .on('add', x => gulp.start('metagen'))
        .on('unlink', x => gulp.start('metagen'));
});

[daedalus28]

Also as a side note to answer your question on how this is useful for me - my project is a single entry point SPA and there are a few cases where I've effectively got a JS implementation of the strategy pattern where I can pull out each entry to its own file. I've also found it useful for framework "registry" type things where I need to expose a bunch of things but they all need to be post processed in certain ways - I can use this to require them all in and not need to have registry related code on each one, keeping the individual definitions as lean as possible. We're not doing any hot reloading yet (and aren't using react), but your article looks really interesting and helpful. Definitely a good read 👍

[daedalus28]

So it looks like the ability to customize the template is really going to be important here. Apparently, when requiring in one of the generated amd modules with amd-load in node (for unit tests), it doesn't work properly - but does if I change the format to be the commonjs sugar syntax:

This does not work

define(['path/to/example/thing/', 'objectifyPaths'], function() {
    return arguments[arguments.length-1]({
        'example/thing': arguments[0],
        ....
    })
})

This does:

define(function(require) {
    var objectify = require('objectifyPaths')
    return objectify({
        'example/thing': require('path/to/example/thing/'),
        ...
    })
})

They are equivalent and both valid in requirejs but I need the commonjs sugar syntax output - while this could be yet another option (e.g., and outputFormat flag), I think it might be best to just be able to take over the output formatting.

[the1mills]

is "objectifypaths" an NPM module? If not, could you publish yours?

[daedalus28]

objectifyPaths is just a local function I threw together and put in a local AMD module just so that my post processing gulp script wouldn't have to put the function definition inside every generated all file. I pasted it in my first comment but here it is again for reference:

var objectifyPaths = function(pathsObject) {
    return _.reduce(pathsObject, function(result, value, key) {
        return _.set(result, key.replace(/\//g, '.'), value);
    }, {});
}

Note that it does have a dependency on lodash.

[the1mills]

that's cool, I will use that, requirejs-metagen already has a lodash dep

[daedalus28]

Just FYI - I was able to meet my use case by doing it without this library in fewer lines of code than it took to simply invoke it before:

I'd still like to use this library, so hopefully this much more compact implementation helps inspire a clean up of this library :) Plus, this doesn't support the native object style includes yet despite being in the commonjs syntax:

var gulp        = require('gulp'),
    Promise     = require('bluebird'),
    recursive   = Promise.promisify(require('recursive-readdir')),
    fs          = Promise.promisifyAll(require('fs'));
var filesRelative = dir => recursive(dir).then(x => x.map(x => x.slice(dir.length)));

var metagen = function(path) {
    return filesRelative(path).then(function(files) {
        var lines = files.map(function(file) {
            return "'" + file.split('.')[0] + "': require('./" + file + "')";
        });
        var result =
`define(function(require) {
    var objectify = require('objectifyPaths');
    return objectify({
        ${ lines.join(',\n        ') }
    })
})`;
        return fs.writeFileAsync(path + 'all.js', result, 'utf-8');
    });
};

var paths = [/*PATHS*/];
gulp.task('metagen', function() {
    return Promise.map(paths, metagen);
});
gulp.task('metagen-watch', function() {
    // Needs to watch for new files added and removed
    require('chokidar').watch(paths)
        .on('add', x => gulp.start('metagen'))
        .on('unlink', x => gulp.start('metagen'));
});

[daedalus28]

So I've generalized this a bit to have a concept of format functions which are mostly template literals, and solved all the use cases I have - including a both nested and non nested CommonJS sugar syntax and traditional AMD. It's much more flexible because you can just use a custom formatting function and get something totally different (I've even played with a markdown formatter which just lists the directory tree for readme generation). The code base is really tiny and I'll happily submit this as a PR, but it shares almost nothing with the current implementation. I'll paste the source here - let me know how you'd like to proceed. The bulk of the code is the individual formatters - the core is only a couple of lines.

var _           = require('lodash/fp'),
    Promise     = require('bluebird'),
    readDir     = Promise.promisify(require('recursive-readdir')),
    fs          = Promise.promisifyAll(require('fs'));

// Path Utils
var filesRelative = (dir, ex) => readDir(dir, ex).map(x => x.slice(dir.length));
var noExt = file => file.slice(0, _.lastIndexOf('.', file));

// Core
var metagen = dir => filesRelative(dir.path, dir.exclusions || [dir.output || 'all.js']).then(files =>
    fs.writeFileAsync(dir.path + (dir.output || 'all.js'), dir.format(files, dir))
);

// Formats
metagen.formats = {};
metagen.formats.commonJS = files => `define(function(require) {
    return {
        ${ files.map(file => `'${ noExt(file) }': require('./${ file }')`).join(',\n        ') }
    };
});`;
metagen.formats.amd = files => `define([
    ${ files.map(file => `'${ noExt(file) }'`).join(',\n    ') }
], function() {
    return {
        ${ files.map((file, i) => `'${ noExt(file) }': arguments[${ i }]`).join(',\n        ') }
    }
});`;

// use FP when https://github.com/lodash/lodash/pull/2503 is released
var zipObjectDeep = require('lodash/zipObjectDeep');

// Deep Formats
var deepKeys    = _.map(_.flow(noExt, _.replace('/', '.'))),
    stringify   = x => JSON.stringify(x, null, 4),
    indent      = _.replace(/\n/g, '\n    '),
    unquote     = _.replace(/"/g, ''),
    deepify     = _.flow(zipObjectDeep, stringify, indent, unquote);

metagen.formats.deepCommonJS = files => `define(function(require) {
    return ${ deepify(deepKeys(files), files.map(file => `require('${ noExt(file) }')`))};
});`;
metagen.formats.deepAMD = files => `define([
    ${ files.map(file => `'${ noExt(file) }'`).join(',\n    ') }
], function() {
    return ${ deepify(deepKeys(files), files.map((file, i) => `arguments[${ i }]`))};
});`;

module.exports = metagen;

These are my only npm dependencies:

"dependencies": {
    "bluebird": "^3.4.1",
    "lodash": "^4.13.1",
    "recursive-readdir": "^2.0.0"
  }

Here it is in action in my gulpfile. I've changed the API a bit as it now supports file exclusions, but it'd be trivial to build a wrapper:

var metagen = require('./metagen');
var metagenPaths = [{
    path: __dirname + '/public/someDir/',
    // exclusions: ['all.js'],
    format: metagen.formats.deepCommonJS
    //output: '__generated-all.js' // relative to path
}];
gulp.task('metagen', x => Promise.map(metagenPaths, metagen));
gulp.task('metagen-watch', function() {
    // Watch for files added and removed
    require('chokidar').watch(_.map(metagenPaths, 'path'))
        .on('add', x => gulp.start('metagen'))
        .on('unlink', x => gulp.start('metagen'));
});

Let me know how you want me to proceed - either as PR, fork, or fresh library.

[daedalus28]

FYI, I have pushed this up into its own repo since it can now be much more than just requirejs: https://github.com/smartprocure/directory-metagen

I'd like to keep these two libraries together - if I make a quick formatter that adheres to your api, would you accept a pull request that leverages this code?

[ORESoftware]

yeah definitely open to it, let me take a look when I have some energy, as you can see from my commit history on my page, I have been taking a step back from coding for the moment :)

[ORESoftware]

@daedalus28 I haven't really done anything yet, but if you want I can just link to your lib from the top of the readme for this project. As long as you think yours is well-tested enough, etc!

[daedalus28]

Sounds good to me 👍 The new library is currently being used in production in a number of different places and it is working great. If you (or any of your users) find anything missing in the new implementation, just open an issue and I'll happily bring it up to par.

[the1mills]

Glad you liked the metagen name - for me it just stands for metadata-generator :)