Category Archives: node.js

ReactJS, MongoDB and Blockly to Generate Data Entry Sites

I’ve been pulling together a proof of concept for some ideas I have about dynamic applications.  The gist is that dynamic languages (like JavaScript), combined with reactive designs (ReactJS/Flux) and with “schema-less” databases) allow a level of fluidity which makes generation of applications from metadata much more attractive.

These ideas have been put together into a production app for the company I currently work for.  The code I will share with you obviously will be a tiny subset of what we have done there, but I think it should be enough to prove to you that this approach has merits for specific types of applications.

The code for this blog is here: https://github.com/DaveWelling/brn

Here’s an example of what the metadata might look like:

Blockly Metadata
Blockly Metadata

 

 

 

 

A few terms you will see in the metadata:

Use Case is the application you are building.  It contains namespaces which as you might guess define broad areas of the application in exactly the same way you would use them to define area of a code library.  Relations are, in essence, your objects. Namespaces also map directly to the databases you are storing in MongoDB.  Relations map to document collections in MongoDB.

How it works

Each of the blocks within the “Layout” section correspond to widgets created using React.  The widgets talk to each other using the Flux architecture.  Specifically, the app uses Redux.

The metadata itself is composed using the Blockly tool made by Google.  My understanding is that the Blockly tool is intended to be primarily a way to teach logic used in coding.  The blocks are used to generated code (such as JavaScript).  I’ve used them instead to generate JSON that contains metadata describing my application.

The information composed by these blocks is also used to generate a dynamic server side architecture that provides a REST API and writes to MongoDB.

I’ve tried to compose all these pieces using docker in a way that allows you to ignore the details of how everything is wired together.  The idea is to allow you to focus on what the code is doing.  That said, here are the details so you can dig in when you are ready (or if you need to debug a piece that is failing):

Here is what the above metadata generates:

Generated ReactJS App
Generated ReactJS App

High Level Pieces:

MongoDb (prefab docker container)

Database that is traditionally served on port 27017.   We will serve it from docker as localhost. If you want to change this, you will need to edit the mongoUrl constant in /server/src/config.js to use your own hostname and port

Nginx (prefab docker container)

Used to compose all the sites together on port 80 using reverse proxy.  If you don’t mind allowing CORS and pointing directly at port numbers, you can do without this.

Blockly Front End (your code)

This is where you define the metadata that drives your site.  This site is vanilla JavaScript, but uses some ES6 so you’ll want to run it using a modern browser.  I use Chrome (just regular not Canary).  This code is copied into the Nginx container to be served.

ReactJS App Front End (your code):

This requires ReactJS (obviously) which is not going to work well if you don’t run it through the provided NPM start script/WebPack configuration.  There are some things like ES6 modules and spread operators that aren’t supported in most browsers.  This code is copied into the Nginx container to be served.

Backend Server (your code)

This is your NodeJS REST API server.  It uses HapiJs because there aren’t many API focused frameworks that are simpler and yet (what I hope you’ll consider) feature complete.  This code is put into its own container and the Nginx reverse proxies to it.

Site Arrangement:

  • Blockly Front End: port 1350
  • ReactJS App Front End: port 3000
  • NodeJS Backend Server: port 10437

These are copied to or reverse proxied from nginx so that all are available on port 80 as follows:

  • Blockly Front End: /blockly
  • React App Front End (site root): /
  • Back End Server: /api

DerbyJs or Racer on Windows

You could argue this just isn’t meant to be … and you might be right.  Unfortunately for me, flailing around in my Ubuntu VM is just a slow way for me to develop.  I know that makes me a terrible person, but sometimes the tools you use every day belong to the dark side.

In any case, I set out to get DerbyJs and Racer working on my windows machine.

DerbyJs and Racer are created by pretty much the same group of people.  They are JavaScript frameworks that run on Node.Js and use Operational Transformation to synchronize data in real-time across clients.

The tweaks

The first smack in the face is Redis.  You’ll need to install the windows version of that here, but that’s not remotely the hard part.  The difficulty is when you try to do your npm install for the example repositories of DerbyJs or Racer  .  Then all hell breaks loose, and you run into the “we don’t support windows” contingent with npm install redis.

Turns out the lovely people at hiredis do support windows though.  So here’s the trick(s).  Install hiredis globally FIRST (npm install -g hiredis).  Then go into your global npm cache and copy everything from %appdata%\npm\node_modules\hiredis\build\Release one directory down to %appdata%\npm\node_modules\hiredis\build (because REASONS).

Now npm install type things will magically start working — if you’ve put %appdata%\npm\node_modules into your system environment variables as NODE_PATH.  NPM doesn’t do that on installation because… it’s fun to google?

System Node Environment Variable
How to set your npm cache path

 

Now after all this effort  (at least at the time of this writing)  nothing will work.  You’ve got a couple more tweaks to make.

For Racer

You need to go in and remove all the “release kind of like this one” stuff from your package.json. (See JSON below with red things to remove.)  This is because the current release of Racer will not work with the example code.  No idea why because the problem seems to be somewhere in the Operational Transformation code — which is fiddly stuff.

"dependencies": {
"coffeeify": "~0.6.0",
"express": "~3.4.8",
"handlebars": "^1.3.0",
"livedb-mongo": "~0.4.0",
"racer": "^0.6.0-alpha32",
"racer-browserchannel": "^0.3.0",
"racer-bundle": "~0.1.1",
"redis": "^2.4.2"
}

BTW, below is, I think, my favorite bit of code ever.  When running the “Pad” example of Racer, it is fired over and over by a dependency (called syntax-error) of browserfy:

module.exports = function (src, file) {
if (typeof src !== 'string') src = String(src);
try {
eval('throw "STOP"; (function () { ' + src + '})()');
return;
}
catch (err) {
if (err === 'STOP') return undefined;
if (err.constructor.name !== 'SyntaxError') throw err;
return errorInfo(src, file);
}
};

I’m sure there is a good reason for it, but I can’t fathom it myself.  I had to comment it out.

 Now for DerbyJs

There is some sort of issue with how it is creating the paths for your views.  To get it to work, there is a patch you’ll need to add to your package.json.

After the patch is installed, you’ll need to go into the index.js of each of the examples and add it before the require(‘derby’):

require('derby-patch');
var app = module.exports = require('derby').createApp('hello', __filename);

I think that should do it.  I was able to get things running fairly well after that.

One last thing, if you happen to be using Visual Studio, I can recommend the Node.Js Tools for Visual Studio with some confidence now.  During the beta stage they were pretty bad about crashing my IDE, but (except when running unit tests) I actually prefer them to WebStorm now .  I know… sacrilege.

 

Running cucumber.js in WebStorm

If you are like me, the instructions for running Cucumber.js in WebStorm did not yield good results.  The configuration dialog expects an Executable path, but the instructions on the github page only show the .js file  in the command line execution.  I assumed this meant that I needed to run the command line directly in node, but that didn’t seem to work either.  Ugh.

In a nutshell, you need to specify node as your executable in the configuration dialog and pass your cucumber.js file as an argument.  Like so:

WebStorm Cucumber.js Configuration

Turns out this will occasionally fail when WebStorm inserts an argument between your cucumber.js path and node.  Grrrrrr.

To make it work anyway, create a batch file (cucumber.bat) with the following contents:

CALL “<insert your node path here>\node.exe” <insert your cucumber bin path here>\cucumber.js %

Then change your configuration to look like so:

Cucumber Configuration With Batch

I hope this saves you some pain.

 

Synchronization Using Interval Tree Clocks in JavaScript

As a follow up to my previous post, I’ve implemented the Interval Tree Clock code in JavaScript with tests.  I’ve also begun a synchronization framework to go with it.

Github ITC in Javascript

The framework would be for synchronizing documents in full mesh mode — so peer to peer.

Everything has tests, so you can easily see the direction and progress by just reviewing the tests.

I stab the Synchronization with big knife

Building a CouchApp using Node.js on Windows

In this post, I’d like to help out Windows users who wish to create a Node.js application on a windows box and are interested in a easy alternative to traditional web server hosting. 

I love relational database applications, and have been writing them for (literally) decades now.  Recently I’ve been trying to explore what NoSql has to offer, and I discovered an interesting use of NoSql, that is, hosting. 

CouchDB offers a capability called CouchApp which allows your Couch to host data and act as a web server simultaneously.  Evidently, this does scale surprisingly well, and because of CouchDB’s built in replication capabilities, it offers a pretty flexible solution in a dead simple package.

Quite a few of the people who pioneer and describe this tech are UNIX/Mac users.  I don’t have a UNIX or Mac development machine set up right now, so I thought I’d give it a try on Windows.

Most of the information in this post was adapted from Max Ogden’s video post.

You’ll need the following installs:

You can use NPM to install CouchApp

Create a directory for your application.  I’ve put mine in C:\Temp\MomsTattoos.

If you’ve installed Node.Js in the normal place, you should be able to open up a command prompt in your new directory and type something like this, to get CouchApp installed:

%PROGRAMFILES%\nodejs\npm install couchapp

My Node JS is in the system path, so this is what I did:

image001

Take a look around

You can see your new couch app options by typing: node_modules\.bin\couchapp

image

The boiler option creates a generic version of a CouchApp, but there may be a small problem when you try it.

image

This can be fixed by opening the main.js file in node_modules\.bin\couchapp

and editing line 2 from

, sys = require(‘sys’)

to

, sys = require(‘util’)

image003

If you attempt the boiler command again you will get:

image004

This is because the command actually did generate the app the first time it was used (despite the error).  Your project should now look something like this:

image005

Now you can open up the starting point of your application, App.js.  This file contains code similar to the routing constructs in MS MVC (if you are familiar with that).  Go ahead and change the _id property in the ddoc variable to match your application.  Here’s what mine looks like:

var couchapp = require('couchapp')

  , path = require('path')

  ;

 

ddoc =

  { _id:'_design/MomsTattoos'

  , rewrites :

    [ {from:"/", to:'index.html'}

    , {from:"/api", to:'../../'}

    , {from:"/api/*", to:'../../*'}

    , {from:"/*", to:'*'}

    ]

  }

  ;

 

ddoc.views = {};

 

ddoc.validate_doc_update = function (newDoc, oldDoc, userCtx) {

  if (newDoc._deleted === true && userCtx.roles.indexOf('_admin') === -1) {

    throw "Only admin can delete documents on this database.";

  }

}

 

couchapp.loadAttachments(ddoc, path.join(__dirname, 'attachments'));

 

module.exports = ddoc;

view rawapp.js

Open up Index.html and alter it slightly to show some content in the body.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<html>

  <head>

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

    <meta name="keywords" content="" />

    <meta name="description" content="" />

   

    <link rel="shortcut icon" type="image/x-icon" href="favicon.ico" />

    <link rel="shortcut icon" type="image/png" href="favicon.png" />

   

    <title></title>

    <link href="layout.css" rel="stylesheet" type="text/css" />

    <script language="javascript" type="text/javascript" src="jquery-1.4.4.min.js"></script>

    <script language="javascript" type="text/javascript" src="sammy/sammy.js"></script>

    <script language="javascript" type="text/javascript" src="site.js"></script>

  </head>

 

  <body>

    <h1>Hello World</h1>

  </body>

</html>

view rawIndex.html

You will need a database in the couch to store the application.  Open a browser and go to

http://127.0.0.1:5984/_utils/

Create a database and name it momstattoos (or whatever your app is called).

CreateDbInFuton

Now save yourself some hassle by copying the app files (app.js and attachments folder) one level down (to the first MomsTattoos directory) and delete the inner MomsTattoos directory.

Now your directory tree should look more like this:

DirectoryTreeAfterCombining

Deploy your application

From the MomsTattoos directory, use this command: node_modules\.bin\couchapp push %CD%\app.js http://localhost:5984/momstattoos

PushAppToCouch

Now you’re hosting your first CouchApp. To check it out go to http://127.0.0.1:5984/momstattoos/_design/MomsTattoos/%5Cindex.html.

In Summary

In this post, I pulled together the basics for building a hello world CouchApp on a Windows box.  In my next post, I’ll try to get a more complete application running with the goal of accessing NoSql data from a more complex client application.