nodejs Archives - MAD Blog

Kindle Universal Dashboard

Kindle is Super awesome !

Because of its e-ink display. I wanted a display to present data which has – Least power consumption, Not painful to eyes & obviously one which doesn’t emit blue light.
E-Ink displays fit perfectly to my requirement – acquiring a display which can be driven using Raspberry Pi or Arduino is hard, size/cost ratio is much higher.

On googling I found some display modules which were more or near to 70-80$ – even smaller display – which impulsed me to get a Kindle Touch (8th Generation) at around 74$ approx.

Kindle comes with an experimental browser but it is narrowed version of WebKit, which is pretty useful if you want to display static content or just want to make a control panel, it can easily render normal websites which use js/CSS & works pretty well. But support for HTML5 is almost absent – so you can’t use WebSockets to push data, using long polling/fast polling is only solutions so far.

Moreover, there was another problem which I had to fix – Kindle has fixed timeout which sends it to sleep mode – for mine it was 10min, after digging I found you can use ~ds or similar but for me, nothing worked.

We can only hope that support to remove timeout or change timeout period will be added in future releases. I think old kindle supports.

If you can’t change timeout or you want to use few other features I suggest you to jailbreak. Follow steps mentioned here http://www.mobileread.com/forums/showthread.php?t=275877 , Don’t jump, It works for kindle touch 8th generation. Tried, tested, working ! For KT3 you will need to install MRPI along with KUAL. Once done your kindle is officially out of warranty 😀 . Post that you need to install USBNet – it will allow you to ssh to your kindle. All this will allow you to execute “lipc-set-prop com.lab126.powerd preventScreenSaver 1″ this on the kindle. It will simply disable screensaver. 🙂

Once you have your kindle whose screen doesn’t lock you can simply go & execute a simple nodejs script to push data.

Note : Kindle doesn’t support WebSocket & none of transport methods in socket.io except “polling”.

Below is nodejs server code :

var http = require('http'),
	fs = require('fs');
	//sanitize = require('validator').sanitize;
	
var app = http.createServer(function (request, response) {
	fs.readFile("client.html", 'utf-8', function (error, data) {
		response.writeHead(200, {'Content-Type': 'text/html'});
		response.write(data);
		response.end();
	});
}).listen(1337);

var io = require('socket.io')({
    "polling duration": 60,
    'log level' :1
}).listen(app);

io.set('transports', [ 'polling']);

io.sockets.on('connection', function(socket) { 
	socket.on('message_to_server', function(data) { 
		var escaped_message = (data["message"]);
		io.sockets.emit("message_to_client",{ message: escaped_message }); 
	});
});

var http = require('http'),

fs = require('fs');

//sanitize = require('validator').sanitize;

var app = http.createServer(function (request, response) {

fs.readFile("client.html", 'utf-8', function (error, data) {

response.writeHead(200, {'Content-Type': 'text/html'});

response.write(data);

response.end();

});

}).listen(1337);

var io = require('socket.io')({

"polling duration": 60,

'log level' :1

}).listen(app);

io.set('transports', [ 'polling']);

io.sockets.on('connection', function(socket) {

socket.on('message_to_server', function(data) {

var escaped_message = (data["message"]);

io.sockets.emit("message_to_client",{ message: escaped_message });

});

Below is code of client.html

<!DOCTYPE html>
<html>
	<head>
		<script src="/socket.io/socket.io.js"></script>
		<script type="text/javascript">
			var socketio = io.connect();

			socketio.on("message_to_client", function(data) {
				document.getElementById("chatlog").innerHTML = ("<hr/>" + data['message'] + document.getElementById("chatlog").innerHTML);
			});

			function sendMessage() {
				var msg = document.getElementById("message_input").value;
				socketio.emit("message_to_server", { message : msg});
			}
		</script>
	</head>
	<body>
		<input type="text" id="message_input"/>
		<button onclick="sendMessage()">send</button>
		<div id="chatlog"></div>
	</body>
</html>

<!DOCTYPE html>

<html>

<head>

var socketio = io.connect();

socketio.on("message_to_client", function(data) {

document.getElementById("chatlog").innerHTML = ("<hr/>" + data['message'] + document.getElementById("chatlog").innerHTML);

});

function sendMessage() {

var msg = document.getElementById("message_input").value;

socketio.emit("message_to_server", { message : msg});

}

</script>

</head>

<body>

</body>

</html>

Voila – It works !

Below is video of working in case you want to see demo before getting hands dirty 🙂

Warning & Update : This method might consume more power than expected, as experimental browser has loading status – which continuously refreshes a section of the screen. To overcome this problem I will be polling server with long interval difference – which will be adjustable by the server.

Note for nerds : Since this method uses browser – it’s more flexible – but if you are possessive about power consumption & screen space – You can use JAVA to develop kindlet application. Lightweight pub/sub protocols like MQTT should help you in the way.

Designing a wall holder : You can google for covers or design own or use some double sided foam tape. Since i had access to 3d printer i got two of http://www.thingiverse.com/thing:832273 printed & hung it on the wall – it just helped me in reading few books apart from using it only as display. SWAP!

Use it as :

Scoreboard
Notification
Weather system
Wallpaper slideshow
News/RSS feeds display
Home automation control
Anything
Book reader 🙂

At the end, even if you place it behind your monitor it won’t hurt or push the new data to your eyes & spoil the code castle you were building.

NodeJs – Improving performance using native binding

This is known fact that javascript is really slow in terms of sync operations like multiplications, divisions etc. – Since nodejs is based on js – Node inherits the curse.
Check http://benchmarksgame.alioth.debian.org/u64/compare.php?lang=v8&lang2=gpp this article to cross check the fact. Nodejs is fast in some aspects. I love it 😛

Doesn’t matter how much you improve your algorithm but c++ will outperform if you are running a sophisticated algorithm but one can push through imprecation.
Nodejs is based on V8 (Most ports), v8 based on C++, You must be already aware of that you can use C++ in nodejs – But it seems complex.

But it is not – It is easy to build a native extension & compile in seconds https://medium.com/@devlucky/how-to-get-a-performance-boost-using-node-js-native-addons-fd3a24719c85#.3uzqa9r4w this article does well in explaining basics. but if you want to dive deep check official documentation.

if you want a really explanatory presentation check https://ivanvergiliev.github.io/node-cpp/#opening.

Optimal way of scheduling long job – nodejs/js

If you are using nodejs for some backend stuff to schedule some-work, probably you will hate scheduling because of two factors

You can’t set long timeout
Timers are removed when process restart

You would probably favor cronjobs for scheduling – or may be polling database for regular interval. Loading nodejs script every 5min is a costly job – developers understand, so you can’t follow php/short interval scheduling.

First problem can be solved by of setting a recursive call to a function, with time out. Many solutions on web.
But there lies a problem – You can’t cancel the timeout if you needed to after a recursive call. To solve the recursive call problem i wrote few lines of code.

function setLongTimeout(callback, timeout_ms, updateTimerID) {
    //if timout is more than 32bit int
    var timer;
    if (timeout_ms > 2147483647) {
        //now wait until the max wait time passes then call this function again with
        timer = setTimeout(function () {
            var id = setLongTimeout(callback, (timeout_ms - 2147483647), updateTimerID);
            if (typeof updateTimerID ==='function')
                updateTimerID(id);
        }, 2147483647);
    } else {
        timer = setTimeout(callback, timeout_ms);
    }
    if (typeof updateTimerID ==='function')
        updateTimerID(timer);
    return timer;
}

function setLongTimeout(callback, timeout_ms, updateTimerID) {

//if timout is more than 32bit int

var timer;

if (timeout_ms > 2147483647) {

//now wait until the max wait time passes then call this function again with

timer = setTimeout(function () {

var id = setLongTimeout(callback, (timeout_ms - 2147483647), updateTimerID);

if (typeof updateTimerID ==='function')

updateTimerID(id);

}, 2147483647);

} else {

timer = setTimeout(callback, timeout_ms);

}

if (typeof updateTimerID ==='function')

updateTimerID(timer);

return timer;

}

Above code behaves same as setTimeout if you don’t need timer object updates, else you can register for that.

Second problem – Timers are removed when process are restarted. – That shouldn’t be hard – You can create a function to execute atomic functions,
Note atomic – atomic because if your setTimeout code will depend on variables state you will need to load those variables from database which will make job harder – better way is to schedule something like – send email to [email protected] instead of send message to currently loggedin users.

Solution of second problem really depends on your problem – but if you analyze your scheduled jobs closely you will find that they are really atomic, you made them non-atomic !.

if you really need that your jobs are never lost from memory better way is to run a stable nodeserver & use IPC. – but that would be practically hard to maintain.

Converting Opensource Dictionary to JSON

Folddoc provide free online computer dictionary but they don’t have any standard tool or format to import dictionary in database or so.

Dictionary can be downloaded form http://foldoc.org/Dictionary.txt

Run the following nodejs code it will dump dictionary to OUTPUT.json and don’t forget to remove initial lines of disclaimer or they will add to your dictionary.

var fs  = require("fs");
var words=[];
var word,meaning="";
fs.readFileSync('./Dictionary.txt').toString().split('\n').forEach(function (line) { 
	if(line.length==0){
		meaning+="\n";
		return;
	}
    if(line.indexOf("\t")==0)
		meaning+=line.trim()+"\n";
	else{
		if(meaning.length>0)
			words.push({title:word,definition:meaning.trim()});
		word = line.trim();
		meaning = "";
	}
    
});

fs.appendFileSync("./output.json", JSON.stringify(words));

var fs = require("fs");

var words=[];

var word,meaning="";

fs.readFileSync('./Dictionary.txt').toString().split('\n').forEach(function (line) {

if(line.length==0){

meaning+="\n";

return;

}

if(line.indexOf("\t")==0)

meaning+=line.trim()+"\n";

else{

if(meaning.length>0)

words.push({title:word,definition:meaning.trim()});

word = line.trim();

meaning = "";

}

});

fs.appendFileSync("./output.json", JSON.stringify(words));

if you want to import to mongo use mongoimport --db mad --collection foldoc --file foldoc.json --jsonArray

At time of writing this article it had 15093 terms but importing resulted in 15110 documents, this might be because of some false positive.

Importing large CSV file to Mongo.

Someday you will need to import a large file to mongo or some other DBMS ,
You start writing code for achieving same using NodeJS, but when you run memory usage start increasing and either core of your CPU is at 100%.

Continue reading Importing large CSV file to Mongo.