Uncategorized Archives

Life : A simple advice

A thoughtful advice given to me long back, just posting it here for general reference, should work for you too 😉

Stand strong always, may the sun always shine brightly for you after heavy rain.

Anger is paralyzing, its just the absence of control, You will have no use of it whatsoever.

So be patient and calm, make people quiet wid your skills, mindset and behavior if these three don’t work then don’t think ur words would matter.
Anonymous, June 2014

PDF Signing using hardware token (DSC) in Python

Hardware based tokens are widely used in India to generate signed PDF’s like invoices and agreement. We wrote small Python code to sign the invoices automatically where token was attached to a local server.

Windows drives are widely available but rare to find linux drivers are listed https://www.e-mudhra.com/Repository/index.html

You can uncomment to get the token name print(self.pkcs11.getSlotList(tokenPresent=True))
print(self.pkcs11.getTokenInfo(1)) to get token name, for PROXKey the name "WD PROXKey" was generated.

#!/usr/bin/env vpython3
# *-* coding: utf-8 *-*
import sys
import datetime
from endesive import pdf, hsm

import os
import sys

if sys.platform == 'win32':
    dllpath = r'c:\windows\system32\cryptoCertum3PKCS.dll'
else:
    dllpath = '/usr/lib/WatchData/ProxKey/lib/libwdpkcs_SignatureP11.so'

import PyKCS11 as PK11

class Signer(hsm.HSM):
    def certificate(self):
        #print(self.pkcs11.getSlotList(tokenPresent=True))
        #print(self.pkcs11.getTokenInfo(1))
#        print(self.pkcs11.getTokenInfo(2))
#        print(self.pkcs11.getTokenInfo(3))


#        print(self.pkcs11.getSlotInfo(1))
        self.login("WD PROXKey","12345678") # WF PROXKey is token name.
        keyid = &#91;0x5e, 0x9a, 0x33, 0x44, 0x8b, 0xc3, 0xa1, 0x35, 0x33, 0xc7, 0xc2, 0x02, 0xf6, 0x9b, 0xde, 0x55, 0xfe, 0x83, 0x7b, 0xde]
        #keyid = &#91;0x3f, 0xa6, 0x63, 0xdb, 0x75, 0x97, 0x5d, 0xa6, 0xb0, 0x32, 0xef, 0x2d, 0xdc, 0xc4, 0x8d, 0xe8]
        keyid = bytes(keyid)
        try:
            pk11objects = self.session.findObjects(&#91;(PK11.CKA_CLASS, PK11.CKO_CERTIFICATE)])
            all_attributes = &#91;
                #PK11.CKA_SUBJECT,
                PK11.CKA_VALUE,
                #PK11.CKA_ISSUER,
                #PK11.CKA_CERTIFICATE_CATEGORY,
                #PK11.CKA_END_DATE,
                PK11.CKA_ID,
            ]

            for pk11object in pk11objects:
                try:
                    attributes = self.session.getAttributeValue(pk11object, all_attributes)
                except PK11.PyKCS11Error as e:
                    continue

                attrDict = dict(list(zip(all_attributes, attributes)))
                cert = bytes(attrDict&#91;PK11.CKA_VALUE])
                #if keyid == bytes(attrDict&#91;PK11.CKA_ID]):
                return bytes(attrDict&#91;PK11.CKA_ID]), cert
        finally:
            self.logout()
        return None, None

    def sign(self, keyid, data, mech):
        self.login("WD PROXKey","12345678")
        try:
            privKey = self.session.findObjects(&#91;(PK11.CKA_CLASS, PK11.CKO_PRIVATE_KEY)])&#91;0]
            mech = getattr(PK11, 'CKM_%s_RSA_PKCS' % mech.upper())
            sig = self.session.sign(privKey, data, PK11.Mechanism(mech, None))
            return bytes(sig)
        finally:
            self.logout()

def main():
    date = datetime.datetime.utcnow() - datetime.timedelta(hours=12)
    date = date.strftime('%Y%m%d%H%M%S+00\'00\'')
    dct = {
        "sigflags": 3,
        "sigpage": 0,
        "sigbutton": True,
        "contact": "madhurendra@tikaj.com",
        "location": 'India',
        "signingdate": date.encode(),
        "reason": 'Sample sign',
        "signature": 'Madhurendra Sachan',
        "signaturebox": (0, 0, 100, 100),
    }
    clshsm = Signer(dllpath)
    fname = 'sample.pdf'
    datau = open(fname, 'rb').read()
    datas = pdf.cms.sign(datau, dct,
        None, None,
        &#91;],
        'sha256',
        clshsm,
    )
    fname = fname.replace('.pdf', '-signed.pdf')
    with open(fname, 'wb') as fp:
        fp.write(datau)
        fp.write(datas)


main()

#!/usr/bin/env vpython3

# *-* coding: utf-8 *-*

import sys

import datetime

from endesive import pdf, hsm

import os

import sys

if sys.platform == 'win32':

dllpath = r'c:\windows\system32\cryptoCertum3PKCS.dll'

else:

dllpath = '/usr/lib/WatchData/ProxKey/lib/libwdpkcs_SignatureP11.so'

import PyKCS11 as PK11

class Signer(hsm.HSM):

def certificate(self):

#print(self.pkcs11.getSlotList(tokenPresent=True))

#print(self.pkcs11.getTokenInfo(1))

# print(self.pkcs11.getTokenInfo(2))

# print(self.pkcs11.getTokenInfo(3))

# print(self.pkcs11.getSlotInfo(1))

self.login("WD PROXKey","12345678") # WF PROXKey is token name.

keyid = [0x5e, 0x9a, 0x33, 0x44, 0x8b, 0xc3, 0xa1, 0x35, 0x33, 0xc7, 0xc2, 0x02, 0xf6, 0x9b, 0xde, 0x55, 0xfe, 0x83, 0x7b, 0xde]

#keyid = [0x3f, 0xa6, 0x63, 0xdb, 0x75, 0x97, 0x5d, 0xa6, 0xb0, 0x32, 0xef, 0x2d, 0xdc, 0xc4, 0x8d, 0xe8]

keyid = bytes(keyid)

try:

pk11objects = self.session.findObjects([(PK11.CKA_CLASS, PK11.CKO_CERTIFICATE)])

all_attributes = [

#PK11.CKA_SUBJECT,

PK11.CKA_VALUE,

#PK11.CKA_ISSUER,

#PK11.CKA_CERTIFICATE_CATEGORY,

#PK11.CKA_END_DATE,

PK11.CKA_ID,

]

for pk11object in pk11objects:

try:

attributes = self.session.getAttributeValue(pk11object, all_attributes)

except PK11.PyKCS11Error as e:

continue

attrDict = dict(list(zip(all_attributes, attributes)))

cert = bytes(attrDict[PK11.CKA_VALUE])

#if keyid == bytes(attrDict[PK11.CKA_ID]):

return bytes(attrDict[PK11.CKA_ID]), cert

finally:

self.logout()

return None, None

def sign(self, keyid, data, mech):

self.login("WD PROXKey","12345678")

try:

privKey = self.session.findObjects([(PK11.CKA_CLASS, PK11.CKO_PRIVATE_KEY)])[0]

mech = getattr(PK11, 'CKM_%s_RSA_PKCS' % mech.upper())

sig = self.session.sign(privKey, data, PK11.Mechanism(mech, None))

return bytes(sig)

finally:

self.logout()

def main():

date = datetime.datetime.utcnow() - datetime.timedelta(hours=12)

date = date.strftime('%Y%m%d%H%M%S+00\'00\'')

dct = {

"sigflags": 3,

"sigpage": 0,

"sigbutton": True,

"contact": "[email protected]",

"location": 'India',

"signingdate": date.encode(),

"reason": 'Sample sign',

"signature": 'Madhurendra Sachan',

"signaturebox": (0, 0, 100, 100),

}

clshsm = Signer(dllpath)

fname = 'sample.pdf'

datau = open(fname, 'rb').read()

datas = pdf.cms.sign(datau, dct,

None, None,

[],

'sha256',

clshsm,

)

fname = fname.replace('.pdf', '-signed.pdf')

with open(fname, 'wb') as fp:

fp.write(datau)

fp.write(datas)

main()

.dll/.so path for common tokens

For Windows the file will be in Windows\SysWOW64″ or “WINDOWS\system32” or “WINNT\system32”
For Linux machine the file will be in /usr/local/lib/ or /usr/lib/

Hardware Token Type	Library file (Windows)	Library file (Linux)
SafeSign	aetpkss1.dll	aetpkss1.so
eMudhra	eMudhra\eMudhra CSPV1.0\wdpkcs.dll	1. WatchData/eMudhra_3.4.3/lib/libpkcs11wrapper.so 2. WatchData/eMudhra_3.4.3/lib/libwdpkcs_eMudhra_343.so
Trust Key	1. TRUST KEY\TRUST KEY CSP V1.0\wdpkcs.dll 2. C:\Windows\System32\TRUSTKEYP11_ND_v34.dll	1. WatchData/TRUSTKEY/lib/libpkcs11wrapper.so 2. WatchData/TRUSTKEY/lib/libwdpkcs_TRUSTKEY.so
Belgium eID MiddleWare	beidpkcs11.dll	beidpkcs11.so
Gemalto Cryptocard Token	libgtop11dotnet.dll	libgtop11dotnet.so
EPass	eps2003csp11.dll
Aladdin eToken	eTPKCS11.dll
Safenet iKey	dkck201.dll
Starkey	aetpkss1.dll
Watchdata PROXkey	SignatureP11.dll	WatchData/ProxKey/lib/libwdpkcs_SignatureP11.so

.DLL or .so path’s for linux.

Why books matter not just reading

So I am developer and maker – basically a keen person who is interested in almost everything which sounds logical. As hard it is to understand more interested I become in something.

In current “vast” space of information, On an average I read 10-15 articles a day that is being done since last 8 years but there hasn’t been a sense of knowledge satisfaction. If I compare it to my school days where I used to read through books which were related to computers, technical but outdated there is significant difference in satisfaction. (Do note, here I am not debating about articles in digital form & books in physical form.)

To understand the problem I started by creating a new habit.

Reading books – few pages as I wake up, few pages in evening. (Not a strict goal but 2 books/month – where each book averages to 200 pages and isn’t fictional )

So far it has been good and satisfaction level is quiet high !

but why ?

It seems it has to do with us :

We skim not read, We skim news paper for information, we usually skim articles for information which we seek, we skim most of the content. With time, when brain see’s similar organization of content it activates the skimming mode.
Information online is volatile; You might find similar or same piece of advise everywhere, copied & in few days it might disappear – & when not used it is trashed by brain. We need to see something once in a while to automatically recall & become more persistent.

Eg: You read a book, keep it somewhere, you might stumble across it someday & open it – might check few things – you brain will start recalling many events but how many times you re-read same article ?
Content doesn’t have a sense of authenticity; You come across articles, read them, understand them but mostly you can’t trust a content. How many people spend a month or a year in writing an article ?
Articles have one key focus SEO, even if content is genuine it should have click baits, keywords etc but authenticity is not something search engines seek.
Effort is missing; Again I am not talking about every article but publishing an article doesn’t require enough effort. While writing a book takes money, getting a publication house to agree and years of effort to put down thought – because once it is printed – it is printed.

if you write something wrong – you can’t take book from people, there is error – it will persist for your lifetime. People will judge you by your book and it’s cover.
Information is not organized; When you get a book, it’s always on specific theme and in all those pages it will talk just about that. You read related content such that slowly it starts persisting in your brain. And that how I got a sense of knowledge.

Irony is – this is an article.

This doesn’t mean that articles are bad but I have following perception on how articles should or shouldn’t be :

Not for SEO : Yes, check the site if a lot of articles are of clickbait nature they will actually not convey good enough information. if a person is writing articles very frequently – those are there just for sake of being there.
Should convey meaningful information : Now days most articles are just DERIVED from some other article, it is a good thing if article can innovate in certain manner or atleast put effort to organise certain information.
There is no way but experience – on one hand you should not trust every source and should get information from trusted sources, but on other a good source doesn’t write frequently or broad topics. With time if one observes he/she can come to know how to identify bogus sources.
Refer to official sources for information, yes just as in programming there are millions of tutorials but if I ever want to learn about something new I refer to official documentation – it doesn’t matter how badly it is written but usually documentations are accurate and explain logic as they are written by developers not content writer having bleak understanding about underlying principles. Same if for everything – if you want to read about some rules refer to government issued rule book – might not sound good – but it’s best way to have accurate information.

I know this is not a popular blog and there are not million followers but since you have read this articles feel free to comment and share what I missed – what else should be incorporated.

Adding a display over network !

You are doomed if your laptop has only one HDMI Port & you are running Linux in that box.

Unlike windows – where there are tons of easy to run solution, & there are still tons of solutions if you have big numbers in your pocket.

Certainly – I neither wish to spend money on external VGA/HDMI extender or docking station nor to change OS. So after googling a lot I discovered a solution which utilized a different machine to act as the streaming client. You can use RPI or an old p4 machine.

Below is how it works.

Adds a virtual display on your machine, usually all graphics card support at least one virtual display. You do all this using xrandr
Create a VNC Server to stream that display – but since you cannot each time run two commands just to connect display – we are running VNCViewer in listen mode
Connect to vnc viewer & keep running it in the background.

You Laptop —[Display Data]—> Network —-> VNCViewer

As suggested above you data is being streamed over the network – you cannot run 4k data. But if you have good Lan speed you won’t face any problem.

I have also optimized settings for best experience – so far I can use terminal, watch videos – the only drawback is you feel the lag when you use keyboard or mouse for realtime feedback.

In Client Machine, i.e. your laptop.

You will need to install x11vnc & screen

#!/bin/bash
#Run VNC server in remote device. 10.0.0.2 is my machine which has monitor connected.
ssh  root@10.0.0.2 "nohup /root/vncserver.sh > /dev/null 2>&1 &"
xrandr --addmode VIRTUAL1 1920x1080
xrandr --output VIRTUAL1 --mode 1920x1080 --right-of eDP1 # find your main display
screen -X -S vnc quit
killall x11vnc
# adjust postion 1920x1080+3511+180, i.e. 3511+180 accordingly
# i have different resultion display so had to move.
screen -dmS vnc  x11vnc -connect 10.0.0.2:5500  -display :0 -clip 1920x1080+3511+180 -wirecopyrect  -viewonly --nossl -ncache 10 -ncache_cr

#!/bin/bash

#Run VNC server in remote device. 10.0.0.2 is my machine which has monitor connected.

ssh root@10.0.0.2 "nohup /root/vncserver.sh > /dev/null 2>&1 &"

xrandr --addmode VIRTUAL1 1920x1080

xrandr --output VIRTUAL1 --mode 1920x1080 --right-of eDP1 # find your main display

screen -X -S vnc quit

killall x11vnc

# adjust postion 1920x1080+3511+180, i.e. 3511+180 accordingly

# i have different resultion display so had to move.

screen -dmS vnc x11vnc -connect 10.0.0.2:5500 -display :0 -clip 1920x1080+3511+180 -wirecopyrect -viewonly --nossl -ncache 10 -ncache_cr

In server Machine (LAN address 10.0.0.2)

if [[ $(ps -ef | grep -c vncviewer)  -ne 1 ]];
then
    echo "Running"
else
   export DISPLAY=:0.0;
    vncviewer -listen 0 -fullscreen -owncmap -viewonly;
fi

if [[ $(ps -ef | grep -c vncviewer) -ne 1 ]];

then

echo "Running"

else

export DISPLAY=:0.0;

vncviewer -listen 0 -fullscreen -owncmap -viewonly;

You will need to install VNC Viewer in client side.

NodeJs – Improving performance using native binding

This is known fact that javascript is really slow in terms of sync operations like multiplications, divisions etc. – Since nodejs is based on js – Node inherits the curse.
Check http://benchmarksgame.alioth.debian.org/u64/compare.php?lang=v8&lang2=gpp this article to cross check the fact. Nodejs is fast in some aspects. I love it 😛

Doesn’t matter how much you improve your algorithm but c++ will outperform if you are running a sophisticated algorithm but one can push through imprecation.
Nodejs is based on V8 (Most ports), v8 based on C++, You must be already aware of that you can use C++ in nodejs – But it seems complex.

But it is not – It is easy to build a native extension & compile in seconds https://medium.com/@devlucky/how-to-get-a-performance-boost-using-node-js-native-addons-fd3a24719c85#.3uzqa9r4w this article does well in explaining basics. but if you want to dive deep check official documentation.

if you want a really explanatory presentation check https://ivanvergiliev.github.io/node-cpp/#opening.

Converting Opensource Dictionary to JSON

Folddoc provide free online computer dictionary but they don’t have any standard tool or format to import dictionary in database or so.

Dictionary can be downloaded form http://foldoc.org/Dictionary.txt

Run the following nodejs code it will dump dictionary to OUTPUT.json and don’t forget to remove initial lines of disclaimer or they will add to your dictionary.

var fs  = require("fs");
var words=[];
var word,meaning="";
fs.readFileSync('./Dictionary.txt').toString().split('\n').forEach(function (line) { 
	if(line.length==0){
		meaning+="\n";
		return;
	}
    if(line.indexOf("\t")==0)
		meaning+=line.trim()+"\n";
	else{
		if(meaning.length>0)
			words.push({title:word,definition:meaning.trim()});
		word = line.trim();
		meaning = "";
	}
    
});

fs.appendFileSync("./output.json", JSON.stringify(words));

var fs = require("fs");

var words=[];

var word,meaning="";

fs.readFileSync('./Dictionary.txt').toString().split('\n').forEach(function (line) {

if(line.length==0){

meaning+="\n";

return;

}

if(line.indexOf("\t")==0)

meaning+=line.trim()+"\n";

else{

if(meaning.length>0)

words.push({title:word,definition:meaning.trim()});

word = line.trim();

meaning = "";

}

});

fs.appendFileSync("./output.json", JSON.stringify(words));

if you want to import to mongo use mongoimport --db mad --collection foldoc --file foldoc.json --jsonArray

At time of writing this article it had 15093 terms but importing resulted in 15110 documents, this might be because of some false positive.

Importing large CSV file to Mongo.

Someday you will need to import a large file to mongo or some other DBMS ,
You start writing code for achieving same using NodeJS, but when you run memory usage start increasing and either core of your CPU is at 100%.

Continue reading Importing large CSV file to Mongo.

Getting whatsapp statistics – WhatsappWeb

There are no tool available for whatsapp which give us how many messages a user has contributed to group, Sometimes you need.

Whatsapp web exposes whatsapp to JS, whiche makes things more easy,
I wrote a simple code which logs Number of messages sent by each user of group, obviously you need full conversation open in web, You can use Home Key to do that,

The code was written in few minutes so it doesn’t have proper validations or automation,

But is good for purpose.
Open whatsappWeb and after opening complete conversation of group execute the following code in Google Chrome – Developer Console

var v = document.querySelectorAll('.msg.msg-group .text-clickable');
var t, count ={};
for(var i=0,l=v.length;i<l;++i){
t = v[i].textContent.trim();
 if(typeof count[t]=== 'undefined')
	count[t]=0;
 ++count[t];
}

console.log(count);

var v = document.querySelectorAll('.msg.msg-group .text-clickable');

var t, count ={};

for(var i=0,l=v.length;i<l;++i){

t = v[i].textContent.trim();

if(typeof count[t]=== 'undefined')

count[t]=0;

++count[t];

}

console.log(count);

Redis Vs MongoDB

Redis and Mongodb are considered to solve a common problem that “There is something wrong with RDBMS”
So what is difference ?

Redis is stored in memory and swapped with disk, So data is not lost and you can enjoy performance of your RAM. (Not to be confused with Memcached it is memory only database) whereas Mongodb is disk only database.
Redis can store many type of object strings, hashes, lists, sets, sorted sets whereas Mongo can only store key value pair in form of document (mongo is schemaless).
Redis is hard to learn whereas mongo is easier to understand and get started.
Redis has no support for clustering (Clustering support has been added since version 3.0) whereas mongo has inbuilt clustering support.

Why not nodejs v4.1.1

Recently i updated nodejs installation to v4.1.1 using

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs

1 2	curl -sL https://deb.nodesource.com/setup_4.x \| sudo -E bash - sudo apt-get install -y nodejs

and i ended up fixing many bug in other modules, but i couldn’t move far in this decision.

Nodejs has a history of breaking compatibility with older code in every new release, so if you have project with multiple dependencies don’t upgrade you nodejs version to latest, Instead use NVM to test your code for compatibility in different versions.