Pages - Menu

標籤

AWS (1) bash (1) Boost (2) C (2) CMake (2) Concurrency_Programming (3) CPP (37) Database (2) DNS (1) Docker (4) Docker-Compose (1) ELK (1) emacs (4) gcp (1) gdrive (1) git (1) gitbash (2) gitlab (1) kvm (4) Linux (5) MT4 (4) MT5 (4) Multicast (2) MySQL (2) Nijatrader8 (1) OpenCV (1) Python (4) QT5 (1) R (1) rdp (3) screenshot (1) ssh (3) Tabnine (1) TCP (1) TensorFlow (1) Tools (12) Ubuntu_1904 (11) Ubuntu_20_04 (5) UDP (1) VS2010 (1) VS2015 (1) VS2019 (1) WebServer (1) Win10 (1) winmerge (1) WSL (1) xrdp (1)

搜尋此網誌

2020年3月21日星期六

Tabnine in spacemacs

Learn From

Environment

  • ubuntu 19.04
  • Spacemacs 0.200.13@26.3
  • Emacs 26.3

Aim

Backup the way how to install tabnine in spacemacs. Only use the following content when the above link is not available.

Steps

We would like to use tabnine as a back-end service provider. auto-completion is the front-end. We will do the following setting auto-completion -> company -> company-tabnine.

Create tabnine layer

  1. In spacemacs, type SPC SPC
  2. configuration-layer/create-layer
  3. Asking for Configuration layer path, use default path ~/.emacs.d/private/
  4. Asking for Configuration layer name, type tabnine
  5. Asking for Create readme? (y or n), type n

Inside ~/.emacs.d/private/tabnine/package.el


(defconst tabnine-packages
  '(company-tabnine)
  )

;;; initial company-tabnine library
(defun tabnine/init-company-tabnine()
  (use-package company-tabnine
    :ensure t
    :defer t
    :init
    :config)
  )
;;; use company-tabnine as company backend
(defun tabnine/post-init-company-tabnine()
  (with-eval-after-load 'company
    (add-to-list 'company-backends #'company-tabnine)
    )
  )

Add tabnine layer inside ~/.spacemacs


;; List of configuration layers to load.
dotspacemacs-configuration-layers
   '(
     auto-completion
     better-defaults
     version-control
     emacs-lisp
     git
     ivy
     tabnine
     )

Final step

  1. Restart spacemacs
  2. You may need to install tabnine binary, SPC SPC, company-tabnine-install-binary
  3. Try it out or enable the layer by SPC SPC, company-mode
    1. notice: Company mode enabled in current buffer

2020年3月18日星期三

Integrate DLL and MT4/5 VS2019 example

Aim

This is an example followed VS 2010 example. Here we will use VS 2019 to repeat the same example.

Steps

  • Create new project

  • In framework.h add

#define _DLLAPI extern "C" __declspec(dllexport)

Integrate DLL and MT4/5 VS2010 example

Aim

We would like to create a Win32 DLL by Visual Studio 2010. A script running on MT4 will then use that DLL as an example.

Learning Source

Steps

  • Create Win32 DLL project



  • In stdafx.h, add
#define _DLLAPI extern "C" __declspec(dllexport)

  • In MQL5DLLSamples.cpp

// MQL5DLLSamples.cpp : Defines the exported functions for the DLL application.
//

#include "stdafx.h"


//+------------------------------------------------------------------+
//| Passing and receving of simple variables                         |
//+------------------------------------------------------------------+
_DLLAPI int __stdcall fnCalculateSpeed(int &res1,double &res2)
  {
   int    res_int=0;
   double res_double=0.0;
   int    start=GetTickCount();
//--- simple math calculations
   for(int i=0;i<=10000000;i++)
     {
      res_int+=i*i;
      res_int++;
      res_double+=(double)i*(double)i;
      res_double++;
     }
//--- set calculation results
   res1=res_int;
   res2=res_double;
//--- return calculation time
   return(GetTickCount()-start);
  }
//+------------------------------------------------------------------+
//| Filling the array with values                                    |
//+------------------------------------------------------------------+
_DLLAPI void __stdcall fnFillArray(int *arr,const int arr_size)
  {
//--- check input variables
   if(arr==NULL || arr_size<1) return;
//--- fill array with values
   for(int i=0;i<arr_size;i++) arr[i]=i;
  }
//+------------------------------------------------------------------+
//| The substring replacement of the text string                     |
//| the string is passed as direct reference to the string content   |
//+------------------------------------------------------------------+
_DLLAPI void fnReplaceString(wchar_t *text,wchar_t *from,wchar_t *to)
  {
   wchar_t *cp;
//--- parameters checking
   if(text==NULL || from==NULL || to==NULL) return;
   if(wcslen(from)!=wcslen(to))             return;
//--- search for substring
   if((cp=wcsstr(text,from))==NULL)         return;
//--- replace it 
   memcpy(cp,to,wcslen(to)*sizeof(wchar_t));
  }
//+------------------------------------------------------------------+
//| Call for the crush                                               |
//+------------------------------------------------------------------+
_DLLAPI void __stdcall fnCrashTest(int *arr)
  {
//--- wait for receipt of a zero reference to call the exception
   *arr=0;
  }
//+------------------------------------------------------------------+


  • Build the dll file and put that into MQL4/Libraries in the MT4 folder
  • Prepare the mql4 script in MQL4/Scripts, DLLTest.mq4

//+------------------------------------------------------------------+
//|                                                      DLLTest.mq4 |
//|                        Copyright 2020, MetaQuotes Software Corp. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2020, MetaQuotes Software Corp."
#property link      "https://www.mql5.com"
#property version   "1.00"
#property strict
//---
#import "MQL5DLLSamples.dll"
int  fnCalculateSpeed(int &res1,double &res2);
void fnFillArray(int &arr[],int arr_size);
void fnReplaceString(string text,string from,string to);
void fnCrashTest(int arr);
#import
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//---
   //--- calling the function for calculations
   int    speed=0;
   int    res_int=0;
   double res_double=0.0;

   speed=fnCalculateSpeed(res_int,res_double);
   Print("Time ",speed," msec, int: ",res_int," double: ",res_double);
//--- call for the array filling
   int    arr[];
   string result="Array: "; 
   ArrayResize(arr,10);
   
   fnFillArray(arr,ArraySize(arr));
   for(int i=0;i<ArraySize(arr);i++) result=result+IntegerToString(arr[i])+" ";
   Print(result);
//--- modifying the string
   string text="A quick brown fox jumps over the lazy dog"; 
   
   fnReplaceString(text,"fox","cat");
   Print("Replace: ",text);
//--- and finally call a crash
//--- (the execution environment will catch the exception and prevent the client terminal crush)
   fnCrashTest(NULL);
   Print("You won't see this text!");
//---
  }
//+------------------------------------------------------------------+


  • Build that script and test on MT4
  • DONE

2020年3月17日星期二

Integrate MT4/5 with other process through ZMQ

Aim

We would like to treat MT4 as a server that can provide market data, backtest environment, trading execution. Other applications written by different programming language such as python, c++, R, c# can use ZMQ library as an interface to communicate with MT4.

Learning Source

Installation Steps --- Publish market data from MT4

  1. git clone
    1. darwinex github example
    2. mql-zmq github for mt4
  2. copy DWX_ZeroMQ_Server_v2.0.1_RC8.mq4
    • From dwx-zeromq-connector/v2.0.1/mql4/
    • To MT4/MQL4/Experts
  3. copy all folders and files
    • From mql-zmq/Include/
    • To MT4/MQL4/Include
  4. copy all dll files
    • From mql-zmq/Library/MT4/
    • To MT4/MQL4/Libraries
  5. Compile EA from MT4, drag and drop the EA and enable dll import
  6. Try to publish market data
    • You will see the following logs when success

Installation Steps --- Receive market data using python

  1. copy DWX_ZeroMQ_Connector_v2_0_1_RC8.py
    • From dwx-zeromq-connector/v2.0.1/python/api/
  2. Set up vitrualenv with python3
    1. Windows
      • Install python 3
      • open cmd
      • pip install virtualenv
      • virtualenv ./pythonEnv
      • ./pythonEnv/Script/activate
    2. Linux
      1. steps
  3. Install packages
    • pip install zmq
    • pip install pandas
  4. test.py, reference Example

import sys
sys.path.insert(1, '..')

import ZMQConnector.api.DWX_ZeroMQ_Connector_v2_0_1_RC8 as zmq

# ----------------------------------- Main
if __name__ == "__main__":
    _zmq = zmq.DWX_ZeroMQ_Connector()
    _zmq._DWX_MTX_SUBSCRIBE_MARKETDATA_('EURUSD')

2020年2月16日星期日

Add ssh key to gitlab

On Linux

Create ssh key on linux machine

# generate ssh key
ssh-keygen -t rsa -b 4096 -C "user@gamil.com"
ls
--- newKey
--- newKey.pub

# list ssh agent
ssh-add -l
...
ssh-add -L
...

# add to ssh agent
ssh-add ~/.ssh/newKey


# list ssh agent
ssh-add -l
...
ssh-add -L
...


Add your public key to gitlab, and clone project through ssh.

On Window

Install git bash. Then we have to create your private key as shown on the above steps and you have to:

$ cat ~/.bashrc

# ssh-agent
SSH_ENV="$HOME/.ssh/environment"

function start_agent {
    echo "Initialising new SSH agent..."
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo succeeded
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add;
    ssh-add $HOME/.ssh/<user private key>
}

# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
    . "${SSH_ENV}" > /dev/null
    #ps ${SSH_AGENT_PID} doesn't work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi

On Gitlab

You have to add public key to gitlab:


Clone project from gitlab through ssh method

 

 

2020年1月15日星期三

Linux dynamic library installation, ldconfig, .so file, application distribution

Problem

  • Using ubuntu 19.04
  • Local build cpp application
  • Installed to /usr/local/yourLib
  • Error message
    • cannot open shared object file: No such file or directory

Structure

  • /usr/local/yourLib/bin
  • /usr/local/yourLib/lib
  • /usr/local/yourLib/include

Solution

  • Add path permanent
    • sudo touch /etc/ld.so.conf.d/user.conf
      sudo echo "/usr/local/yourLib/lib" >> /etc/ld.so.conf.d/user.conf
      sudo ldconfig
      
  • Add path temporary
    • export LD_LIBRARY_PATH=/usr/local/yourLib/lib
      

2020年1月12日星期日

Ubuntu 19.04 cannot access external disk after install chrome-remote-desktop

Problem

  • Using ubuntu 19.04
  • Installed chrome-remote-desktop
  • Cannot auto mount external usb drive, authorization problem

Reason

  • user under group chrome-remote-desktop

Work around

  • Remove user from that group

Learn from

Current status

  • Cannot find a way to solve this problem yet.

2020年1月7日星期二

Install dlib

Install dlib on python 3

Learn from

Installation Steps

$ pip install numpy
$ pip install scipy
$ pip install -U scikit-image
$ pip install dlib

Install dlib on c++

# build dlib locally
wget http://dlib.net/files/dlib-19.6.tar.bz2
tar xvf dlib-19.9.tar.bz2
cd dlib-19.6/
mkdir build
cd build
cmake ..
cmake --build . --config Release
sudo make install
# update the link libraries, so that programs knows where to find the dynamic lib during run time
sudo ldconfig
cd ..
# Now you can use pkg-config to provide path to Dlib’s include directory and link Dlib library file. This is useful during build time
pkg-config --libs --cflags dlib-1

2020年1月6日星期一

Install TensorFlow 2.0 on Ubuntu 19.04, Python 3

Learn From

Install TensorFlow 2.0

sudo apt-get install screen
sudo apt-get install libxmu-dev libxi-dev libglu1-mesa libglu1-mesa-dev 
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-del libv4l-dev 
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev 
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install libopenblas-dev libatlas-base-dev liblapack-dev gfortran
sudo apt-get install libhdf5-dev
sudo apt-get install python3-dev python3-tk python-pil.imagetk
sudo apt-get install libgtk-3-dev

Install CUDA 10.0

You may try to install the latest CUDA.

cd ~
mkdir installers
cd installers/
wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux

===========
= Summary =
===========

Driver:   Not Selected
Toolkit:  Installed in /usr/local/cuda-10.0
Samples:  Installed in /home/sulfred

Please make sure that
 -   PATH includes /usr/local/cuda-10.0/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-10.0/lib64, or, add /usr/local/cuda-10.0/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run the uninstall script in /usr/local/cuda-10.0/bin

Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-10.0/doc/pdf for detailed information on setting up CUDA.

Add CUDA Path

vim ~/.bashrc

# NVIDIA CUDA Toolkit
export PATH=/usr/local/cuda-10.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:$LD_LIBRARY_PATH

Install cuDNN

Download from : download page

You may download the latest cudnn. Here we are using cudnn-10.0, for example: cudnn-10.0-linux-x64-v7.4.2.24.tgz to ~/installers


cd ~/installers
tar -zxf cudnn-10.0-linux-x64-v7.4.2.24.tgz
cd cuda
sudo cp -P lib64/* /usr/local/cuda/lib64/
sudo cp -P include/* /usr/local/cuda/include/

Prepare python virtual environment

# standard image processing
pip install opencv-contrib-python
pip install scikit-image
pip install pillow
pip install imutils

# machine learning libraries
pip install scikit-learn
pip install matplotlib
pip install pregressbar2
pip install beautifulsoup4
pip install pandas

2020年1月5日星期日

Create cpp tags for code navigation on spacemacs

Aim

We would like to use cscope and gtags to create tags for c++ development.

prepareSpacemacsTags.sh

#!/bin/bash
buildPath="$PWD"

# remove old tags
rm ${buildPath}/GPATH
rm ${buildPath}/GRTAGS
rm ${buildPath}/GTAGS

rm ${buildPath}/cscope.files
rm ${buildPath}/cscope.out
rm ${buildPath}/cscope.in.out
rm ${buildPath}/cscope.po.out

# make tags
echo "######### make GTag"
gtags
echo "######### make cscope"
find . -name "*.cc" -o -name "*.c" -o -name "*.cpp" -o -name "*.h" -o -name "*.hpp" > cscope.files
cscope -q -R -b -k -i cscope.files

Usage

$ cd /path/to/project/root
$ prepareSpacemacsTags.sh

prepareTags.YourProject.sh

#!/bin/bash
basePath=`pwd`
tempTagPath="$basePath/../tempTagFolder"
appFolders=("APPFolderName") # filter out

# move all folders beyond root
if [ ! -d "$tempTagPath" ]; then
    mkdir "$tempTagPath"
fi
if [ ! -d "$tempTagPath/app" ]; then
    mkdir "$tempTagPath/app"
fi
for folder in ${appFolders[@]}; do
    nextFolder="$basePath/app/$folder"
    mv $nextFolder $tempTagPath/app
done

prepareSpacemacsTags.sh

mv $tempTagPath/app/* ./app

if [ -d "$tempTagPath/app" ] && [ -z "$(ls -A $tempTagPath/app/)" ]; then
    rm -rf $tempTagPath/app
else
    echo "No need to have further action $tempTagPath/app"
fi
if [ -d "$tempTagPath" ] && [ -z "$(ls -A $tempTagPath)" ]; then
    rm -rf $tempTagPath
else
    echo "No need to have further action $tempTagPath"
fi

Usage

$ cd /path/to/project/root
$ ls
./app
./lib
$ prepareTags.YourProject.sh

2020年1月3日星期五

Ubuntu 19.04 stay in splash screen after installed nvidia driver

Spec:

  • Ubuntu 19.04
  • GTX 1070
  • HP 420 workstation
Problems:
  • Stay in splash screen and cannot login after installed Nvidia driver 418
  • No problem when using Nouveau open source display driver
Work around:
  • Install Ubuntu without installing 3rd party and updates
  • edit /etc/default/grub
    • from
      • GRUB_TIMEOUT=0
      • GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
    • to
      • GRUB_TIMEOUT=8
      • GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
    • Now OS boot without display driver enabled in kernel
Final solution: