ELK Setup Tutorial

Installation

OS

Java

Elasticsearch

Kibana

Logstash

Filebeat

Log Index

Kibana Dashboard Setup

https://phoenixnap.com/kb/how-to-install-elk-stack-on-ubuntu

 

https://blog.johnwu.cc/article/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-red-hat.html

 

https://medium.com/%E7%A8%8B%E5%BC%8F%E4%B9%BE%E8%B2%A8/elk-%E6%95%99%E5%AD%B8%E8%88%87%E4%BB%8B%E7%B4%B9-c54af6f06e61

https://www.itread01.com/hkhklxhkq.html

Multiple Git User

This example is using gitbash on Windows.

Config file locations:

• C:\Users\<name>\.gitconfig
• C:\Users\<name>\path_to_github_project\.git\config

# Set up global user
$ git config --global user.name "Your_Name_001"
$ git config --global user.email "Your_Email_001"

# Set up local user and ssh key
$ cd /path_to_your_project
$ git config user.name "Your_Name_002"
$ git config user.email "Your_Email_002"
$ git config --local --add core.sshCommand 'ssh -i ~/.ssh/Your_SSH_Private_Key_002'

Done

Install spacemacs on windows

Steps for spacemacs installation

1. Download and install emacs on windows
1. https://www.gnu.org/software/emacs/download.html
2. Install the spacemacs config to appdata
1.  %USERPROFILE%\appdata\roaming\.emacs.d
2. https://www.spacemacs.org/#

  git clone https://github.com/syl20bnr/spacemacs C:\%USERPROFILE%\appdata\roaming\.emacs.d  

Steps for gtags and cscope

1. Get the binaries from web or
1. from one drive
2. Put those binaries to a folder in the windows environment path
3. Use it with the scrip: 
1. https://github.com/SulfredLee/DailyProblem/blob/master/devTools/prepareSpacemacsTags.bat
   

Cheat sheet compilation

 Bash

# Get space usage
du -chs ./* | sort -rh
du -kh --max-depth=1 | sort -rh

# find files
find [path] -name "lost+found" -prune -o -name "abc*.log.gz" -print0

 Cpp

# cpp debugs
## cpp heap memory checking
sudo cat /proc/18594/maps |grep heap # cat the process memory

55da5171a000-55da5178a000 rw-p 00000000 00:00 0 [heap]
55da5178a000-55db1c7c3000 rw-p 00000000 00:00 0 [heap]

55db1c7c3000-55da5178a000= 3,406,008,320 around 3G mem.

## Then dump the heap memory to a file(example from UAT):
dd if=/proc/29619/mem bs=1 skip=$((0x55db1c7c3000)) count=$((0x55db1c7c3000-0x55da5178a000)) status=none | strings > /tmp/heap_log.txt

 Docker Compose

# up and run in background
docker-compose up -d

# stop --- will keep the data
docker-compose stop

# shutdown --- will remove the data
docker-compose down -v

# check service environment variables
docker-compose run web env

# check config --- this can print the real value of the variable
docker-compose config

Packaging Python Project

Reference

https://packaging.python.org/tutorials/packaging-projects/

https://dzone.com/articles/executable-package-pip-install 

https://setuptools.pypa.io/en/latest/userguide/declarative_config.html

Cheat sheet

# Install all needed packages
python -m pip install --upgrade pip setuptools wheel
python -m pip install tqdm
python -m pip install --upgrade twine
python -m pip install --upgrade build

# Prepare the package structure and all needed files

# Build the package
python -m build

# Upload to test
python -m twine upload --repository testpypi dist/*
# Upload to prod
python -m twine upload --repository pypi dist/*

# Install the package by specific the repo
python -m pip install --index-url https://test.pypi.org/simple/ --no-deps example-pkg-YOUR-USERNAME-HERE

Testing code snippets

 

#include <iostream>

void main(int argc, char* argv[])
{
    return 0;
}

Useful DNS List

 Google DNS

 Cloudflare

C++ package manager CMake vcpkg

 How to use vcpkg with CMake

Steps

• Create C++ project by CCreate
• CCreate -p TradingPlatform
• cd TradingPlatform/Projects/TradingPlatform/app
• CCreate -a ThePlatform
• Install vcpkg

• https://docs.microsoft.com/en-us/cpp/build/install-vcpkg?view=msvc-160&tabs=linux
• git clone https://github.com/microsoft/vcpkg
• cd ./vcpkg
• ./bootstrap-vcpkg.sh
• Install package
• cd ./vcpkg
• ./vcpkg search | grep json
• ./vcpkg install rapidjson

Computing installation plan...
The following packages will be built and installed:
    rapidjson[core]:x64-linux -> 2020-09-14
Detecting compiler hash for triplet x64-linux...
Could not locate cached archive: /home/sulfred/.cache/vcpkg/archives/16/16e324f00567b9fbcd2f106e12e631094b31b5de.zip
Starting package 1/1: rapidjson:x64-linux
Building package rapidjson[core]:x64-linux...
-- Downloading https://github.com/Tencent/rapidjson/archive/ce81bc9edfe773667a7a4454ba81dac72ed4364c.tar.gz -> Tencent-rapidjson-ce81bc9edfe773667a7a4454ba81dac72ed4364c.tar.gz...
-- Extracting source /home/sulfred/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform/Projects/vcpkg/downloads/Tencent-rapidjson-ce81bc9edfe773667a7a4454ba81dac72ed4364c.tar.gz
-- Using source at /home/sulfred/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform/Projects/vcpkg/buildtrees/rapidjson/src/c72ed4364c-35829b36f2.clean
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /home/sulfred/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform/Projects/vcpkg/packages/rapidjson_x64-linux/share/rapidjson/copyright
-- Installing: /home/sulfred/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform/Projects/vcpkg/packages/rapidjson_x64-linux/share/rapidjson/usage
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /home/sulfred/.cache/vcpkg/archives/16/16e324f00567b9fbcd2f106e12e631094b31b5de.zip
Building package rapidjson[core]:x64-linux... done
Installing package rapidjson[core]:x64-linux...
Installing package rapidjson[core]:x64-linux... done
Elapsed time for package rapidjson:x64-linux: 1.485 s

Total elapsed time: 1.76 s

The package rapidjson provides CMake integration:

    find_package(RapidJSON CONFIG REQUIRED)
    target_include_directories(main PRIVATE ${RAPIDJSON_INCLUDE_DIRS})

• Cmake setting

• -DCMAKE_TOOLCHAIN_FILE=../Projects/vcpkg/scripts/buildsystems/vcpkg.cmake

sulfred@sulfred-PC:~/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform/Debug$ cat readme.txt | grep cmake
cmake -G Ninja ../Projects -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=../Install -DCMAKE_TOOLCHAIN_FILE=../Projects/vcpkg/scripts/buildsystems/vcpkg.cmake
sulfred@sulfred-PC:~/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform/Debug$ cd ..
sulfred@sulfred-PC:~/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform$ ll
total 24
drwxrwxr-x 6 sulfred sulfred 4096 Mar 27 20:05 ./
drwxrwxr-x 3 sulfred sulfred 4096 Mar 27 20:10 ../
drwxrwxr-x 4 sulfred sulfred 4096 Mar 27 20:29 Debug/
drwxrwxr-x 3 sulfred sulfred 4096 Mar 27 20:30 Install/
drwxrwxr-x 4 sulfred sulfred 4096 Mar 27 20:30 Projects/
drwxrwxr-x 2 sulfred sulfred 4096 Mar 27 20:05 Release/
sulfred@sulfred-PC:~/Documents/SoftwareDev/github/SulfredLee/DailyProblem/cpp/app/ProjectCreater/bin/TradingPlatform$ ll Projects/
total 24
drwxrwxr-x  4 sulfred sulfred 4096 Mar 27 20:30 ./
drwxrwxr-x  6 sulfred sulfred 4096 Mar 27 20:05 ../
-rw-rw-r--  1 sulfred sulfred 1025 Mar 27 20:28 CMakeLists.txt
drwxrwxr-x  5 sulfred sulfred 4096 Mar 27 20:30 TradingPlatform/
-rw-rw-r--  1 sulfred sulfred  560 Mar 27 20:05 TradingPlatform.cmake
drwxrwxr-x 14 sulfred sulfred 4096 Mar 27 20:12 vcpkg/

k

• cat ../Projects/CMakeLists.txt
• find_package(RapidJSON CONFIG REQUIRED)
• message(STATUS "Rapid include PATH: " ${RAPIDJSON_INCLUDE_DIRS})

Use winmerge in git bash

 .gitconfig

location

•  C:\Users\<name>\.gitconfig

.gitconfig

[diff]
       tool = winmerge
       prompt = false
[difftool "winmerge"]
       path = C:/Program Files/WinMerge/winmergeu.exe
	   cmd = \"C:/Program Files/WinMerge/winmergeu.exe\" /e /u /wl /dl "$LOCAL" /dr "$REMOTE" "$LOCAL" "$REMOTE"
[merge]
       tool = winmerge
       prompt = false
[mergetool "winmerge"]
       path = C:/Program Files/WinMerge/winmergeu.exe
       cmd = \"C:/Program Files/WinMerge/winmergeu.exe\" /e /u /wl /dl "$LOCAL" /dr "$REMOTE" "$LOCAL" "$REMOTE" -o "$MERGED"
       trustExitCode = true
       keepBackup = false

[diff]
       tool = meld
       prompt = false
[difftool "meld"]
       path = C:/Program Files (x86)/Meld/Meld.exe
	   cmd = \"C:/Program Files (x86)/Meld/Meld.exe\" "$LOCAL" "$MERGED" "$REMOTE" --output "$MERGED"
[merge]
       tool = meld
       prompt = false
[mergetool "meld"]
       path = C:/Program Files (x86)/Meld/Meld.exe
       cmd = \"C:/Program Files (x86)/Meld/Meld.exe\" "$LOCAL" "$MERGED" "$REMOTE" --output "$MERGED"
       trustExitCode = true
       keepBackup = false
[user]
	email = sflee1112@gmail.com
	name = SulfredLee

How cmake find package works

 Aim

Here are some examples showing how to use cmake find packet.

 Learn from

Here

Examples - use default modules

ALSA library

You can install the library first.

$ sudo apt-get install libasound2

You can put the following code into CMakeLists.txt.

# Handle ALSA / ASound libraries
find_package(ALSA)
if (ALSA_FOUND)
  message (STATUS "Info - Found ALSA library: ${ALSA_LIBRARY}")
  message (STATUS "Info - Found ALSA include: ${ALSA_INCLUDE_DIR}")
else (ALSA_FOUND)
  message (FATAL_ERROR "Info - ALSA is not found")
endif (ALSA_FOUND)

How can I know which variables I can use?

• Go to official web site 001 or official web site 002
  
• In ubuntu you can go to the installation path and find all the modules
  
• /usr/share/cmake-3.16/Modules
• FindALSA.cmake
  

 

Install xrdp on ubuntu

 Ubuntu 20.04

Install and use (Basic)

• Install xrdp by

$ sudo apt-get install xrdp

• Logout, since you cannot remote login while the session is begin used.
  

• Connect from other PC, for example:

 

• DONE

If not working

• By default Xrdp uses the /etc/ssl/private/ssl-cert-snakeoil.key file which is readable only by users that are members of the “ssl-cert” group. Learn From
  
• Add to group ssl-start.
  

$ sudo adduser xrdp ssl-start

Change close lip behavior

• Learn From

• Restart service to activate your change

 $ sudo systemctl systemd-logind.service

Core dump file in ubuntu

 Aim

When running c++ application, we would like to get the core dump when the application is crashed. How to get those files under linux?

Learn from

Link

Steps

Add to ~/.bashrc 

ulimit -c unlimited

Add to /etc/sysctl.conf

kernel.core_pattern = core.%e.%p.%t

Run command:

$ sudo sysctl --system

Ubuntu KVM guest os network

 On Ubuntu 20.04

You can use the default setting first:

If the guest OS cannot connect to internet, you may try the host nic and bridge it:

If that is not okay, you may need to create a new network source. The easiest way is to install VMWare workstation or Virtual box. They will create the needed network configuration for their VMs. Now you can use the network source created by them. Here is an example:

Ubuntu KVM OpenGL

 On Ubuntu 20.04

Setup

This acceleration can be use only locally. You have to disable the spice server first.

Then you can start your VM through graphical console

Ubuntu KVM multiple monitors

 On Ubuntu 20.04

Learn From

• Link

 Steps

Add one more Video QXL 

 

Set Display Spice and open port for example 5907

 Start VM without using virt-manager graphical console

Finally, you need to use remote-viewer and open up the guest firewall:

You may try directly remote-view the instance first, if not work, you may add the firewall steps

$ sudo apt-get install firewalld
$ sudo firewall-cmd --permanent --add-service=vdsm
$ sudo firewall-cmd --reload
$ remote-viewer spice://localhost:5907

Screenshot tools

 Windows

 Install Greenshot

Ubuntu

Install Flameshot

Steps

Install applications

sudo apt-get install flameshot

Disable default shortcut

 

Add shortcut

Update Filename

Concurrency message queue in C++

Aim

In this post, I would like to see how to build a concurrency message queue in C++ that is fast. The fastest message queue I can make at this moment is the Lock Free Ring Buffer message queue. There is more details in the result part.

Source

github

Testing method

There is a speed test framework implemented in SpeedTest class. No matter what type of message queue we are using, a number of message will be pushed into the MsgQ by multiple number of threads at the same time there are the same number of threads reading message from that MsgQ.

We then have a timer to count the total time that is needed to process all the messages.

We have different type of message queue:

Lock Free MsgQ

1. MsgQLockFree with Ring Buffer

Using Linux Concurrency library

1. MsgQOneLock with Ring Buffer
2. MsgQTwoLock with Ring Buffer
3. MsgQOneLock with LinkList 
4. MsgQTwoLock with LinkList

Using C++ standard library

1. MsgQOneLock with Ring Buffer
2. MsgQTwoLock with Ring Buffer
3. MsgQOneLock with LinkList 
4. MsgQTwoLock with LinkList

As one can see that there are currently 3 types of message queue:

1. Lock free
2. One lock
3. Two lock

We would like to see their difference in speed.

More on Queue

Lock Free

I learn this from here. I made a simplified version. This is a concurrency message queue but only support 2 threads. 1 thread is pushing message and 1 thread is getting message.

In this message queue we have no mutex lock. There are 2 pointers, one is pointing to the next empty slot(Tail) and the other is pointing to the next ready message slot(Head). The length of the message queue is a power of 2. Since there is no mutex lock, __sync_val_compare_and_swap() is used.

One Lock

The other type of message queue is a queue that is using one lock.

There is one mutex protecting the Head and Tail pointers. More than 2 threads can be supported.

Two Lock

There are two mutex protecting the Head and Tail pointers. More than 2 threads can be supported.

The Test

Hardware

• Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
• 4 core
• 4 threads
• Ubuntu 20.04 LTS

Test Info

• 300000 messages
• Threads
• 1 in, 1 out
• 2 in , 2 out
• 4 in, 4 out
• 10 in, 10 out
• 5 total tests and get the average time spend

Result

MsgQ Type	Thread IN	Thread OUT	Total Message	Total test	Time spend Msec
Lock Free	1	1	300000	5	46
One Lock	1	1	300000	5	115
Two Lock	1	1	300000	5	102.5

The above grape comparing 1 thread in and 1 thread out case. The lock free message queue is the fastest one. This is expected.

Thread IO	Total Message	Total test	Msec One Lock	Msec Two Lock
Two	300000	5	122	135.5
Four	300000	5	133.5	119
Ten	300000	5	150.5	124

The above graph shows that the speed of MsgQ with 2 locks is better. We have 2 threads IN and OUT, 4 threads and 10 threads in these tests.

Raw Result

MsgQLockFree_Linux	1	1	300000	5	46
MsgQLockFree_LinkList_Std	1	1	300000	5	100
MsgStackLockFree_LinkList_Std	1	1	300000	5	296
MsgStackLockFree_LinkList_Std	2	2	300000	5	406
MsgStackLockFree_LinkList_Std	4	4	300000	5	806
MsgStackLockFree_LinkList_Std	10	10	300000	5	568
MsgQOneLock_RingBuf_Linux	1	1	300000	5	111
MsgQOneLock_RingBuf_Linux	2	2	300000	5	109
MsgQOneLock_RingBuf_Linux	4	4	300000	5	135
MsgQOneLock_RingBuf_Linux	10	10	300000	5	186
MsgQTwoLock_RingBuf_Linux	1	1	300000	5	86
MsgQTwoLock_RingBuf_Linux	2	2	300000	5	135
MsgQTwoLock_RingBuf_Linux	4	4	300000	5	132
MsgQTwoLock_RingBuf_Linux	10	10	300000	5	128
MsgQOneLock_RingBuf_Std	1	1	300000	5	119
MsgQOneLock_RingBuf_Std	2	2	300000	5	137
MsgQOneLock_RingBuf_Std	4	4	300000	5	106
MsgQOneLock_RingBuf_Std	10	10	300000	5	115
MsgQTwoLock_RingBuf_Std	1	1	300000	5	100
MsgQTwoLock_RingBuf_Std	2	2	300000	5	136
MsgQTwoLock_RingBuf_Std	4	4	300000	5	106
MsgQTwoLock_RingBuf_Std	10	10	300000	5	120
MsgQOneLock_LinkList_Linux	1	1	300000	5	229
MsgQOneLock_LinkList_Linux	2	2	300000	5	201
MsgQOneLock_LinkList_Linux	4	4	300000	5	195
MsgQOneLock_LinkList_Linux	10	10	300000	5	197
MsgQTwoLock_LinkList_Linux	1	1	300000	5	179
MsgQTwoLock_LinkList_Linux	2	2	300000	5	268
MsgQTwoLock_LinkList_Linux	4	4	300000	5	225
MsgQTwoLock_LinkList_Linux	10	10	300000	5	213
MsgQOneLock_LinkList_Std	1	1	300000	5	250
MsgQOneLock_LinkList_Std	2	2	300000	5	189
MsgQOneLock_LinkList_Std	4	4	300000	5	209
MsgQOneLock_LinkList_Std	10	10	300000	5	223
MsgQTwoLock_LinkList_Std	1	1	300000	5	166
MsgQTwoLock_LinkList_Std	2	2	300000	5	267
MsgQTwoLock_LinkList_Std	4	4	300000	5	212
MsgQTwoLock_LinkList_Std	10	10	300000	5	219
MsgQOneLock_Queue_Linux	1	1	300000	5	203
MsgQOneLock_Queue_Linux	2	2	300000	5	158
MsgQOneLock_Queue_Linux	4	4	300000	5	152
MsgQOneLock_Queue_Linux	10	10	300000	5	172
MsgQOneLock_Queue_Std	1	1	300000	5	178
MsgQOneLock_Queue_Std	2	2	300000	5	151
MsgQOneLock_Queue_Std	4	4	300000	5	150
MsgQOneLock_Queue_Std	10	10	300000	5	154

List of time complexity for standard data structure

CPP

Container	Insert	Append	Access
Vector, String	O(n)	Back: O(1) or O(n)	O(1)
Dequeue	O(n)	Front or Back: O(1)	O(1)
List	O(1)or O(n)	O(1)	O(1) or O(n)
Set, Map	O(logN)	-	O(logN)
unordered_set, unordered_map	O(1) or O(n)	-	O(1) or O(n)

Python

Python List ~ CPP List

Python Dict ~ CPP unordered_map

Python dequeue ~ CPP Dequeue

Big O cheat sheet

link

PcapReplayer

Aim

This is an application build for replay .pcal files for network debugging.

This application can:

• Run on Ubuntu
• Run on Windows
• Replay a list of pcap files, drag and drop
• Schedule a replay event
• Replay at a customized speed, change speed anytime
• Pause in the middle of the replay process
• Know current replay status
• Select different interface
• Loop replay

Source

github

Features

Run on Ubuntu

Download the binaries from here.

• $ export LD_LIBRARY_PATH=/usr/local/yourLib/lib
• $ sudo ./bin/PcapReplayer

Run on Windows

Download the binaries from here.

Replay a list of pcap files, Drag and Drop

Schedule a reply report

Replay at a customized speed, change speed anytime

Know the current replay status

Select different interface

Loop Replay

You can replay the same list again and again.

Inherit example in C

Aim

In this example we simulate inheritance in C.

Source

github

#include <iostream>
#include "allFood.h"

int main(int argc, char *argv[])
{
    food* pApple = createApple();
    ((apple*)pApple)->nNumber = 10;
    pApple->printName("001");
    fprintf(stdout, "We have %d apples.\n", ((apple*)pApple)->nNumber);
    food* pOrange = createOrange();
    pOrange->printName("002");
    return 0;
}

Output

$ ./src/Inherit_C 
I am an apple 001
We have 10 apples.
I am an orange 002