Usm gps 2 EN

en | ru

This module is a replacement for the deprecated us_gps module

Description

usm_gps_2 - A module that can receive information from GPS trackers and monitoring programmes to record the position of personnel and vehicles. Information about the position of these objects and their routes can help in faster solution of emergency situations, as these objects are displayed on the coverage map and you can quickly understand which employee, what is nearby should be sent to solve problems.

The second version of the module differs from the first version in its internal structure. Now for each of GPS protocols runs its own independent "listener", and as a network core uses a powerful asynchronous superserver, capable of processing hundreds of thousands of TCP-connections simultaneously. This makes the module quite stable and reliable.

Agreement

We do not position this module as a vehicle monitoring system and therefore you should not expect such functionality from the module. Its main purpose is to display the current location of an employee or vehicle, not to monitor mileage, fuel consumption, etc.

Requirements

Pre-release version (alpha, beta) requires USERSIDE version 3.12.24 or higher.

Release version (RC, release) requires USERSIDE version no lower than 3.13 (still under development).

Supported GPS tracker protocols

The following protocols are currently supported. Please note that the list of supported devices may not be complete, as we are not aware of all devices manufactured with support for this or that protocol. It is also important to know that the device model does not necessarily have to be similar to the protocol that is actually flashed in this device.

OsmAnd

Default port: 5055

Supported devices: This protocol is mainly used by mobile applications such as OsmAnd, GpsLogger for Android, SendLocation, Locus Pro Android, Custodium, Traccar and others.

Since this protocol is based on an HTTP request with all data being transmitted through the parameters of this request, it can be used in any application. To transfer coordinates, it is enough to execute an HTTP request: http://module.ip.address:5055/?id={id}&lat={lat}&lon={lon}&timestamp={timestamp}&hdop={hdop}&speed={speed} where the following variables are substituted for the required ones:

• {id} - device identifier (IMEI, serial number or any other that allows to identify a particular tracker)
• {lat} - geographic latitude in decimal degree format, for example, 46.488085
• {lon} - is geographic longitude in decimal degrees, for example, 30.741091.
• {timestamp} - integer timestamp in UNIXTIME format
• {hdop} - the precision reduction coefficient in the horizontal positioning plane (HDOP). A floating point number. The parameter can be left out.
• {speed} is the instantaneous speed in km/h. Floating point number.

For example, for GpsLogger, you should enter the following address in the CustomURL setting: http://module.ip.address:5055/?id=%ser&lat=%lat&lon=%lon&timestamp=%timestamp&hdop=%hdop&speed=%spd.

GT02A

Default port: 5022

Supported devices: This protocol is implemented in a large number of Chinese devices, so it is impossible to make an exact list of them. However, it is one of the most common protocols found in cheap noname GPS trackers.

Messages of this protocol begin with hh header characters or as HEX: 0x68 0x68.

GT06

Default port: 5023

Supported devices: GT06, GT06N, GT09, Heacent 908, GT03A, GT03B, GS503, ET100, GT100, GT06D, GK301, JM01, JM08, GT02D, IB-GT102, CRX1, JV200, TP06A, BW08, TR06, JI09, Concox GT300, WeTrack 2

Messages of this protocol begin with xx header characters or as HEX: 0x78 0x78.

Meiligao

Default port: 5009

Supported devices: GT30i, GT60, VT300, VT310, VT400, GT30, GT30X, PST-AVL01, PT03, PT60, PT300X, PT30, GT-110P, GT-110K, GT-110M, GT-110ES, GT-110ZS, AVL-011, VT900, P008, GT 30, CT01, CT03, CT04, CT04-R, CT04-X, OCT600, MT01, MT02, PT01, PT03, VT1000, GSY007, T200, iStartek, VT310N

Messages of this protocol begin with $$ header characters or as HEX: 0x24 0x24 0x24.

Autofon v4

Default port: 5079

Supported devices: AutoFon 4.4, AutoFon 4.5

Autofon v5

Default port: 5077

Supported devices: AutoFon SE, AutoFon SE+, AutoFon D, AutoFon D-Moto, AutoFon Dialog Mayak, AutoFon S, AutoFon GL Mayak

Autofon v7

Default port: 5099

Supported devices: AutoFon Alpha, AutoFon Alpha XL, AutoFon Alpha 2XL

Autofon v9 (Micro-Mayak)

Default port: 9109'

Supported devices: Autofon Mikro-Mayak, Autofon Mikro-Mayak+.

Starting from module version 2.3

Wialon

Default port: 5039

Supported devices: Wialon IPS, MasterKit, MasterKit BM8009, NeoTech TR1000, and also uonline.com.ua service transmits navigation information via this protocol (if you have trackers connected to this service, it can act as a proxy server, transmitting information to usm_gps_2 module).

Messages of this protocol begin with a # header character or as HEX: 0x23.

Important! Mobile application GpsTag from Gurtam (Wialon protocol developer) is not supported (Gurtam developers specially changed the protocol so that the free application could be used only on Gurtam servers).

OKO-NAVI

Default port: 5098

Supported devices: All devices of the company OKO

The messages of this protocol are enclosed in curly braces {...} or as HEX: 0x7b ... 0x7d.

MikroTik

Default port: 5200

Supported devices: All devices equipped with GSP module, e.g. LtAP mini The protocol is based on sending JSON data in the body of an HTTP-POST request:

{
  "tracker_id": "12345678",
  "lat": "N 12 34' 5.678",
  "lon": "E 87 65' 4.321",
  "speed": "1.234567 km/h",
  "heading": "128.039993 deg. True",
  "hdop": 122
}

Therefore, it can be used not only for MikroTik.

The module is designed to receive coordinates in DMS format - this mode is enabled in MikroTik by default. If you have changed the coordinate format before, you need to switch to this format with the command:

/system gps set coordinate-format=dms

A simple script to be added to the MikroTik scheduler by first changing the values of the first three variables:

{
:local trackerID "12345678" 
:local url "http://usm_gps-url-here/" 
:local port 5200
:local lat
:local lon
:local spd
:local heading
:local hdop
/system gps monitor once do={
    :set $lat $("latitude")
    :set $lon $("longitude")
    :set $spd $("speed")
    :set $heading $("true-bearing")
    :set $hdop $("horizontal-dilution")
}

/tool fetch mode=http url=$url port=$port http-method=post http-header-field="Content-Type: application/json" \
http-data=("{\"tracker_id\": \"".$trackerID."\",\"lat\": \"".$lat."\",\"lon\": \"".$lon."\",\"speed\": \"".$spd."\",\"heading\": \"".$heading."\",\"hdop\": 
\"".$hdop."\"}")
}

The variable trackerID contains the unique identifier of this Microtik within USERSIDE. Make it up yourself. In classic GPS trackers the identifier is sewn into each tracker and is usually all or part of IMEI or serial number of the tracker. But since Microtik doesn't have any such identifiers, you have to implement it yourself. It can be any character string, not necessarily consisting of numbers.

You can also refine this script so that, for example, the identifier is the MAC address of the first interface (not to enter it manually, if it is easier), and you can also refine the script to store the previous state in global variables and send information to the server only if the new values differ from the previous ones - this will save traffic and also get rid of "blots" on parking places when the speed is 0 km/h.

H02

Default port: 5013

Supported devices: H02, H-02A, H-02B, TX-2, H-06, H08, GTLT3, TK110, TK102B-MT61, NT201, NT202, S31, LK109, LK106, LK208, LK206, LK310, LK206A, LK206B, MI-G6, CC830, CCTR, CCTR-630, AT-18, GRTQ, LK210, TR-02

Both text and binary protocols are supported (from module version 2.3 onwards).

Text protocol messages begin with a *HQ header characters or as HEX: 0x2A 0x48 0x51.

Binary protocol messages begin with a $ header character or as HEX: 0x24.

GPS103 (GPS102, Coban)

Default port: 5001

Supported devices: TK103-B, TK103-2B, TK104, TK106, GPS-103, GPS-103-A, TW-MD1101, GPS102B, GPS104, TK110

Messages of this protocol begin with a ## header characters or as HEX: 0x68 0x68.

TK103

Default port: 5002

Supported devices: EC-546, TT0024, T1024, T1080, T2024, T2124, T12, T4400, T8800, T15400, TK05, TK10, TK15, TK20, TK110, T18, T18H, T16, GPS105, P168

The messages of this protocol are enclosed in parentheses (...) or as HEX: 0x28 ... 0x29.

Teltonika

Default port: 5027

Supports various devices using codecs: 8, 8 Extended

Messages of this protocol begin with HEX: 0x00.

Navtelecom (NTCB+FLEX)

Default port: 9000

Supported devices: All Navtelecom devices

Messages of this protocol begin with a @NTC header characters or as HEX: 0x40 0x4e 0x54 0x43.

TEST

Default port: 9999

The protocol is required to take samples from an unknown tracker in order to submit them in a request for a new protocol design, or to determine the actual protocol of a device based on header (and tail) characters.

Configure your tracker on port 9999 and watch the log. In it will appear lines with the mark RX-SAMPLE: - this is the data in hex form, which is sent by your tracker. Try to find a suitable protocol with 2..3 header characters (in hex format) among the protocols described above. If you can't find it, write to us. Perhaps the protocol for your tracker is not implemented in Userside yet.

Installing the module

Uploading

Download the module from your personal cabinet from the downloads page: https://my.userside.eu/soft/downloads.

Unzip the contents of the archive to, for example, the /opt/usm directory. Eventually in the /opt/usm, a usm_gps directory will be formed containing the module files. In further instructions it is the /opt/usm/usm_gps directory that will be used. If you have placed the files in a different location, take this into account when following the instructions.

Stopping and deleting an old version

If you were using an older (first) version of the module before, stop it and delete all crontab entries to start the module.

Installing Python3 and dependencies

The module requires Python3 with a version of at least 3.7. It is preferable to have the latest version. It is also recommended to use the venv package to create a virtual Python environment and all dependencies needed by the module, so as not to clog up the operating system.

If you have an older version for some reason, then install the latest version. Also install the pip3 package manager and the venv package to implement the virtual environment:

sudo apt install -y python3 python3-dev python3-venv python3-pip

For controlled operation of the module it is recommended to use supervisor. If you have not installed it yet (it is required for USERSIDE 3.17 and newer), then install it:

sudo apt install -y supervisor

Go to the module directory and create a virtual environment there:

cd /opt/usm/usm_gps
sudo python3 -m venv venv
sudo ./venv/bin/pip install --upgrade pip setuptools

Now install the dependencies for the module:

sudo ./venv/bin/pip install --upgrade -r requirements.txt

Now perform the configuration and the first test run of the module not in the background. More details about running the module below in First Run and Debugging.

Setup

If this is the first installation of the usm_gps_2 module, you will need to copy the .env-example configuration file under the new name .env:

 cp .env-example .env

If you are upgrading the usm_gps module, you will probably need to move the settings from the settings.ini file to the .env file.

Instead, you can use the environment variables described in the .env-example file in the environment itself.

Environment variables are prefixed (following between USM_GPS__ and the following __).

API prefix

This block contains settings for connecting to the core USERSIDE via API. You need to specify the URL of your USERSIDE, as well as the KEY you specified in the configuration file (see: UserSide API Key). If your USERSIDE uses an https connection with a self-signed certificate, set the SSL_VERIFY option to False.

The TRACKERS_AUTH_REQUIRED option determines whether trackers need to be authenticated on the server. If it is set to True, trackers with serial numbers (or IMEI) missing in USERSIDE will be denied authorisation and their data will not be added to the database. This is the recommended mode necessary to filter out trackers that are not registered in the system. If this behaviour is not suitable for you, you can disable tracker authentication by setting this option to False.

Prefix PROC (processing)

This block contains settings of the position handler. The handler contains three filters that will help to exclude presumably incorrect data. These filters should be configured individually. You should also be aware that the filters ignore some geographic positions based on their analysis, which can result in loss of normal data if not configured correctly. Be careful when selecting parameters.

The HDOP_THRESHOLD option defines the threshold value for the horizontal positioning accuracy loss factor (HDOP) value. The default preset value is 10. All navigation data received from the tracker with an HDOP value above the threshold will be discarded and will not be transmitted to USERSIDE. You can adjust this threshold value to your preference, or disable it by setting it to 100. The following HDOP values are considered standard: <1 - Perfect, 1...2 - Excellent, 2...5 - Good, 5...10 - Average, 10...20 - Poor, >20 - Very poor. To ensure that the database contains only points with normal accuracy, set the threshold value to 5. It should be understood that if the coordinates were determined with an error exceeding the specified accuracy loss factor, these coordinates will be discarded (the log file will contain a corresponding record about it). If the GPS tracker protocol does not provide for calculation and transmission of the accuracy loss factor value to the server, this value is taken as 0.

Parking filter (GPS drift)

This filter is designed to detect and filter out GPS drift during parking. When the vehicle is parked and the GPS tracker is working and transmitting the position to the server, you can see the following on the map:

GPS receivers have an average position error of about 6 metres. Each time the GPS receiver tries to determine its position, it will be determined with an accuracy of about 6 metres, regardless of the previous position (GPS receivers do not store status). The reason for this behaviour may be uncertain satellite reception, interference (the car is located in a dense multi-storey building), or partial visibility of the entire available satellite constellation. For example, if the antenna is located inside the car, it "sees" only part of the sky. This is quite a normal situation for civil GPS. To detect parking and reject such positions, the parking filter is used. It can be enabled by the parameter PARKED_FILTER_ENABLE set to True. This filter starts working if the previous and current positions had a speed value not higher than "close to zero" - it is set in the parameter PARKED_SPEED_THRESHOLD, and if the distance between the previous and current position is not more than "position deviation" - it is set in the parameter PARKED_POSITION_DEVIATION (on average it is 6 metres). As soon as the car starts moving (moves to a distance greater than the "position deviation" or develops a speed greater than "close to zero"), the filter will stop working and all positions will be transmitted to the server. To roughly determine the "position deviation", simply measure the length of the average bar on the map with a ruler. Decrease and increase this value for best effect. Remember that values that are too large can also screen out the desired positions.

To improve the quality of geographical positioning, use modern GNSS receivers capable of working with at least two GNSS systems simultaneously (GPS, Galileo, Glonass) and external antennas fixed in such a way that the sky is not blocked even partially by the car body.

Departures filter

This filter is designed to detect and filter out erroneous track departures. On a map it may look like this:

This can sometimes happen due to poor reception, interference or just a poor quality receiver. It looks like a point away from the track at a distance that the vehicle could not have reached in any way. The Departure Filter compares the actual distance between the previous point and the current point and determines if that distance could have been reached by the vehicle travelling at the maximum speed of those two points. To enable the departure filter, set the RANGE_FILTER_ENABLE parameter to True. This filter also has several settings. The first setting RANGE_DELTA_TIME - is the maximum time in seconds between the points for which the filter will work. There is no point in using the filter if too much time has passed between points, as the vehicle between two points may have moved faster than it is recorded at those points, and the filter may already be inaccurate or even wrong. The second setting RANGE_DISTANCE_FACTOR - is the coefficient of the estimated path length, to which the actual distance travelled is compared. This factor allows you to slightly "lengthen" the estimated path value to eliminate the error that occurs at the boundaries of the calculation. The default coefficient is 1.1 and the maximum time between points is 60 seconds, but it is up to you to adjust it more correctly when you enable the filter.

Remember - the filter filters out geographic positions without knowing anything about future positions! Be careful when using it. If you have problems with coordinates detection in Userside, disable this filter first of all while performing diagnostics.

LOG prefix

This block contains the settings for event logging.

The FILE option defines the path to the directory where the module log file will be placed. The STDOUT option can be used to output messages to the console instead of outputting them to a file. The LEVEL option defines the level of logging. The default is INFO (possible are DEBUG, WARNING, ERROR, CRITICAL).

Prefix PROTO

This prefix block lists the TCP port numbers on which the protocol listeners reside. You can override the port numbers at your discretion, but keep in mind that two different protocols cannot occupy the same port.

You can also comment out unused ports using # symbols to disable the protocols. For example, if all your trackers communicate using the same protocol.

First run and debugging

If this is the first time you are running the usm_gps_2 module, and if you are having trouble getting it to work, follow these steps.

• In the .env file, set the USM_GPS__LOG__FILE variable to STDOUT:

USM_GPS__LOG__FILE=STDOUT

• Also, in the USM_GPS__PROTO__ prefix block, uncomment the protocol(s) you need.

• Start the module manually:

sudo venv/bin/python3 main.py

• You should see the following in the console:

2018-04-25 12:37:16,604 | INFO     | *** Module usm_gps version 2.4.0 is starting...
2018-04-25 12:37:16,801 | INFO     | GT02A protocol listener started at port 8822
2018-04-25 12:37:16,802 | INFO     | GT06 protocol listener started at port 8833
2018-04-25 12:37:16,802 | INFO     | MEILIGAO protocol listener started at port 8844
2018-04-25 12:37:16,803 | INFO     | AUTOFON4 protocol listener started at port 8854
2018-04-25 12:37:16,804 | INFO     | AUTOFON5 protocol listener started at port 8855
2018-04-25 12:37:16,804 | INFO     | AUTOFON7 protocol listener started at port 8856
2018-04-25 12:37:16,805 | INFO     | WIALON protocol listener started at port 8866
2018-04-25 12:37:16,806 | INFO     | OSMAND (Traccar) protocol listener started at port 5055
2018-04-25 12:37:16,807 | INFO     | *** Module usm_gps was started and ready to receive connections. PID=13878

• Wait until the trackers start connecting to the module. You will see information about this process in the console. If the connection is correct, trackers are authorised correctly, trackers start sending information to USERSIDE correctly, we can consider the test run a success. If there are any problems at this stage, please contact the support team by creating a ticket or ask the community in our Telegram group.

• Stop the module with ctrl+c and specify the log file name for the USM_GPS__LOG__FILE variable:

USM_GPS__LOG__FILE=/var/log/userside/usm_gps/usm_gps.log

The module is now ready to be used.

Supervisor

To automatically monitor the module's operation and restart it in case of problems, we recommend using supervisor. A sample config file for supervisord is supplied with the module - it is etc/supervisor/conf.d/usm_gps.conf-example. Once you are in the directory with the module, copy the config to your file system, keeping the path but with a different name (without -example):

sudo cp etc/supervisor/conf.d/usm_gps.conf-example /etc/supervisor/conf.d/usm_gps.conf

By default, the example contains the same paths as in the instructions, but if you used a different directory to host the module, change the paths to the appropriate ones.

Restart supervisor to have it reread the configuration files:

sudo systemctl restart supervisor

Now you can see what state the module is in:

sudo supervisorctl status usm_gps

Also keep an eye on the module and supervisor logs.

If you need to stop the module, e.g. for updating, use the command:

sudo supervisorctl stop usm_gps

To start the module again:

sudo supervisorctl start usm_gps

Log file rotation

The log files created by the module need to be rotated. To do this, the standard log rotation tools in the operating system should be used.

The following configuration will rotate all *.log files every day and store 7 archive copies (for 7 days)

Rotation in FreeBSD

On FreeBSD, the newsyslog daemon is responsible for rotation. Create the file /usr/local/etc/newsyslog.conf.d/userside, in which place the following line (don't forget the path if you changed it):

/var/log/userside/*.log         644 7    *  @T00  GJC    /path_to_module/usm_gps_2/usm_gps.pid

Rotation in Linux

In Linux, the logrotate daemon is responsible for rotation. Create a file /etc/logrotate.d/userside, in which put the following text (don't forget the path if you change it):

/var/log/userside/*.log {
    rotate 7
    daily
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Upgrading the module

• Download the archive with the new version from your personal cabinet to the server where the module is running
• Stop the module

 sudo supervisorctl stop usm_gps

• Delete all module files, except settings.ini
• Unzip the files from the archive with the new version in place of the old files.
• Be sure to install (update) the dependencies:

 sudo ./venv/bin/pip install --upgrade -r requirements.txt

• Start the module

 sudo supervisorctl start usm_gps

Requests to implement a new protocol

Trackers of the same model may have different telecommunication protocols for communication with the server. This is especially true for cheap Chinese trackers like GT06. The tracker hardware may be manufactured by the same factory, and the customer vendor may install its own firmware in the tracker.

Therefore, we don't use the concept of tracker model support, but use the concept of protocol support.

If you have purchased a tracker, the first thing to find out is what protocol it supports. Usually this information can be found in the device description. If there is no such information, you can try to find the protocol by tracker model in the list above. Enable all protocols available in the module and restart it. Then change the port in the tracker settings so that different protocols of the module are used. If a suitable protocol is found, you are in luck. Use it. If not, you can use the special protocol "test" (default port 9999), which simply logs the packet received from the tracker in hexadecimal representation. Next you need to create a ticket through the ticket system with the subject usm_gps protocol support.... In the body of the ticket, describe your tracker and attach the module lines starting with RX-SAMPLE:.

If you have information about the protocol you want to add to the module, as well as any other information (documentation or where to get it) - be sure to attach all available information to the ticket. This will greatly increase the chances of faster development for you.