Usm gps 2 EN
This module is a replacement for the deprecated us_gps module
Description
usm_gps_2 - A module that can receive information from GPS trackers and monitoring programmes to record the position of personnel and vehicles. Information about the position of these objects and their routes can help in faster solution of emergency situations, as these objects are displayed on the coverage map and you can quickly understand which employee, what is nearby should be sent to solve problems.
The second version of the module differs from the first version in its internal structure. Now for each of GPS protocols runs its own independent "listener", and as a network core uses a powerful asynchronous superserver, capable of processing hundreds of thousands of TCP-connections simultaneously. This makes the module quite stable and reliable.
Agreement
We do not position this module as a vehicle monitoring system and therefore you should not expect such functionality from the module. Its main purpose is to display the current location of an employee or vehicle, not to monitor mileage, fuel consumption, etc.
Requirements
Pre-release version (alpha, beta) requires USERSIDE version 3.12.24 or higher.
Release version (RC, release) requires USERSIDE version no lower than 3.13 (still under development).
Supported GPS tracker protocols
The following protocols are currently supported. Please note that the list of supported devices may not be complete, as we are not aware of all devices manufactured with support for this or that protocol. It is also important to know that the device model does not necessarily have to be similar to the protocol that is actually flashed in this device.
OsmAnd
Default port: 5055
Supported devices: This protocol is mainly used by mobile applications such as OsmAnd, GpsLogger for Android, SendLocation, Locus Pro Android, Custodium, Traccar and others.
Since this protocol is based on an HTTP request with all data being transmitted through the parameters of this request, it can be used in any application. To transfer coordinates, it is enough to execute an HTTP request:
http://module.ip.address:5055/?id={id}&lat={lat}&lon={lon}×tamp={timestamp}&hdop={hdop}&speed={speed}
where the following variables are substituted for the required ones:
- {id} - device identifier (IMEI, serial number or any other that allows to identify a particular tracker)
- {lat} - geographic latitude in decimal degree format, for example, 46.488085
- {lon} - is geographic longitude in decimal degrees, for example, 30.741091.
- {timestamp} - integer timestamp in UNIXTIME format
- {hdop} - the precision reduction coefficient in the horizontal positioning plane (HDOP). A floating point number. The parameter can be left out.
- {speed} is the instantaneous speed in km/h. Floating point number.
For example, for GpsLogger, you should enter the following address in the CustomURL setting: http://module.ip.address:5055/?id=%ser&lat=%lat&lon=%lon×tamp=%timestamp&hdop=%hdop&speed=%spd.
GT02A
Default port: 5022
Supported devices: This protocol is implemented in a large number of Chinese devices, so it is impossible to make an exact list of them. However, it is one of the most common protocols found in cheap noname GPS trackers.
Messages of this protocol begin with hh header characters or as HEX: 0x68 0x68.
GT06
Default port: 5023
Supported devices: GT06, GT06N, GT09, Heacent 908, GT03A, GT03B, GS503, ET100, GT100, GT06D, GK301, JM01, JM08, GT02D, IB-GT102, CRX1, JV200, TP06A, BW08, TR06, JI09, Concox GT300, WeTrack 2
Messages of this protocol begin with xx header characters or as HEX: 0x78 0x78.
Meiligao
Default port: 5009
Supported devices: GT30i, GT60, VT300, VT310, VT400, GT30, GT30X, PST-AVL01, PT03, PT60, PT300X, PT30, GT-110P, GT-110K, GT-110M, GT-110ES, GT-110ZS, AVL-011, VT900, P008, GT 30, CT01, CT03, CT04, CT04-R, CT04-X, OCT600, MT01, MT02, PT01, PT03, VT1000, GSY007, T200, iStartek, VT310N
Messages of this protocol begin with $$ header characters or as HEX: 0x24 0x24 0x24.
Autofon v4
Default port: 5079
Supported devices: AutoFon 4.4, AutoFon 4.5
Autofon v5
Default port: 5077
Supported devices: AutoFon SE, AutoFon SE+, AutoFon D, AutoFon D-Moto, AutoFon Dialog Mayak, AutoFon S, AutoFon GL Mayak
Autofon v7
Default port: 5099
Supported devices: AutoFon Alpha, AutoFon Alpha XL, AutoFon Alpha 2XL
Autofon v9 (Micro-Mayak)
Default port: 9109'
Supported devices: Autofon Mikro-Mayak, Autofon Mikro-Mayak+.
Starting from module version 2.3
Wialon
Default port: 5039
Supported devices: Wialon IPS, MasterKit, MasterKit BM8009, NeoTech TR1000, and also uonline.com.ua service transmits navigation information via this protocol (if you have trackers connected to this service, it can act as a proxy server, transmitting information to usm_gps_2 module).
Messages of this protocol begin with a # header character or as HEX: 0x23.
Important! Mobile application GpsTag from Gurtam (Wialon protocol developer) is not supported (Gurtam developers specially changed the protocol so that the free application could be used only on Gurtam servers).
OKO-NAVI
Default port: 5098
Supported devices: All devices of the company OKO
The messages of this protocol are enclosed in curly braces {...} or as HEX: 0x7b ... 0x7d.
MikroTik
Default port: 5200
Supported devices: All devices equipped with GSP module, e.g. LtAP mini The protocol is based on sending JSON data in the body of an HTTP-POST request:
{
"tracker_id": "12345678",
"lat": "N 12 34' 5.678",
"lon": "E 87 65' 4.321",
"speed": "1.234567 km/h",
"heading": "128.039993 deg. True",
"hdop": 122
}
Therefore, it can be used not only for MikroTik.
The module is designed to receive coordinates in DMS format - this mode is enabled in MikroTik by default. If you have changed the coordinate format before, you need to switch to this format with the command:
/system gps set coordinate-format=dms
A simple script to be added to the MikroTik scheduler by first changing the values of the first three variables:
{
:local trackerID "12345678"
:local url "http://usm_gps-url-here/"
:local port 5200
:local lat
:local lon
:local spd
:local heading
:local hdop
/system gps monitor once do={
:set $lat $("latitude")
:set $lon $("longitude")
:set $spd $("speed")
:set $heading $("true-bearing")
:set $hdop $("horizontal-dilution")
}
/tool fetch mode=http url=$url port=$port http-method=post http-header-field="Content-Type: application/json" \
http-data=("{\"tracker_id\": \"".$trackerID."\",\"lat\": \"".$lat."\",\"lon\": \"".$lon."\",\"speed\": \"".$spd."\",\"heading\": \"".$heading."\",\"hdop\":
\"".$hdop."\"}")
}
The variable trackerID contains the unique identifier of this Microtik within USERSIDE. Make it up yourself. In classic GPS trackers the identifier is sewn into each tracker and is usually all or part of IMEI or serial number of the tracker. But since Microtik doesn't have any such identifiers, you have to implement it yourself. It can be any character string, not necessarily consisting of numbers.
You can also refine this script so that, for example, the identifier is the MAC address of the first interface (not to enter it manually, if it is easier), and you can also refine the script to store the previous state in global variables and send information to the server only if the new values differ from the previous ones - this will save traffic and also get rid of "blots" on parking places when the speed is 0 km/h.
H02
Default port: 5013
Supported devices: H02, H-02A, H-02B, TX-2, H-06, H08, GTLT3, TK110, TK102B-MT61, NT201, NT202, S31, LK109, LK106, LK208, LK206, LK310, LK206A, LK206B, MI-G6, CC830, CCTR, CCTR-630, AT-18, GRTQ, LK210, TR-02
Both text and binary protocols are supported (from module version 2.3 onwards).
Text protocol messages begin with a *HQ header characters or as HEX: 0x2A 0x48 0x51.
Binary protocol messages begin with a $ header character or as HEX: 0x24.
GPS103 (GPS102, Coban)
Default port: 5001
Supported devices: TK103-B, TK103-2B, TK104, TK106, GPS-103, GPS-103-A, TW-MD1101, GPS102B, GPS104, TK110
Messages of this protocol begin with a ## header characters or as HEX: 0x68 0x68.
TK103
Default port: 5002
Supported devices: EC-546, TT0024, T1024, T1080, T2024, T2124, T12, T4400, T8800, T15400, TK05, TK10, TK15, TK20, TK110, T18, T18H, T16, GPS105, P168
The messages of this protocol are enclosed in parentheses (...) or as HEX: 0x28 ... 0x29.
Teltonika
Default port: 5027
Supports various devices using codecs: 8, 8 Extended
Messages of this protocol begin with HEX: 0x00.
Default port: 9000
Supported devices: All Navtelecom devices
Messages of this protocol begin with a @NTC header characters or as HEX: 0x40 0x4e 0x54 0x43.
TEST
Default port: 9999
The protocol is required to take samples from an unknown tracker in order to submit them in a request for a new protocol design, or to determine the actual protocol of a device based on header (and tail) characters.
Configure your tracker on port 9999 and watch the log. In it will appear lines with the mark RX-SAMPLE: - this is the data in hex form, which is sent by your tracker. Try to find a suitable protocol with 2..3 header characters (in hex format) among the protocols described above. If you can't find it, write to us. Perhaps the protocol for your tracker is not implemented in Userside yet.
Installing the module
Uploading
Download the module from your personal cabinet from the downloads page: https://my.userside.eu/soft/downloads.
Unzip the contents of the archive to, for example, the /opt/userside directory. Eventually in the /opt/userside, a usm_gps directory will be formed containing the module files. In further instructions it is the /opt/userside/usm_gps directory that will be used. If you have placed the files in a different location, take this into account when following the instructions.
Stopping and deleting an old version
If you were using an older (first) version of the module before, stop it and delete all crontab entries to start the module.
Installing Python3 and dependencies
The module requires Python3 with a version of at least 3.7. It is preferable to have the latest version. It is also recommended to use the venv package to create a virtual Python environment and all dependencies needed by the module, so as not to clog up the operating system.
If you have an older version for some reason, then install the latest version. Also install the pip3 package manager and the venv package to implement the virtual environment:
sudo apt install -y python3 python3-dev python3-venv python3-pip
For controlled operation of the module it is recommended to use supervisor. If you have not installed it yet (it is required for USERSIDE 3.17 and newer), then install it:
sudo apt install -y supervisor
Go to the module directory and create a virtual environment there:
cd /opt/userside/usm_gps sudo python3 -m venv venv sudo ./venv/bin/pip install --upgrade pip setuptools
Now install the dependencies for the module:
sudo ./venv/bin/pip install --upgrade -r requirements.txt
Now perform the configuration and the first test run of the module not in the background. More details about running the module below in First Run and Debugging.
Setup
If this is the first installation of the usm_gps_2 module, you will need to copy the settings.ini-example configuration file under the new name settings.ini:
cp settings.ini-example settings.ini
If you are updating the usm_gps module, check the settings.ini-example file for new parameters and add them to your settings.ini with the desired values.
Block api
This block contains settings for connecting to the core USERSIDE via API. You need to specify the url of your USERSIDE (without /oper/), as well as the key you specified in the configuration file (see: UserSide API Key). If your USERSIDE uses an https connection with a self-signed certificate, then set the ssl_verify option to no.
The trackers_auth_required option determines whether trackers need to be authenticated to the server. If it is set to yes, then trackers with serial numbers (or IMEI) missing in USERSIDE will be denied authorisation and their data will not be added to the database. This is the recommended mode necessary to filter out trackers that are not registered in the system. If this behaviour is not suitable for you, you can disable tracker authentication by setting this option to no.
Processing block
This block contains settings of the position handler. The handler contains three filters that will help to exclude presumably incorrect data. These filters should be configured individually. You should also be aware that the filters ignore some geographic positions based on their analysis, which can result in loss of normal data if not configured correctly. Be careful when selecting parameters.
The hdop_threshold option defines the threshold value for the horizontal positioning accuracy loss factor (HDOP) value. The default preset value is 10. All navigation data received from the tracker with an HDOP value above the threshold will be discarded and will not be transmitted to USERSIDE. You can adjust this threshold value to your preference, or disable it by setting it to 100. The following HDOP values are considered standard: <1 - Perfect, 1...2 - Excellent, 2...5 - Good, 5...10 - Average, 10...20 - Poor, >20 - Very poor. To ensure that the database contains only points with normal accuracy, set the threshold value to 5. It should be understood that if the coordinates were determined with an error exceeding the specified accuracy loss factor, these coordinates will be discarded (the log file will contain a corresponding record about it). If the GPS tracker protocol does not provide for calculation and transmission of the accuracy loss factor value to the server, this value is taken as 0.
Parking filter (GPS drift)
This filter is designed to detect and filter out GPS drift during parking. When the vehicle is parked and the GPS tracker is working and transmitting the position to the server, you can see the following on the map:
GPS receivers have an average position error of about 6 metres. Each time the GPS receiver tries to determine its position, it will be determined with an accuracy of about 6 metres, regardless of the previous position (GPS receivers do not store status). The reason for this behaviour may be uncertain satellite reception, interference (the car is located in a dense multi-storey building), or partial visibility of the entire available satellite constellation. For example, if the antenna is located inside the car, it "sees" only part of the sky. This is quite a normal situation for civil GPS. To detect parking and reject such positions, the parking filter is used. It can be enabled by the parameter parked_filter_enable = yes. This filter starts working if the previous and current positions had a speed value not higher than "close to zero" - it is set in the parameter parked_speed_threshold, and if the distance between the previous and current position is not more than "position deviation" - it is set in the parameter parked_position_deviation (on average it is 6 metres). As soon as the car starts moving (moves to a distance greater than the "position deviation" or develops a speed greater than "close to zero"), the filter will stop working and all positions will be transmitted to the server. To roughly determine the "position deviation", simply measure the length of the average bar on the map with a ruler. Decrease and increase this value for best effect. Remember that values that are too large can also screen out the desired positions.
To improve the quality of geographical positioning, use modern GNSS receivers capable of working with at least two GNSS systems simultaneously (GPS, Galileo, Glonass) and external antennas fixed in such a way that the sky is not blocked even partially by the car body.
Departures filter
This filter is designed to detect and filter out erroneous track departures. On a map it may look like this:
This can sometimes happen due to poor reception, interference or just a poor quality receiver. It looks like a point away from the track at a distance that the vehicle could not have reached in any way. The Departure Filter compares the actual distance between the previous point and the current point and determines if that distance could have been reached by the vehicle travelling at the maximum speed of those two points. To enable the departure filter, use the range_filter_enable = yes parameter. This filter also has several settings. The first setting range_delta_time' - is the maximum time in seconds between the points for which the filter will work. There is no point in using the filter if too much time has passed between points, as the vehicle between two points may have moved faster than it is recorded at those points, and the filter may already be inaccurate or even wrong. The second setting range_distance_factor - is the coefficient of the estimated path length, to which the actual distance travelled is compared. This factor allows you to slightly "lengthen" the estimated path value to eliminate the error that occurs at the boundaries of the calculation. The default coefficient is 1.1 and the maximum time between points is 60 seconds, but it is up to you to adjust it more correctly when you enable the filter.
Remember - the filter filters out geographic positions without knowing anything about future positions! Be careful when using it. If you have problems with coordinates detection in Userside, disable this filter first of all while performing diagnostics.
Log block
This block contains settings of event logging.
The path option defines the path to the directory where the module log file will be placed. The level option defines the logging level from the most detailed (for debugging) 1 to the "critical" 5. However, it is not recommended to use a journaling level higher than 3 in this module and it is better to stop at the default level 2.
The to_console option defines the purpose of outputting log events. If the option is enabled (set to yes), all log entries will be output to the console. This can be handy when debugging the module, and is also standard behaviour for running the module in a Docker container. The default behaviour is no.
Port block
This block lists the TCP port numbers on which the protocol listeners reside. You can override the port numbers at your discretion, but keep in mind that two different protocols cannot occupy the same port.
You can also comment out unused ports using the # or ; symbols to disable the protocols. For example, if all your trackers communicate using the same protocol.
First run and debugging
If this is the first time you are running the usm_gps_2 module, and if you are having trouble getting it to work, follow these steps.
- In the settings.ini file in the log block, set the following values:
[log] level = 1 to_console = yes
- Also in the ports block, uncomment the protocol(s) you need.
- Start the module manually:
sudo /opt/userside/usm_gps/venv/bin/python3 usm_gps.py
- You should see the following in the console:
2018-04-25 12:37:16,604 | INFO | *** Module usm_gps version 2.4.0 is starting... 2018-04-25 12:37:16,801 | INFO | GT02A protocol listener started at port 8822 2018-04-25 12:37:16,802 | INFO | GT06 protocol listener started at port 8833 2018-04-25 12:37:16,802 | INFO | MEILIGAO protocol listener started at port 8844 2018-04-25 12:37:16,803 | INFO | AUTOFON4 protocol listener started at port 8854 2018-04-25 12:37:16,804 | INFO | AUTOFON5 protocol listener started at port 8855 2018-04-25 12:37:16,804 | INFO | AUTOFON7 protocol listener started at port 8856 2018-04-25 12:37:16,805 | INFO | WIALON protocol listener started at port 8866 2018-04-25 12:37:16,806 | INFO | OSMAND (Traccar) protocol listener started at port 5055 2018-04-25 12:37:16,807 | INFO | *** Module usm_gps was started and ready to receive connections. PID=13878
- Wait until the trackers start connecting to the module. You will see information about this process in the console. If the connection is correct, trackers are authorised correctly, trackers start sending information to USERSIDE correctly, we can consider the test run a success. If there are any problems at this stage, please contact the support team by creating a ticket or ask the community in our Telegram group.
- Stop the module by pressing ctrl+c and restore the settings of the log block of the settings.ini file:
[log] level = 2 to_console = no
The module is now ready to be used.
Supervisor
To automatically monitor the module's operation and restart it in case of problems, we recommend using supervisor. A sample config file for supervisord is supplied with the module - it is etc/supervisor/conf.d/usm_gps.conf-example. Once you are in the directory with the module, copy the config to your file system, keeping the path but with a different name (without -example):
sudo cp etc/supervisor/conf.d/usm_gps.conf-example /etc/supervisor/conf.d/usm_gps.conf
By default, the example contains the same paths as in the instructions, but if you used a different directory to host the module, change the paths to the appropriate ones.
Restart supervisor to have it reread the configuration files:
sudo systemctl restart supervisor
Now you can see what state the module is in:
sudo supervisorctl status usm_gps
Also keep an eye on the module and supervisor logs.
If you need to stop the module, e.g. for updating, use the command:
sudo supervisorctl stop usm_gps
To start the module again:
sudo supervisorctl start usm_gps
Log file rotation
The log files created by the module need to be rotated. To do this, the standard log rotation tools in the operating system should be used.
The following configuration will rotate all *.log files every day and store 7 archive copies (for 7 days)
Rotation in FreeBSD
On FreeBSD, the newsyslog daemon is responsible for rotation. Create the file /usr/local/etc/newsyslog.conf.d/userside, in which place the following line (don't forget the path if you changed it):
/var/log/userside/*.log 644 7 * @T00 GJC /path_to_module/usm_gps_2/usm_gps.pid
Rotation in Linux
In Linux, the logrotate daemon is responsible for rotation. Create a file /etc/logrotate.d/userside, in which put the following text (don't forget the path if you change it):
/var/log/userside/*.log {
rotate 7
daily
compress
delaycompress
missingok
notifempty
copytruncate
}
Upgrading the module
- Download the archive with the new version from your personal cabinet to the server where the module is running
- Stop the module
sudo supervisorctl stop usm_gps
- Delete all module files, except settings.ini
- Unzip the files from the archive with the new version in place of the old files.
- Be sure to install (update) the dependencies:
sudo ./venv/bin/pip install --upgrade -r requirements.txt
- Start the module
sudo supervisorctl start usm_gps
Requests to implement a new protocol
Trackers of the same model may have different telecommunication protocols for communication with the server. This is especially true for cheap Chinese trackers like GT06. The tracker hardware may be manufactured by the same factory, and the customer vendor may install its own firmware in the tracker.
Therefore, we don't use the concept of tracker model support, but use the concept of protocol support.
If you have purchased a tracker, the first thing to find out is what protocol it supports. Usually this information can be found in the device description. If there is no such information, you can try to find the protocol by tracker model in the list above. Enable all protocols available in the module and restart it. Then change the port in the tracker settings so that different protocols of the module are used. If a suitable protocol is found, you are in luck. Use it. If not, you can use the special protocol "test" (default port 9999), which simply logs the packet received from the tracker in hexadecimal representation. Next you need to create a ticket through the ticket system with the subject usm_gps protocol support.... In the body of the ticket, describe your tracker and attach the module lines starting with RX-SAMPLE:.
If you have information about the protocol you want to add to the module, as well as any other information (documentation or where to get it) - be sure to attach all available information to the ticket. This will greatly increase the chances of faster development for you.