Resources consumption really changes between different clusters and XpoLog. Nowadays, it is possible to define very large machines or multiple small machines for applications deployment – which is preferred?
From our experience, the “rule” is medium-large machines that will balance between performance results and not too many cluster nodes to be managed.
There is no fixed formula as XpoLog is used for a wide variety of tasks. The details below will some important key notes based on our experience.
Generally, for low daily volume of data (<5 GB/Day) a laptop or standard machine is sufficient, but when it comes production deployments, there are a several recommendations to consider.
The following operating systems are supported for standalone installation:
Linux Kernel versions 2.6.x and later (x86 (only 64-bit is supported))
Windows Server 2008 R2, Windows Server 2012, Server 2012 R2, and Server 2016, Windows 8, 8.1 and 10 (x86 (only 64-bit is supported))
Mac OS X 10.11 and Mac OS 10.12 and 10.13
The following are supported Application Servers for deployment of XpoLog as an application (requires JAVA 1.8):
Note: XpoLog is known to run on additional application servers. However, it was tested only on the above servers.
Memory is an important resource - aggregations are memory consumers and many data structures are stored in it. Depends on your daily volume, consider a 32GB or 16 GB machines. Less than 8 GB is less effective and impacts performance (it is not recommended to use machines with more than 64 GB). Java 1.8 is mandatory , and while allocating memory to XpoLog (XpoLog lax file) ensure that the Min (Xms) and Max (Xmx) allocated memory are identical – this will prevent heap size from resizing in runtime.
Always leave memory to the OS that XpoLog runs on:
In any case, do not allocated more than 32GB to XpoLog as crossing it mainly reduces memory management efficiency, CPUs usage and GC handling of large heap size.
Finally, pay attention to Swapping – swapping memory to disk significantly reduces performance. Try to lower swappiness as much as possible - this value controls the OS swapping policy.
The following browsers are supported:
Google Chrome (Version 50+) - recommended
Mozilla Firefox (Version 50+)
MS Internet Explorer (Version 11+)
The following are the default ports which should be opened:
30303 - HTTP client access to XpoLog web interface and communication between different XpoLog instances (can be modified if needed)
22 - In order to enable XpoLog to establish connections to remote machines over SSH (can be modified if needed)
You should choose a modern processor with multiple cores. Standard machines running XpoLog utilize 4-8 cores.
If you need to choose between faster CPUs or more cores, choose more cores.
Disks are important for XpoLog - XpoLog processing engines mostly perform write operations but not only, while XpoLog UI engines mostly perform read operations but not only.
XpoLog does many bulk reads/writes and many disk seeks and as disks are usually the slowest subsystem in a server, they easily become the bottleneck of the XpoLog cluster.
If you can afford SSDs, they are by far superior to any spinning media. If SSD is not an option, try to obtain the fastest disks possible (high-performance server disks, 15K RPM).
Network Attached Storage (NAS) is optional but note that NAS is often slower and displays larger latency. RAID 0, 10, 01, 0+1 will give the best performance, while RAID 5 will offer the worst performance.
It is mandatory to perform tests on your set of data and configuration deployment to determine optimized performance since it may vary between different types of data and searches XpoLog performs.
Note: In Linux deployments, it is mandatory to allocate a high number of open files and processes to the user that runs XpoLog - for more information please see Post Installation Recommendations or contact firstname.lastname@example.org for more details.
A fast and reliable network is obviously important to performance in a distributed system. Modern data-center networking (1 GbE, 10 GbE) is sufficient for most clusters.
From our experience it is recommended to avoid clusters that are distributed over multiple data centers / geographical locations
The following is hardware recommendation for up to 3 concurrent users, and < 10 GB of daily logs volume:
The following is hardware recommendation for up to 5 concurrent users, and < 10 GB of daily logs volume:
The following is hardware recommendation for up to 25 concurrent users, and higher daily logs volume:
(contact email@example.com to determine if a cluster is needed. More information about clustering can be found here. See installations details at XpoLog Cluster Installation)