The server environment is dangerous to run applications with the root account, making it easy for the shell to become a chicken. So teams with a little bit of awareness build a low-level user to run JAVA programs.

Low Clearance, a bit unlike his own son, especially in difficult times of resource constraints can be seen.

Phenomenon

The problem is that it happened on a machine in a public test environment, and the formal environment is not replicated. This server deployed dozens of services, and the deployment account recently switched from root to Xjbot.

After running it for a while, the server frequently had problems. First, a large number of connections are in the CLOSE state, which was once thought to be a passive shutdown problem. But it's not.

NETSTAT-ANTP | GREP CLOSE | Awk'{ print $7}'| sort | UNIQ-C

Oddly, logging in using the root account or other accounts works just fine. However, when switching to an xjbot account, the following error is reported:

Sudo su-xjjbot

Bash: Fork : no child processes

Bash: Fork : no child processes

Bash: Fork : no child processes

Bash: Fork : no child processes

Bash: Fork Resource temporarily unavailable

This is a system-level error message. In this case, the JVM will also have the corresponding error, but I am afraid you have no chance to see (you can use other system users to see oh) .

• CAN NOT CREATE GC thread. Out of system resources

• Java. (phone rings) Lang. OUTOFMEMORYERROR: unable to create new native thread

Why

The cause is the lack of resources, specifically the process of resources.

A Linux thread is actually a process, so Java is also, specifically, called "light weight process (LWP) " -- Light-weight process.

LWP shares all (or most) of the logical address space and system resources with other processes. A process can create multiple lwps so that they share most of the resources; LWP has its own process identifier and has parent-child relationships with other processes; . . Lwp Is kernel-managed and scheduled as a normal process

Use The following command to see how many process resources are being used by a user

Ps-eLf | GREP XJJBOT (UID) | wc-l

Use The following command to see how many threads each process has started

PS-O NLWP, Pid, lwp, args-u xjjbot (UID) | sort-n

Fix It

According to the Linux everything is a file rule, the first thing that comes to mind is modifying the ULIMIT parameters, but it's not, because it's already big enough. If you think about Elasticsearch, you need to configure something called NPROC when you install it, and that's probably where the problem is that you don't have enough process resources.

Related Profiles:

/ etc / Security / limits. CONF

There are also minor differences between kernel versions. Like what

/ etc / Security / limits. D / *

Will, at some point, overwrite limits. CONF configuration. So if the configuration doesn't work, be sure to check.

For these reasons, you can use limits. All the configurations in D are commented out and unified in limits. CONF configuration.

Here's the original configuration

• Soft NPROC 4096

Root soft NPROC unlimited

Change 4096 to a larger number, or simply change it to unlimited.

Configuration of ElasticSearch system parameters

Now that we mention es, let's take a look at what system configurations need to be changed for an es installation. These experiences are shared and can be extrapolated.

https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html

Disable swap

Swap is a performance killer, so the ES can't take it anymore. Just turn it off.

Sudo SWAPOFF-A

This parameter can also be added to the configuration file, and the JVM locks the memory so that it can not be swapped with the swap partition.

Bootstrap. MEMORY: True

Virtual memory

Es Uses mmapfs to map some data, but the default system parameters are too small for it and need to be modified.

SYSCTL-W VM. Max

To take effect permanently, you need to modify / etc / SYSCTL. CONF

File handle

ULIMIT

There is a limit to the number of File descriptor open on Linux. If your application is dealing with many small files at the same time, you need to configure this parameter.

Sudo Su

ULIMIT-N 65536

Su elasticsearch

/ etc / Security / limits. CONF

Ok, here's the file we just changed. To make the above configuration permanent, you need to change this file.

65536

Number of threads

As we mentioned above, I can think of it quickly because I installed es-. -

Therefore, do not open a large number of random threads, in addition to increase scheduling time, but also easy to top the ceiling of the system.

Under von Neumann's framework, don't they all work the same way?

With the same fate, struggling but unable to escape.