Tomcat Server Slow Due To Blocked SeedGenerator Thread on Insufficient Random Number On Linux

Problem:
An application deployed on Tomcat server (via Coldfusion11) suffers slowness problem. Sometimes opening up a https page could take 20 seconds or so. The problem was not experienced on CF9 (which doesn’t use Tomcat). And the same application runs normally on the Windows version of Coldfusion11.

Troubleshooting:
This section provided step by step troubleshooting notes.

Step#1: Check Linux OS usage using top command.

The screenshot was captured when experiencing the slowness. From here you can see the major bottleneck is the cpu where the user process used 99.7% and the java process 9847 (the tomcat) alone used 99.3%.

Step#2: Figure out why this Tomcat is slow.

I enabled JmxRemote by providing these JVM arguments
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9017 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
and restarted it. Then I opened JConsole to watch the java usage. It turned out that a blocked thread called “SeedGenerator” consumed a lot of CPU.

Step#3: Figure out what’s going on with the SeedGenerator.

From JConsole I could see the stack of this thread as below
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
sun.security.provider.SeedGenerator$ThreadedSeedGenerator.run(Unknown Source)
java.lang.Thread.run(Unknown Source)

And JConsole also told me that this thread was blocked many times.

You can easily find out that SeedGenerator is used by SecureRandom to generate seed with random numbers generated by the OS. If you look into the SeedGenerator class, you will find that the random numbers are read from a source defined by

 String egdSource = SunEntries.getSeedSource();

If you further check the SunEntries class, you can find that the seed source can be specified by jvm argument “java.security.egd”.

Step#4: What is the seed source and which one should you use?

When Tomcat performs some security related operations, such as generating session id and performing other SSL (https)tasks, it needs those random numbers as seeds. Most of the Linux distributions provide a blocking random number pool from /dev/random and non-blocking pool from /dev/urandom. Although the former is more secure, it can also inevitably slow down the application if it can’t generate those random number fast enough. The random numbers are normally generated by hardware based on environment noises. Mouse and keyboard activities can generate such noise and speed up the random number generation. By default, Tomcat uses /dev/random for higher security.

Step#5: Confirm the system can’t generate random numbers fast enough

The /proc/sys/kernel/random folder contains a few files you can view the current snapshot of the random numbers related to /dev/random. A few interesting ones are:

• entrop_avail: tells you how many random bits are available at this time
• poolsize: tells you the max number of random bits in the pool. It is normally 4096 for the newer linux distributions

There are also read and write thresholds and you can read their details from “man 4 random”.

The most important file related to this troubleshooting is entrop_avail. If it contains less number of bits than required by your security algorithm, the application will wait until enough bits are generated. For example, if there are only 120 bits available and your algorithm needs 160 bits, the SeedGenerator thread will be blocked. You can continually watch the available number by

sudo watch -n 0.1 cat /proc/sys/kernel/random/entropy_avail

If you want to to further to confirm the random number call is blocked by the OS, you can use strace to monitor the system calls. More than often, you will find that attaching an existing process to strace using -p may not work. In such case, you can restart your Java process by appending it to the strace command. For example

strace -r -e open,read java tomcatPath

You should see output like

0.000000 open("/dev/random", O_RDONLY) = 8
0.000221 read(8, "@", 1) = 1
0.003221 read(3, "PK\3\4\nRyzB\36\320\267\325u\4u\4 ", 30) = 30

Solutions:
There are a few solutions based on the above analysis.

• Solution#1: use /dev/urandom if the security is not a concern for your application. In such a case, you can simply add “-Djava.security.egd=file:/dev/./urandom” to your jvm arguments and restart it.
• Solution#2: use random number software to beef up random number generation, for example rngd
• Solution#3: Add hardware random number generator.This is the most secure solution.