Real-World Timeout Bug Identification and Fixing in Production Server Systems

Background

Cloud server systems (e.g., Hadoop, Cassandra, HDFS, Spark) have become increasingly complex, which often consist of many inter-dependent components. It is challenging to achieve reliability in cloud server applications because 1) different components need to communicate frequently with each other via unreliable networks and 2) individual component may fail at any time. Timeout is one of the commonly used mechanisms to handle unexpected failures in distributed com- puting environments. Timeout mechanism can be used in both intra-node and inter-node communication failover. For example, when a component C1 sends a request to another component C2, C1 sets a timer timeout value and waits for the response from C2 until the timer expires. In case C2 fails or a message loss occurs, C1 can break out of the waiting state triggered by the timeout event and take proper actions (e.g., retrying or skipping) accordingly.

However, many real-world cloud server applications lack proper configuration and handling of those timeout events. As the scale of server applications grow, the likelihood of timeout bugs also increases. In 2015, Amazon DynamoDB service was down for five hours. The service outage is caused by a timeout bug in the metadata server. When the metadata server was already overloaded, the new requests from storage servers to the metadata server failed due to timeout. Storage servers kept retrying, causing further failures and retries, creating a cascading failure.

Publications

• Jingzhu He, Ting Dai, and Xiaohui Gu,
  "TFix: Automatic Timeout Bug Fixing in Production Server Systems",
  Proc. of IEEE International Conference on Distributed Computing Systems (ICDCS), Dallas, Texas, July, 2019.
  


• Jingzhu He, Ting Dai, and Xiaohui Gu,
  "TScope: Automatic Timeout Bug Identification for Server Systems",
  Proc. of IEEE International Conference on Autonomic Computing (ICAC), Trento, Italy, September, 2018.
  


• Ting Dai, Jingzhu He, Xiaohui Gu, Shan Lu,
  "Understanding Real-World Timeout Problems in Cloud Server Systems",
  Proc. of IEEE International Conference on Cloud Engineering (IC2E), Orlando, FL, April, 2018. (best paper nominee)
  
  

  Benchmark

  • The following table shows 19 performance bugs, including 12 timeout bugs and 7 non-timeout bugs, identified by TScope.
    
  
  

  #	Bug name	Timeout or Non-timeout	System Version	Error Message	Identified
  					TScope	PerfScope	Clustering	SOM-all	SOM-parameter
  1	Flume-1316	Timeout	v1.1.0	Missing	✔	✔	✗	✔	✔
  2	Flume-1819	Timeout	v1.3.0	Missing	✔	✔	✗	✔	✗
  3	MapReduce-5066	Timeout	v2.0.3-alpha	Missing	✔	✔	✗	✔	✗
  4	Hadoop-11252	Timeout	v2.6.4	Missing	✔	✔	✗	✗	✗
  5	HDFS-10223	Timeout	v2.6.4	Missing	✔	✔	✗	✔	✔
  6	Tomcat-56684	Timeout	v6.0.39	Missing	✔	✔	✔	✔	✔
  7	Zookeeper-1366	Timeout	v3.5.0	Missing	✔	✔	✔	✔	✔
  8	Phoenix-2496	Timeout	v4.6.0	Missing	✗	✗	✗	✗	✔
  9	Hadoop-11252	Timeout	v2.5.0	Missing	✔	✔	✔	✔	✗
  10	Cassandra-7886	Timeout	v2.2.0-beta	Misleading	✔	✔	✔	✔	✗
  11	Flume-1842	Timeout	v1.3.0	Missing	✔	✔	✔	✔	✔
  12	MapReduce-5724	Timeout	v3.0.0-alpha1	Missing	✔	✔	✔	✔	✔
  13	Cassandra-5064	Non-timeout	v1.2.0-rc2	Missing	✔	✗	✗	✔	✗
  14	Apache-37680	Non-timeout	v2.2.2	Missing	✔	✗	✗	✔	✗
  15	Tomcat-48827	Non-timeout	v6.0.24	Correct	✔	✗	✗	✗	✗
  16	Tomcat-53450	Non-timeout	v7.0.28	Correct	✔	✗	✔	✔	✔
  17	MapReduce-3738	Non-timeout	v2.0.0-alpha	Missing	✔	✗	✗	✗	✗
  18	MySQL-65615	Non-timeout	v5.6.5-m8	Missing	✔	✗	✗	✗	✔
  19	MySQL-54332	Non-timeout	v5.5.5-m3	Missing	✔	✗	✗	✗	✔

  
  
  
• The following table shows 13 timeout bugs, including 8 misused timeout bugs and 5 missing timeout bugs, classified by TFix. The table also shows the fixing results for the 8 misused timeout bugs.
  

#	Bug name	System Version	Missing or Misused Timeout	Classification	Timeout Affected Functions	Misused Timeout Variable	Fixed
1	Hadoop-9106	v2.0.3-alpha	Misused	✔	Client.setupConnection()	ipc.client.connect.timeout	✔
2	Hadoop-11252	v2.6.4	Misused	✔	RPC.getProtocolProxy()	ipc.client.rpc-timeout.ms	✔
3	HDFS-4301	v2.0.3-alpha	Misused	✔	TransferImage.doGetUrl()	dfs.image.transfer.timeout	✔
4	HDFS-10223	v2.8.0	Misused	✔	DFSUtilClient.peerFromSocketAndKey()	dfs.client.socket-timeout	✔
5	MapReduce-6263	v2.7.0	Misused	✔	YARNRunner.killJob()	yarn.app.mapreduce.am.hard-kill-timeout-ms	✔
6	MapReduce-4089	v2.7.0	Misused	✔	TaskHeartbeatHandler.PingChecker.run()	mapreduce.task.timeout	✔
7	HBase-15645	v1.3.0	Misused	✔	RpcRetryingCaller.callWithRetries()	hbase.client.operation.timeout	✔
8	HBase-17341	v1.3.0	Misused	✔	ReplicationSource.terminate()	replication.source.maxretriesmultiplier	✔
9	Hadoop-11252	v2.5.0	Missing	✔	--	--	--
10	HDFS-1490	v2.0.2-alpha	Missing	✔	--	--	--
11	MapReduce-5066	v2.0.3-alpha	Missing	✔	--	--	--
12	Flume-1316	v1.1.0	Missing	✔	--	--	--
13	Flume-1819	v1.3.0	Missing	✔	--	--	--

Sponsors

• NSF CNS1513942 grant and NSF CNS1149445 grant