The following document describes 22 TPCH querys and it's Informix and Hive formats. You can get additional information for other databases in the following links:
And a more complet TPCH evaluation here:
To access a grid from Axional DBStudio use var grid = new Ax.Grid("NameOfGrid");
1 Q1 - Pricing Summary Report Query
This query reports the amount of business that was billed, shipped, and returned.
1.1 Business Question
The Pricing Summary Report Query provides a summary pricing report for all line items shipped as of a given date. The date is within 60-120 days of the greatest ship date contained in the database.
1.2 Functional Query Definition
The query lists totals for extended price, discounted extended price, discounted extended price plus tax, average quantity, average extended price, and average discount. These aggregates are grouped by RETURNFLAG and LINESTATUS, and listed in ascending order of RETURNFLAG and LINESTATUS. A count of the number of line items in each group is included.
Copy
select
l_returnflag,
l_linestatus,
sum(l_quantity) as sum_qty,
sum(l_extendedprice) as sum_base_price,
sum(l_extendedprice * (1-l_discount)) as sum_disc_price,
sum(l_extendedprice * (1-l_discount) * (1+l_tax)) as sum_charge,
avg(l_quantity) as avg_qty,
avg(l_extendedprice) as avg_price,
avg(l_discount) as avg_disc,
count(*) as count_order
from
lineitem
where
l_shipdate <= mdy (12, 01, 1998 ) - 90 units day
group by
l_returnflag,
l_linestatus
order by
l_returnflag,
l_linestatus;
Copy
select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= '1998-09-16' group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus;
Copy
Estimated Cost: 757763328
Estimated # of Rows Returned: 100513344
Temporary Files Required For: Order By Group By
1) informix.lineitem: SEQUENTIAL SCAN
Filters: informix.lineitem.l_shipdate <= 02-09-1998
Copy
hive> select
> l_returnflag,
> l_linestatus,
> sum(l_quantity) as sum_qty,
> sum(l_extendedprice) as sum_base_price,
> sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,
> sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,
> avg(l_quantity) as avg_qty,
> avg(l_extendedprice) as avg_price,
> avg(l_discount) as avg_disc,
> count(*) as count_order
> from
> lineitem
> where
> l_shipdate <= '1998-09-16'
> group by
> l_returnflag,
> l_linestatus
> order by
> l_returnflag,
> l_linestatus;
Query ID = hadoop_20181126103204_6fde47f7-2645-483a-a2c6-41bbe849881b
Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 311
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0002, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0002/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0002
Hadoop job information for Stage-1: number of mappers: 297; number of reducers: 311
2018-11-26 10:32:13,336 Stage-1 map = 0%, reduce = 0%
2018-11-26 10:32:26,613 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 16.45 sec
2018-11-26 10:32:45,944 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 40.62 sec
2018-11-26 10:33:04,223 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 65.49 sec
2018-11-26 10:33:23,500 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 90.37 sec
2018-11-26 10:33:41,777 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 115.26 sec
2018-11-26 10:34:00,012 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 140.56 sec
2018-11-26 10:34:21,289 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 165.81 sec
2018-11-26 10:34:39,517 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 189.91 sec
2018-11-26 10:34:57,754 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 213.73 sec
2018-11-26 10:35:16,011 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 238.86 sec
2018-11-26 10:35:36,256 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 264.16 sec
2018-11-26 10:35:55,487 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 289.05 sec
2018-11-26 10:36:14,718 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 314.89 sec
2018-11-26 10:36:32,998 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 339.57 sec
2018-11-26 10:36:52,272 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 364.89 sec
2018-11-26 10:37:11,491 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 390.02 sec
2018-11-26 10:37:32,738 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 415.4 sec
2018-11-26 10:37:45,913 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 431.87 sec
2018-11-26 10:38:05,215 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 457.14 sec
2018-11-26 10:38:24,460 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 482.21 sec
2018-11-26 10:38:43,691 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 507.05 sec
2018-11-26 10:39:01,913 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 532.08 sec
2018-11-26 10:39:21,167 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 556.6 sec
2018-11-26 10:39:39,409 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 580.84 sec
2018-11-26 10:39:58,629 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 605.39 sec
2018-11-26 10:40:16,829 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 629.82 sec
2018-11-26 10:40:36,041 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 653.57 sec
2018-11-26 10:40:53,260 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 677.58 sec
2018-11-26 10:41:11,509 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 702.6 sec
2018-11-26 10:41:30,744 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 727.86 sec
2018-11-26 10:41:48,939 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 752.24 sec
2018-11-26 10:42:09,182 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 777.44 sec
2018-11-26 10:42:28,391 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 801.75 sec
2018-11-26 10:42:46,640 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 825.65 sec
2018-11-26 10:43:04,880 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 849.98 sec
2018-11-26 10:43:23,084 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 874.74 sec
2018-11-26 10:43:42,312 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 899.37 sec
2018-11-26 10:43:59,538 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 923.19 sec
2018-11-26 10:44:17,778 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 947.43 sec
2018-11-26 10:44:35,982 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 971.35 sec
2018-11-26 10:44:55,206 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 996.37 sec
2018-11-26 10:45:14,430 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 1020.99 sec
2018-11-26 10:45:32,630 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 1045.15 sec
2018-11-26 10:45:50,834 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 1070.69 sec
2018-11-26 10:46:10,070 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 1095.51 sec
2018-11-26 10:46:28,293 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 1120.53 sec
2018-11-26 10:46:46,525 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 1145.12 sec
2018-11-26 10:47:05,769 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1170.36 sec
2018-11-26 10:47:24,035 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 1195.02 sec
2018-11-26 10:47:42,268 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1220.15 sec
2018-11-26 10:47:54,407 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 1236.56 sec
2018-11-26 10:48:12,613 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 1260.12 sec
2018-11-26 10:48:30,826 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 1285.69 sec
2018-11-26 10:48:48,046 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 1309.81 sec
2018-11-26 10:49:07,294 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 1334.88 sec
2018-11-26 10:49:26,529 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 1360.07 sec
2018-11-26 10:49:46,781 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 1384.88 sec
2018-11-26 10:50:06,008 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 1409.18 sec
2018-11-26 10:50:23,202 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 1434.45 sec
2018-11-26 10:50:43,477 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 1460.31 sec
2018-11-26 10:51:02,693 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1484.68 sec
2018-11-26 10:51:20,880 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 1508.89 sec
2018-11-26 10:51:40,092 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1533.93 sec
2018-11-26 10:51:58,331 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 1559.29 sec
2018-11-26 10:52:17,577 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 1583.85 sec
2018-11-26 10:52:35,793 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 1608.38 sec
2018-11-26 10:52:53,990 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 1632.14 sec
2018-11-26 10:53:13,191 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 1656.97 sec
2018-11-26 10:53:31,399 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 1681.73 sec
2018-11-26 10:53:49,621 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1707.77 sec
2018-11-26 10:54:07,850 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 1732.16 sec
2018-11-26 10:54:26,058 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 1757.01 sec
2018-11-26 10:54:46,307 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1781.69 sec
2018-11-26 10:55:05,568 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 1806.3 sec
2018-11-26 10:55:24,825 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1831.06 sec
2018-11-26 10:55:43,057 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1856.09 sec
2018-11-26 10:56:02,289 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 1880.6 sec
2018-11-26 10:56:21,496 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1904.38 sec
2018-11-26 10:56:39,691 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 1930.16 sec
2018-11-26 10:56:58,921 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1955.17 sec
2018-11-26 10:57:17,176 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 1978.93 sec
2018-11-26 10:57:36,392 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 2004.07 sec
2018-11-26 10:57:55,622 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 2029.14 sec
2018-11-26 10:58:07,779 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 2046.11 sec
2018-11-26 10:58:26,023 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 2071.52 sec
2018-11-26 10:58:45,289 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 2096.9 sec
2018-11-26 10:59:05,526 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 2121.41 sec
2018-11-26 10:59:24,763 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 2146.67 sec
2018-11-26 10:59:43,031 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 2170.67 sec
2018-11-26 11:00:01,274 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 2194.34 sec
2018-11-26 11:00:19,501 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 2219.36 sec
2018-11-26 11:00:39,718 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 2243.49 sec
2018-11-26 11:00:58,937 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 2267.83 sec
2018-11-26 11:01:17,141 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 2293.1 sec
2018-11-26 11:01:36,402 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 2318.36 sec
2018-11-26 11:01:55,624 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 2342.89 sec
2018-11-26 11:02:13,828 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 2367.07 sec
2018-11-26 11:02:32,032 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 2392.37 sec
2018-11-26 11:02:50,241 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 2416.81 sec
2018-11-26 11:03:14,555 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2448.15 sec
2018-11-26 11:03:20,653 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 2453.81 sec
2018-11-26 11:03:29,752 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 2461.36 sec
2018-11-26 11:03:37,848 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 2469.13 sec
2018-11-26 11:03:46,993 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 2477.36 sec
2018-11-26 11:03:56,144 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 2485.15 sec
2018-11-26 11:04:08,325 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 2495.6 sec
2018-11-26 11:04:17,484 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 2503.36 sec
2018-11-26 11:04:26,618 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 2511.19 sec
2018-11-26 11:04:35,740 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 2519.3 sec
2018-11-26 11:04:44,854 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 2527.02 sec
2018-11-26 11:04:52,940 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 2534.66 sec
2018-11-26 11:05:03,060 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 2542.87 sec
2018-11-26 11:05:11,224 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 2550.64 sec
2018-11-26 11:05:20,372 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 2558.49 sec
2018-11-26 11:05:32,558 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 2568.73 sec
2018-11-26 11:05:41,684 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 2576.97 sec
2018-11-26 11:05:50,806 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 2584.75 sec
2018-11-26 11:05:59,934 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 2592.41 sec
2018-11-26 11:06:09,039 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 2600.15 sec
2018-11-26 11:06:18,139 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 2608.28 sec
2018-11-26 11:06:26,261 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 2616.23 sec
2018-11-26 11:06:35,418 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 2624.26 sec
2018-11-26 11:06:45,596 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 2632.31 sec
2018-11-26 11:06:57,769 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 2642.72 sec
2018-11-26 11:07:06,891 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 2650.64 sec
2018-11-26 11:07:16,013 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 2658.5 sec
2018-11-26 11:07:25,127 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 2666.44 sec
2018-11-26 11:07:34,227 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 2674.28 sec
2018-11-26 11:07:42,329 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 2682.34 sec
2018-11-26 11:07:51,506 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 2690.14 sec
2018-11-26 11:08:00,634 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 2698.34 sec
2018-11-26 11:08:09,774 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2706.17 sec
2018-11-26 11:08:21,976 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 2716.49 sec
2018-11-26 11:08:31,119 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 2724.65 sec
2018-11-26 11:08:40,271 Stage-1 map = 100%, reduce = 35%, Cumulative CPU 2732.66 sec
2018-11-26 11:08:49,363 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 2740.51 sec
2018-11-26 11:08:57,463 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 2748.52 sec
2018-11-26 11:09:06,617 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 2756.43 sec
2018-11-26 11:09:15,793 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 2764.32 sec
2018-11-26 11:09:24,925 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 2772.49 sec
2018-11-26 11:09:34,051 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 2780.49 sec
2018-11-26 11:09:46,205 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 2790.98 sec
2018-11-26 11:09:55,341 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 2799.07 sec
2018-11-26 11:10:04,460 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 2806.76 sec
2018-11-26 11:10:13,572 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 2814.6 sec
2018-11-26 11:10:22,671 Stage-1 map = 100%, reduce = 46%, Cumulative CPU 2822.26 sec
2018-11-26 11:10:30,818 Stage-1 map = 100%, reduce = 47%, Cumulative CPU 2830.11 sec
2018-11-26 11:10:39,924 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 2837.66 sec
2018-11-26 11:10:49,070 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 2845.44 sec
2018-11-26 11:10:59,208 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 2853.31 sec
2018-11-26 11:11:11,390 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 2863.82 sec
2018-11-26 11:11:20,509 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 2871.89 sec
2018-11-26 11:11:29,607 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 2879.73 sec
2018-11-26 11:11:37,705 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 2887.64 sec
2018-11-26 11:11:46,829 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 2895.7 sec
2018-11-26 11:11:56,016 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 2903.8 sec
2018-11-26 11:12:05,165 Stage-1 map = 100%, reduce = 57%, Cumulative CPU 2911.86 sec
2018-11-26 11:12:14,272 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 2919.78 sec
2018-11-26 11:12:23,447 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 2927.64 sec
2018-11-26 11:12:35,619 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 2938.07 sec
2018-11-26 11:12:44,723 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 2945.66 sec
2018-11-26 11:12:52,814 Stage-1 map = 100%, reduce = 62%, Cumulative CPU 2953.54 sec
2018-11-26 11:13:01,948 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 2961.4 sec
2018-11-26 11:13:11,109 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 2969.39 sec
2018-11-26 11:13:20,256 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 2977.38 sec
2018-11-26 11:13:29,379 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 2985.13 sec
2018-11-26 11:13:38,508 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 2993.07 sec
2018-11-26 11:13:47,625 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 3001.16 sec
2018-11-26 11:13:59,782 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 3011.46 sec
2018-11-26 11:14:08,879 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 3019.23 sec
2018-11-26 11:14:17,995 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 3027.3 sec
2018-11-26 11:14:26,112 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 3035.11 sec
2018-11-26 11:14:35,276 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 3042.87 sec
2018-11-26 11:14:44,406 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 3050.82 sec
2018-11-26 11:14:53,547 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 3058.6 sec
2018-11-26 11:15:02,678 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 3066.68 sec
2018-11-26 11:15:11,804 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 3074.88 sec
2018-11-26 11:15:23,972 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 3085.56 sec
2018-11-26 11:15:33,086 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 3093.47 sec
2018-11-26 11:15:41,210 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 3101.47 sec
2018-11-26 11:15:50,362 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 3109.14 sec
2018-11-26 11:15:59,498 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 3116.86 sec
2018-11-26 11:16:08,647 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 3124.85 sec
2018-11-26 11:16:17,768 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 3132.55 sec
2018-11-26 11:16:26,889 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 3140.59 sec
2018-11-26 11:16:36,013 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 3148.63 sec
2018-11-26 11:16:48,163 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 3158.95 sec
2018-11-26 11:16:56,254 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 3166.69 sec
2018-11-26 11:17:05,439 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 3174.34 sec
2018-11-26 11:17:14,575 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 3182.17 sec
2018-11-26 11:17:23,705 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 3189.75 sec
2018-11-26 11:17:32,839 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 3197.49 sec
2018-11-26 11:17:41,984 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 3205.31 sec
2018-11-26 11:17:51,108 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 3213.17 sec
2018-11-26 11:18:00,234 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 3220.9 sec
2018-11-26 11:18:12,370 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 3231.46 sec
2018-11-26 11:18:20,512 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 3239.18 sec
2018-11-26 11:18:29,665 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 3247.08 sec
2018-11-26 11:18:38,823 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 3255.47 sec
2018-11-26 11:18:51,016 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3266.3 sec
MapReduce Total cumulative CPU time: 54 minutes 26 seconds 300 msec
Ended Job = job_1543224159463_0002
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0003, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0003/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0003
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2018-11-26 11:19:02,177 Stage-2 map = 0%, reduce = 0%
2018-11-26 11:19:08,298 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 6.32 sec
2018-11-26 11:19:13,380 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 7.88 sec
MapReduce Total cumulative CPU time: 7 seconds 880 msec
Ended Job = job_1543224159463_0003
MapReduce Jobs Launched:
Stage-Stage-1: Map: 297 Reduce: 311 Cumulative CPU: 3266.3 sec HDFS Read: 79591008376 HDFS Write: 30233 SUCCESS
Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 7.88 sec HDFS Read: 121439 HDFS Write: 762 SUCCESS
Total MapReduce CPU Time Spent: 54 minutes 34 seconds 180 msec
OK
A F 3775127758.00 5660776097194.45 5377736398183.9374 5592847429515.927026 25.4993704232754266844251557 38236.1169843048952520975291771 0.050002243530929 148047881
N F 98553062.00 147771098385.98 140384965965.0348 145999793032.775829 25.5015569568828776144429293 38237.1993888045044881863276570 0.049985284338054 3864590
N O 7497751207.00 11242876164661.45 10680741592194.7490 11107983328512.815019 25.4999720827598752658323410 NULL 0.049997568681657 294029781
R F 3775724970.00 5661603032745.34 5378513563915.4097 5593662252666.916161 25.5000662840653208274042430 38236.6972584529675334508956710 0.050001304339654 148067261
Time taken: 2829.804 seconds, Fetched: 4 row(s)
Copy
<script type="js">
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
select
l_returnflag,
l_linestatus,
sum(l_quantity) as sum_qty,
sum(l_extendedprice) as sum_base_price,
sum(l_extendedprice * (1-l_discount)) as sum_disc_price,
sum(l_extendedprice * (1-l_discount) * (1+l_tax)) as sum_charge,
avg(l_quantity) as avg_qty,
avg(l_extendedprice) as avg_price,
avg(l_discount) as avg_disc,
count(*) as count_order
from
lineitem
where
l_shipdate <= mdy (12, 01, 1998 ) - 90 units day
group by
l_returnflag,
l_linestatus
`
,
`
select l_returnflag,
l_linestatus,
sum(sum_qty) as sum_qty,
sum(sum_base_price) as sum_base_price,
sum(sum_disc_price) as sum_disc_price,
sum(sum_charge) as sum_charge,
avg(avg_qty) as avg_qty,
avg(avg_price) as avg_price,
avg(avg_disc) as avg_disc,
sum(count_order) as count_order
FROM \${temp}
group by
l_returnflag,
l_linestatus
order by
l_returnflag,
l_linestatus
`
);
</script>
1.3 Substitution Parameters
Values for the following substitution parameter must be generated and used to build the executable query text: 1. DELTA is randomly selected within [60. 120].
1998-12-01 is the highest possible ship date as defined in the database population. (This is ENDDATE -
30). The query will include all lineitems shipped before this date minus DELTA days. The intent is to choose
DELTA so that between 95% and 97% of the rows in the table are scanned.
1.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- 90
Sample Output
| l_returnflag | A |
| l_linestatus | F |
| sum_qty | 37734107.00 |
| sum_base_price | 56586554400.73 |
| sum_disc_price | 53758257134.87 |
| sum_charge | 55909065222.83 |
| avg_qty | 25.52 |
| avg_price | 38273.13 |
| avg_disc | .05 |
| count_order | 1478493 |
2 Q2 - Minimum Cost Supplier Query
This query finds which supplier should be selected to place an order for a given part in a given region
2.1 Business Question
The Minimum Cost Supplier Query finds, in a given region, for each part of a certain type and size, the supplier who can supply it at minimum cost. If several suppliers in that region offer the desired part type and size at the same (minimum) cost, the query lists the parts from suppliers with the 100 highest account balances. For each supplier, the query lists the supplier's account balance, name and nation; the part's number and manufacturer; the supplier's address, phone number and comment information.
2.2 Functional Query Definition
Return the first 100 selected rows
Copy
SELECT FIRST 100
s_acctbal,
s_name,
n_name,
p_partkey,
p_mfgr,
s_address,
s_phone,
s_comment
FROM part, supplier, partsupp, nation, region
WHERE
p_partkey = ps_partkey
AND s_suppkey = ps_suppkey
AND p_size = 15
AND p_type LIKE '%BRASS'
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'EUROPE'
AND ps_supplycost = (
SELECT
MIN(ps_supplycost)
FROM
partsupp,
supplier,
nation,
region
WHERE
p_partkey = ps_partkey
AND s_suppkey = ps_suppkey
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'EUROPE'
)
ORDER BY s_acctbal DESC, n_name, s_name, p_partkey
Copy
drop view q2_min_ps_supplycost;
create view q2_min_ps_supplycost as
select
p_partkey as min_p_partkey,
min(ps_supplycost) as min_ps_supplycost
from
part,
partsupp,
supplier,
nation,
region
where
p_partkey = ps_partkey
and s_suppkey = ps_suppkey
and s_nationkey = n_nationkey
and n_regionkey = r_regionkey
and r_name = 'EUROPE'
group by
p_partkey;
select
s_acctbal,
s_name,
n_name,
p_partkey,
p_mfgr,
s_address,
s_phone,
s_comment
from
part,
supplier,
partsupp,
nation,
region,
q2_min_ps_supplycost
where
p_partkey = ps_partkey
and s_suppkey = ps_suppkey
and p_size = 37
and p_type like '%COPPER'
and s_nationkey = n_nationkey
and n_regionkey = r_regionkey
and r_name = 'EUROPE'
and ps_supplycost = min_ps_supplycost
and p_partkey = min_p_partkey
order by
s_acctbal desc,
n_name,
s_name,
p_partkey
limit 100;
Hive requires a syntax fix to run the query. It's done by creating a view to resolve the subquery.
Copy
Estimated Cost: 21985790
Estimated # of Rows Returned: 160000
Temporary Files Required For: Order By
1) informix.region: SEQUENTIAL SCAN
Filters: informix.region.r_name = 'EUROPE'
2) informix.nation: INDEX PATH
(1) Index Name: informix.nation_fk_region
Index Keys: n_regionkey (Serial, fragments: ALL)
Lower Index Filter: informix.nation.n_regionkey = informix.region.r_regionkey
NESTED LOOP JOIN
3) informix.supplier: INDEX PATH
(1) Index Name: informix.supplier_fk_nation
Index Keys: s_nationkey (Serial, fragments: ALL)
Lower Index Filter: informix.supplier.s_nationkey = informix.nation.n_nationkey
NESTED LOOP JOIN
4) informix.partsupp: INDEX PATH
(1) Index Name: informix.partsupp_fk_supplier
Index Keys: ps_suppkey (Serial, fragments: ALL)
Lower Index Filter: informix.supplier.s_suppkey = informix.partsupp.ps_suppkey
NESTED LOOP JOIN
5) informix.part: SEQUENTIAL SCAN
Filters:
Table Scan Filters: (informix.part.p_size = 15 AND informix.part.p_type LIKE '%BRASS' )
DYNAMIC HASH JOIN
Dynamic Hash Filters: informix.part.p_partkey = informix.partsupp.ps_partkey
Other Join Filters: informix.partsupp.ps_supplycost = <subquery>
Subquery:
---------
Estimated Cost: 21
Estimated # of Rows Returned: 1
1) informix.partsupp: INDEX PATH
(1) Index Name: informix.partsupp_fk_part
Index Keys: ps_partkey (Serial, fragments: ALL)
Lower Index Filter: informix.partsupp.ps_partkey = informix.part.p_partkey
2) informix.supplier: INDEX PATH
(1) Index Name: informix.supplier_pk
Index Keys: s_suppkey (Serial, fragments: ALL)
Lower Index Filter: informix.supplier.s_suppkey = informix.partsupp.ps_suppkey
NESTED LOOP JOIN
3) informix.nation: INDEX PATH
(1) Index Name: informix.nation_pk
Index Keys: n_nationkey (Serial, fragments: ALL)
Lower Index Filter: informix.supplier.s_nationkey = informix.nation.n_nationkey
NESTED LOOP JOIN
4) informix.region: INDEX PATH
Filters: informix.region.r_name = 'EUROPE'
(1) Index Name: informix.region_pk
Index Keys: r_regionkey (Serial, fragments: ALL)
Lower Index Filter: informix.nation.n_regionkey = informix.region.r_regionkey
NESTED LOOP JOIN
Copy
hive>
> select
> s_acctbal,
> s_name,
> n_name,
> p_partkey,
> p_mfgr,
> s_address,
> s_phone,
> s_comment
> from
> part,
> supplier,
> partsupp,
> nation,
> region,
> q2_min_ps_supplycost
> where
> p_partkey = ps_partkey
> and s_suppkey = ps_suppkey
> and p_size = 37
> and p_type like '%COPPER'
> and s_nationkey = n_nationkey
> and n_regionkey = r_regionkey
> and r_name = 'EUROPE'
> and ps_supplycost = min_ps_supplycost
> and p_partkey = min_p_partkey
> order by
> s_acctbal desc,
> n_name,
> s_name,
> p_partkey
> limit 100;
No Stats for tpch_100@part, Columns: p_partkey, p_type, p_mfgr, p_size
No Stats for tpch_100@supplier, Columns: s_comment, s_phone, s_nationkey, s_name, s_address, s_acctbal, s_suppkey
No Stats for tpch_100@partsupp, Columns: ps_suppkey, ps_partkey, ps_supplycost
No Stats for tpch_100@nation, Columns: n_nationkey, n_regionkey, n_name
No Stats for tpch_100@region, Columns: r_regionkey, r_name
No Stats for tpch_100@part, Columns: p_partkey
No Stats for tpch_100@partsupp, Columns: ps_suppkey, ps_partkey, ps_supplycost
No Stats for tpch_100@supplier, Columns: s_nationkey, s_suppkey
No Stats for tpch_100@nation, Columns: n_nationkey, n_regionkey
No Stats for tpch_100@region, Columns: r_regionkey, r_name
Query ID = hadoop_20181126112521_e3ba4f0f-9c0e-4de8-a6bb-d9c4bf5b9354
Total jobs = 18
Launching Job 1 out of 18
Number of reduce tasks not specified. Estimated from input data size: 58
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0004, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0004/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0004
Hadoop job information for Stage-8: number of mappers: 56; number of reducers: 58
2018-11-26 11:25:27,807 Stage-8 map = 0%, reduce = 0%
2018-11-26 11:25:32,892 Stage-8 map = 2%, reduce = 0%, Cumulative CPU 5.4 sec
2018-11-26 11:25:37,978 Stage-8 map = 4%, reduce = 0%, Cumulative CPU 11.03 sec
2018-11-26 11:25:43,083 Stage-8 map = 5%, reduce = 0%, Cumulative CPU 16.89 sec
2018-11-26 11:25:48,161 Stage-8 map = 7%, reduce = 0%, Cumulative CPU 22.18 sec
2018-11-26 11:25:53,251 Stage-8 map = 9%, reduce = 0%, Cumulative CPU 27.79 sec
2018-11-26 11:25:58,308 Stage-8 map = 11%, reduce = 0%, Cumulative CPU 33.0 sec
2018-11-26 11:26:02,355 Stage-8 map = 13%, reduce = 0%, Cumulative CPU 38.34 sec
2018-11-26 11:26:06,404 Stage-8 map = 14%, reduce = 0%, Cumulative CPU 44.01 sec
2018-11-26 11:26:11,465 Stage-8 map = 16%, reduce = 0%, Cumulative CPU 49.17 sec
2018-11-26 11:26:18,546 Stage-8 map = 18%, reduce = 0%, Cumulative CPU 58.1 sec
2018-11-26 11:26:26,637 Stage-8 map = 20%, reduce = 0%, Cumulative CPU 66.38 sec
2018-11-26 11:26:33,716 Stage-8 map = 21%, reduce = 0%, Cumulative CPU 75.27 sec
2018-11-26 11:26:40,799 Stage-8 map = 23%, reduce = 0%, Cumulative CPU 83.83 sec
2018-11-26 11:26:48,890 Stage-8 map = 25%, reduce = 0%, Cumulative CPU 92.31 sec
2018-11-26 11:26:55,984 Stage-8 map = 27%, reduce = 0%, Cumulative CPU 101.08 sec
2018-11-26 11:27:03,061 Stage-8 map = 29%, reduce = 0%, Cumulative CPU 109.91 sec
2018-11-26 11:27:11,159 Stage-8 map = 30%, reduce = 0%, Cumulative CPU 118.17 sec
2018-11-26 11:27:18,252 Stage-8 map = 32%, reduce = 0%, Cumulative CPU 126.31 sec
2018-11-26 11:27:25,342 Stage-8 map = 34%, reduce = 0%, Cumulative CPU 134.75 sec
2018-11-26 11:27:32,416 Stage-8 map = 36%, reduce = 0%, Cumulative CPU 143.0 sec
2018-11-26 11:27:39,492 Stage-8 map = 38%, reduce = 0%, Cumulative CPU 151.76 sec
2018-11-26 11:27:46,568 Stage-8 map = 39%, reduce = 0%, Cumulative CPU 160.66 sec
2018-11-26 11:27:54,648 Stage-8 map = 41%, reduce = 0%, Cumulative CPU 168.96 sec
2018-11-26 11:28:01,756 Stage-8 map = 43%, reduce = 0%, Cumulative CPU 177.43 sec
2018-11-26 11:28:09,837 Stage-8 map = 45%, reduce = 0%, Cumulative CPU 186.33 sec
2018-11-26 11:28:17,926 Stage-8 map = 46%, reduce = 0%, Cumulative CPU 194.84 sec
2018-11-26 11:28:25,001 Stage-8 map = 48%, reduce = 0%, Cumulative CPU 203.52 sec
2018-11-26 11:28:32,094 Stage-8 map = 50%, reduce = 0%, Cumulative CPU 212.01 sec
2018-11-26 11:28:40,180 Stage-8 map = 52%, reduce = 0%, Cumulative CPU 220.4 sec
2018-11-26 11:28:47,259 Stage-8 map = 54%, reduce = 0%, Cumulative CPU 229.18 sec
2018-11-26 11:28:55,343 Stage-8 map = 55%, reduce = 0%, Cumulative CPU 238.12 sec
2018-11-26 11:29:02,415 Stage-8 map = 57%, reduce = 0%, Cumulative CPU 246.77 sec
2018-11-26 11:29:09,485 Stage-8 map = 59%, reduce = 0%, Cumulative CPU 255.22 sec
2018-11-26 11:29:16,558 Stage-8 map = 61%, reduce = 0%, Cumulative CPU 263.77 sec
2018-11-26 11:29:23,634 Stage-8 map = 63%, reduce = 0%, Cumulative CPU 272.61 sec
2018-11-26 11:29:31,722 Stage-8 map = 64%, reduce = 0%, Cumulative CPU 281.23 sec
2018-11-26 11:29:38,795 Stage-8 map = 66%, reduce = 0%, Cumulative CPU 289.83 sec
2018-11-26 11:29:45,876 Stage-8 map = 68%, reduce = 0%, Cumulative CPU 298.72 sec
2018-11-26 11:29:53,950 Stage-8 map = 70%, reduce = 0%, Cumulative CPU 307.48 sec
2018-11-26 11:30:01,023 Stage-8 map = 71%, reduce = 0%, Cumulative CPU 315.51 sec
2018-11-26 11:30:08,105 Stage-8 map = 73%, reduce = 0%, Cumulative CPU 324.16 sec
2018-11-26 11:30:15,179 Stage-8 map = 75%, reduce = 0%, Cumulative CPU 332.89 sec
2018-11-26 11:30:21,259 Stage-8 map = 77%, reduce = 0%, Cumulative CPU 341.13 sec
2018-11-26 11:30:28,347 Stage-8 map = 79%, reduce = 0%, Cumulative CPU 349.74 sec
2018-11-26 11:30:35,419 Stage-8 map = 80%, reduce = 0%, Cumulative CPU 358.16 sec
2018-11-26 11:30:42,505 Stage-8 map = 82%, reduce = 0%, Cumulative CPU 367.19 sec
2018-11-26 11:30:50,589 Stage-8 map = 84%, reduce = 0%, Cumulative CPU 375.85 sec
2018-11-26 11:30:57,657 Stage-8 map = 86%, reduce = 0%, Cumulative CPU 384.59 sec
2018-11-26 11:31:04,723 Stage-8 map = 88%, reduce = 0%, Cumulative CPU 392.95 sec
2018-11-26 11:31:11,799 Stage-8 map = 89%, reduce = 0%, Cumulative CPU 401.99 sec
2018-11-26 11:31:19,882 Stage-8 map = 91%, reduce = 0%, Cumulative CPU 410.78 sec
2018-11-26 11:31:26,955 Stage-8 map = 93%, reduce = 0%, Cumulative CPU 419.49 sec
2018-11-26 11:31:35,055 Stage-8 map = 95%, reduce = 0%, Cumulative CPU 428.29 sec
2018-11-26 11:31:43,155 Stage-8 map = 96%, reduce = 0%, Cumulative CPU 436.97 sec
2018-11-26 11:31:48,218 Stage-8 map = 98%, reduce = 0%, Cumulative CPU 443.12 sec
2018-11-26 11:31:51,271 Stage-8 map = 100%, reduce = 0%, Cumulative CPU 446.38 sec
2018-11-26 11:31:56,327 Stage-8 map = 100%, reduce = 2%, Cumulative CPU 451.59 sec
2018-11-26 11:32:00,367 Stage-8 map = 100%, reduce = 3%, Cumulative CPU 456.61 sec
2018-11-26 11:32:05,443 Stage-8 map = 100%, reduce = 5%, Cumulative CPU 461.69 sec
2018-11-26 11:32:09,502 Stage-8 map = 100%, reduce = 7%, Cumulative CPU 466.56 sec
2018-11-26 11:32:14,558 Stage-8 map = 100%, reduce = 9%, Cumulative CPU 471.45 sec
2018-11-26 11:32:18,613 Stage-8 map = 100%, reduce = 10%, Cumulative CPU 476.44 sec
2018-11-26 11:32:22,681 Stage-8 map = 100%, reduce = 12%, Cumulative CPU 481.32 sec
2018-11-26 11:32:26,732 Stage-8 map = 100%, reduce = 14%, Cumulative CPU 486.35 sec
2018-11-26 11:32:30,775 Stage-8 map = 100%, reduce = 16%, Cumulative CPU 491.03 sec
2018-11-26 11:32:34,834 Stage-8 map = 100%, reduce = 17%, Cumulative CPU 495.99 sec
2018-11-26 11:32:39,888 Stage-8 map = 100%, reduce = 19%, Cumulative CPU 500.98 sec
2018-11-26 11:32:43,933 Stage-8 map = 100%, reduce = 21%, Cumulative CPU 505.82 sec
2018-11-26 11:32:48,991 Stage-8 map = 100%, reduce = 22%, Cumulative CPU 510.81 sec
2018-11-26 11:32:53,041 Stage-8 map = 100%, reduce = 24%, Cumulative CPU 515.66 sec
2018-11-26 11:32:57,092 Stage-8 map = 100%, reduce = 26%, Cumulative CPU 520.52 sec
2018-11-26 11:33:02,152 Stage-8 map = 100%, reduce = 28%, Cumulative CPU 525.66 sec
2018-11-26 11:33:07,204 Stage-8 map = 100%, reduce = 29%, Cumulative CPU 530.41 sec
2018-11-26 11:33:11,250 Stage-8 map = 100%, reduce = 31%, Cumulative CPU 535.54 sec
2018-11-26 11:33:16,303 Stage-8 map = 100%, reduce = 33%, Cumulative CPU 540.69 sec
2018-11-26 11:33:21,373 Stage-8 map = 100%, reduce = 34%, Cumulative CPU 545.88 sec
2018-11-26 11:33:26,427 Stage-8 map = 100%, reduce = 36%, Cumulative CPU 550.91 sec
2018-11-26 11:33:30,467 Stage-8 map = 100%, reduce = 38%, Cumulative CPU 556.01 sec
2018-11-26 11:33:35,542 Stage-8 map = 100%, reduce = 40%, Cumulative CPU 560.88 sec
2018-11-26 11:33:39,603 Stage-8 map = 100%, reduce = 41%, Cumulative CPU 565.92 sec
2018-11-26 11:33:43,665 Stage-8 map = 100%, reduce = 43%, Cumulative CPU 570.93 sec
2018-11-26 11:33:47,711 Stage-8 map = 100%, reduce = 45%, Cumulative CPU 575.92 sec
2018-11-26 11:33:51,763 Stage-8 map = 100%, reduce = 47%, Cumulative CPU 580.83 sec
2018-11-26 11:33:56,824 Stage-8 map = 100%, reduce = 48%, Cumulative CPU 585.84 sec
2018-11-26 11:34:00,875 Stage-8 map = 100%, reduce = 50%, Cumulative CPU 590.82 sec
2018-11-26 11:34:04,930 Stage-8 map = 100%, reduce = 52%, Cumulative CPU 595.76 sec
2018-11-26 11:34:08,971 Stage-8 map = 100%, reduce = 53%, Cumulative CPU 600.52 sec
2018-11-26 11:34:13,012 Stage-8 map = 100%, reduce = 55%, Cumulative CPU 605.38 sec
2018-11-26 11:34:17,074 Stage-8 map = 100%, reduce = 57%, Cumulative CPU 610.46 sec
2018-11-26 11:34:22,131 Stage-8 map = 100%, reduce = 59%, Cumulative CPU 615.32 sec
2018-11-26 11:34:26,174 Stage-8 map = 100%, reduce = 60%, Cumulative CPU 620.28 sec
2018-11-26 11:34:30,217 Stage-8 map = 100%, reduce = 62%, Cumulative CPU 625.05 sec
2018-11-26 11:34:34,270 Stage-8 map = 100%, reduce = 64%, Cumulative CPU 629.94 sec
2018-11-26 11:34:38,310 Stage-8 map = 100%, reduce = 66%, Cumulative CPU 634.87 sec
2018-11-26 11:34:42,354 Stage-8 map = 100%, reduce = 67%, Cumulative CPU 639.77 sec
2018-11-26 11:34:46,396 Stage-8 map = 100%, reduce = 69%, Cumulative CPU 644.7 sec
2018-11-26 11:34:50,439 Stage-8 map = 100%, reduce = 71%, Cumulative CPU 649.83 sec
2018-11-26 11:34:55,488 Stage-8 map = 100%, reduce = 72%, Cumulative CPU 654.84 sec
2018-11-26 11:34:58,523 Stage-8 map = 100%, reduce = 74%, Cumulative CPU 659.65 sec
2018-11-26 11:35:02,564 Stage-8 map = 100%, reduce = 76%, Cumulative CPU 664.59 sec
2018-11-26 11:35:06,606 Stage-8 map = 100%, reduce = 78%, Cumulative CPU 669.55 sec
2018-11-26 11:35:10,667 Stage-8 map = 100%, reduce = 79%, Cumulative CPU 674.27 sec
2018-11-26 11:35:14,706 Stage-8 map = 100%, reduce = 81%, Cumulative CPU 679.13 sec
2018-11-26 11:35:19,771 Stage-8 map = 100%, reduce = 83%, Cumulative CPU 683.78 sec
2018-11-26 11:35:23,838 Stage-8 map = 100%, reduce = 84%, Cumulative CPU 688.84 sec
2018-11-26 11:35:28,938 Stage-8 map = 100%, reduce = 86%, Cumulative CPU 693.56 sec
2018-11-26 11:35:32,997 Stage-8 map = 100%, reduce = 88%, Cumulative CPU 698.5 sec
2018-11-26 11:35:37,039 Stage-8 map = 100%, reduce = 90%, Cumulative CPU 703.58 sec
2018-11-26 11:35:42,086 Stage-8 map = 100%, reduce = 91%, Cumulative CPU 708.52 sec
2018-11-26 11:35:46,128 Stage-8 map = 100%, reduce = 93%, Cumulative CPU 713.65 sec
2018-11-26 11:35:51,182 Stage-8 map = 100%, reduce = 95%, Cumulative CPU 718.58 sec
2018-11-26 11:35:55,228 Stage-8 map = 100%, reduce = 97%, Cumulative CPU 723.28 sec
2018-11-26 11:35:59,273 Stage-8 map = 100%, reduce = 98%, Cumulative CPU 728.13 sec
2018-11-26 11:36:03,341 Stage-8 map = 100%, reduce = 100%, Cumulative CPU 733.11 sec
MapReduce Total cumulative CPU time: 12 minutes 13 seconds 110 msec
Ended Job = job_1543224159463_0004
Launching Job 2 out of 18
Number of reduce tasks not specified. Estimated from input data size: 58
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0005, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0005/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0005
Hadoop job information for Stage-16: number of mappers: 56; number of reducers: 58
2018-11-26 11:36:13,105 Stage-16 map = 0%, reduce = 0%
2018-11-26 11:36:22,233 Stage-16 map = 2%, reduce = 0%, Cumulative CPU 8.53 sec
2018-11-26 11:36:29,323 Stage-16 map = 4%, reduce = 0%, Cumulative CPU 16.7 sec
2018-11-26 11:36:36,399 Stage-16 map = 5%, reduce = 0%, Cumulative CPU 25.64 sec
2018-11-26 11:36:43,480 Stage-16 map = 7%, reduce = 0%, Cumulative CPU 33.66 sec
2018-11-26 11:36:50,557 Stage-16 map = 9%, reduce = 0%, Cumulative CPU 42.48 sec
2018-11-26 11:36:56,635 Stage-16 map = 11%, reduce = 0%, Cumulative CPU 50.96 sec
2018-11-26 11:37:04,750 Stage-16 map = 13%, reduce = 0%, Cumulative CPU 59.75 sec
2018-11-26 11:37:11,822 Stage-16 map = 14%, reduce = 0%, Cumulative CPU 68.01 sec
2018-11-26 11:37:18,927 Stage-16 map = 16%, reduce = 0%, Cumulative CPU 76.6 sec
2018-11-26 11:37:26,026 Stage-16 map = 18%, reduce = 0%, Cumulative CPU 84.73 sec
2018-11-26 11:37:33,103 Stage-16 map = 20%, reduce = 0%, Cumulative CPU 92.73 sec
2018-11-26 11:37:40,191 Stage-16 map = 21%, reduce = 0%, Cumulative CPU 101.53 sec
2018-11-26 11:37:48,280 Stage-16 map = 23%, reduce = 0%, Cumulative CPU 110.25 sec
2018-11-26 11:37:55,350 Stage-16 map = 25%, reduce = 0%, Cumulative CPU 118.62 sec
2018-11-26 11:38:02,424 Stage-16 map = 27%, reduce = 0%, Cumulative CPU 126.73 sec
2018-11-26 11:38:08,491 Stage-16 map = 29%, reduce = 0%, Cumulative CPU 135.27 sec
2018-11-26 11:38:16,579 Stage-16 map = 30%, reduce = 0%, Cumulative CPU 143.63 sec
2018-11-26 11:38:23,664 Stage-16 map = 32%, reduce = 0%, Cumulative CPU 152.29 sec
2018-11-26 11:38:31,752 Stage-16 map = 34%, reduce = 0%, Cumulative CPU 160.93 sec
2018-11-26 11:38:38,843 Stage-16 map = 36%, reduce = 0%, Cumulative CPU 169.46 sec
2018-11-26 11:38:45,919 Stage-16 map = 38%, reduce = 0%, Cumulative CPU 177.97 sec
2018-11-26 11:38:53,003 Stage-16 map = 39%, reduce = 0%, Cumulative CPU 186.44 sec
2018-11-26 11:39:00,074 Stage-16 map = 41%, reduce = 0%, Cumulative CPU 194.79 sec
2018-11-26 11:39:07,144 Stage-16 map = 43%, reduce = 0%, Cumulative CPU 203.2 sec
2018-11-26 11:39:14,217 Stage-16 map = 45%, reduce = 0%, Cumulative CPU 211.47 sec
2018-11-26 11:39:21,295 Stage-16 map = 46%, reduce = 0%, Cumulative CPU 219.94 sec
2018-11-26 11:39:28,359 Stage-16 map = 48%, reduce = 0%, Cumulative CPU 228.09 sec
2018-11-26 11:39:35,432 Stage-16 map = 50%, reduce = 0%, Cumulative CPU 236.14 sec
2018-11-26 11:39:42,504 Stage-16 map = 52%, reduce = 0%, Cumulative CPU 244.65 sec
2018-11-26 11:39:49,575 Stage-16 map = 54%, reduce = 0%, Cumulative CPU 253.05 sec
2018-11-26 11:39:56,650 Stage-16 map = 55%, reduce = 0%, Cumulative CPU 261.17 sec
2018-11-26 11:40:03,715 Stage-16 map = 57%, reduce = 0%, Cumulative CPU 269.74 sec
2018-11-26 11:40:09,773 Stage-16 map = 59%, reduce = 0%, Cumulative CPU 277.85 sec
2018-11-26 11:40:17,858 Stage-16 map = 61%, reduce = 0%, Cumulative CPU 286.6 sec
2018-11-26 11:40:24,927 Stage-16 map = 63%, reduce = 0%, Cumulative CPU 294.7 sec
2018-11-26 11:40:32,007 Stage-16 map = 64%, reduce = 0%, Cumulative CPU 302.85 sec
2018-11-26 11:40:39,082 Stage-16 map = 66%, reduce = 0%, Cumulative CPU 311.74 sec
2018-11-26 11:40:47,190 Stage-16 map = 68%, reduce = 0%, Cumulative CPU 320.37 sec
2018-11-26 11:40:54,275 Stage-16 map = 70%, reduce = 0%, Cumulative CPU 329.32 sec
2018-11-26 11:41:02,363 Stage-16 map = 71%, reduce = 0%, Cumulative CPU 337.34 sec
2018-11-26 11:41:09,438 Stage-16 map = 73%, reduce = 0%, Cumulative CPU 345.73 sec
2018-11-26 11:41:16,518 Stage-16 map = 75%, reduce = 0%, Cumulative CPU 354.66 sec
2018-11-26 11:41:24,601 Stage-16 map = 77%, reduce = 0%, Cumulative CPU 362.9 sec
2018-11-26 11:41:31,674 Stage-16 map = 79%, reduce = 0%, Cumulative CPU 371.03 sec
2018-11-26 11:41:38,739 Stage-16 map = 80%, reduce = 0%, Cumulative CPU 379.15 sec
2018-11-26 11:41:45,809 Stage-16 map = 82%, reduce = 0%, Cumulative CPU 387.84 sec
2018-11-26 11:41:53,899 Stage-16 map = 84%, reduce = 0%, Cumulative CPU 396.86 sec
2018-11-26 11:42:01,992 Stage-16 map = 86%, reduce = 0%, Cumulative CPU 405.71 sec
2018-11-26 11:42:10,075 Stage-16 map = 88%, reduce = 0%, Cumulative CPU 414.46 sec
2018-11-26 11:42:18,169 Stage-16 map = 89%, reduce = 0%, Cumulative CPU 423.26 sec
2018-11-26 11:42:26,250 Stage-16 map = 91%, reduce = 0%, Cumulative CPU 432.02 sec
2018-11-26 11:42:34,332 Stage-16 map = 93%, reduce = 0%, Cumulative CPU 440.94 sec
2018-11-26 11:42:42,411 Stage-16 map = 95%, reduce = 0%, Cumulative CPU 449.99 sec
2018-11-26 11:42:50,505 Stage-16 map = 96%, reduce = 0%, Cumulative CPU 459.15 sec
2018-11-26 11:42:56,583 Stage-16 map = 98%, reduce = 0%, Cumulative CPU 465.32 sec
2018-11-26 11:42:59,613 Stage-16 map = 100%, reduce = 0%, Cumulative CPU 469.69 sec
2018-11-26 11:43:05,692 Stage-16 map = 100%, reduce = 2%, Cumulative CPU 476.12 sec
2018-11-26 11:43:11,770 Stage-16 map = 100%, reduce = 3%, Cumulative CPU 482.93 sec
2018-11-26 11:43:17,852 Stage-16 map = 100%, reduce = 5%, Cumulative CPU 489.46 sec
2018-11-26 11:43:23,935 Stage-16 map = 100%, reduce = 7%, Cumulative CPU 496.12 sec
2018-11-26 11:43:30,011 Stage-16 map = 100%, reduce = 9%, Cumulative CPU 502.8 sec
2018-11-26 11:43:36,090 Stage-16 map = 100%, reduce = 10%, Cumulative CPU 509.51 sec
2018-11-26 11:43:42,156 Stage-16 map = 100%, reduce = 12%, Cumulative CPU 516.32 sec
2018-11-26 11:43:48,221 Stage-16 map = 100%, reduce = 14%, Cumulative CPU 522.84 sec
2018-11-26 11:43:54,287 Stage-16 map = 100%, reduce = 16%, Cumulative CPU 529.46 sec
2018-11-26 11:44:00,353 Stage-16 map = 100%, reduce = 17%, Cumulative CPU 535.98 sec
2018-11-26 11:44:06,419 Stage-16 map = 100%, reduce = 19%, Cumulative CPU 542.58 sec
2018-11-26 11:44:12,495 Stage-16 map = 100%, reduce = 21%, Cumulative CPU 549.41 sec
2018-11-26 11:44:18,556 Stage-16 map = 100%, reduce = 22%, Cumulative CPU 555.95 sec
2018-11-26 11:44:23,612 Stage-16 map = 100%, reduce = 24%, Cumulative CPU 562.66 sec
2018-11-26 11:44:29,678 Stage-16 map = 100%, reduce = 26%, Cumulative CPU 569.16 sec
2018-11-26 11:44:35,741 Stage-16 map = 100%, reduce = 28%, Cumulative CPU 575.6 sec
2018-11-26 11:44:41,818 Stage-16 map = 100%, reduce = 29%, Cumulative CPU 582.17 sec
2018-11-26 11:44:47,896 Stage-16 map = 100%, reduce = 31%, Cumulative CPU 588.82 sec
2018-11-26 11:44:53,957 Stage-16 map = 100%, reduce = 33%, Cumulative CPU 595.46 sec
2018-11-26 11:45:00,032 Stage-16 map = 100%, reduce = 34%, Cumulative CPU 602.0 sec
2018-11-26 11:45:06,098 Stage-16 map = 100%, reduce = 36%, Cumulative CPU 608.77 sec
2018-11-26 11:45:12,169 Stage-16 map = 100%, reduce = 38%, Cumulative CPU 615.37 sec
2018-11-26 11:45:18,247 Stage-16 map = 100%, reduce = 40%, Cumulative CPU 622.2 sec
2018-11-26 11:45:24,310 Stage-16 map = 100%, reduce = 41%, Cumulative CPU 628.84 sec
2018-11-26 11:45:30,376 Stage-16 map = 100%, reduce = 43%, Cumulative CPU 635.24 sec
2018-11-26 11:45:36,441 Stage-16 map = 100%, reduce = 45%, Cumulative CPU 641.83 sec
2018-11-26 11:45:42,506 Stage-16 map = 100%, reduce = 47%, Cumulative CPU 648.49 sec
2018-11-26 11:45:48,573 Stage-16 map = 100%, reduce = 48%, Cumulative CPU 655.17 sec
2018-11-26 11:45:53,645 Stage-16 map = 100%, reduce = 50%, Cumulative CPU 661.65 sec
2018-11-26 11:45:59,711 Stage-16 map = 100%, reduce = 52%, Cumulative CPU 668.23 sec
2018-11-26 11:46:05,801 Stage-16 map = 100%, reduce = 53%, Cumulative CPU 674.6 sec
2018-11-26 11:46:11,870 Stage-16 map = 100%, reduce = 55%, Cumulative CPU 681.05 sec
2018-11-26 11:46:17,950 Stage-16 map = 100%, reduce = 57%, Cumulative CPU 687.48 sec
2018-11-26 11:46:24,030 Stage-16 map = 100%, reduce = 59%, Cumulative CPU 693.9 sec
2018-11-26 11:46:30,109 Stage-16 map = 100%, reduce = 60%, Cumulative CPU 700.45 sec
2018-11-26 11:46:36,181 Stage-16 map = 100%, reduce = 62%, Cumulative CPU 707.11 sec
2018-11-26 11:46:42,252 Stage-16 map = 100%, reduce = 64%, Cumulative CPU 713.53 sec
2018-11-26 11:46:48,319 Stage-16 map = 100%, reduce = 66%, Cumulative CPU 720.25 sec
2018-11-26 11:46:54,393 Stage-16 map = 100%, reduce = 67%, Cumulative CPU 726.78 sec
2018-11-26 11:47:00,456 Stage-16 map = 100%, reduce = 69%, Cumulative CPU 733.26 sec
2018-11-26 11:47:06,519 Stage-16 map = 100%, reduce = 71%, Cumulative CPU 739.83 sec
2018-11-26 11:47:12,586 Stage-16 map = 100%, reduce = 72%, Cumulative CPU 746.4 sec
2018-11-26 11:47:18,648 Stage-16 map = 100%, reduce = 74%, Cumulative CPU 753.03 sec
2018-11-26 11:47:23,698 Stage-16 map = 100%, reduce = 76%, Cumulative CPU 759.51 sec
2018-11-26 11:47:29,766 Stage-16 map = 100%, reduce = 78%, Cumulative CPU 766.07 sec
2018-11-26 11:47:36,849 Stage-16 map = 100%, reduce = 79%, Cumulative CPU 772.78 sec
2018-11-26 11:47:41,919 Stage-16 map = 100%, reduce = 81%, Cumulative CPU 779.41 sec
2018-11-26 11:47:47,977 Stage-16 map = 100%, reduce = 83%, Cumulative CPU 785.8 sec
2018-11-26 11:47:54,037 Stage-16 map = 100%, reduce = 84%, Cumulative CPU 792.53 sec
2018-11-26 11:48:00,113 Stage-16 map = 100%, reduce = 86%, Cumulative CPU 799.01 sec
2018-11-26 11:48:06,191 Stage-16 map = 100%, reduce = 88%, Cumulative CPU 805.42 sec
2018-11-26 11:48:12,266 Stage-16 map = 100%, reduce = 90%, Cumulative CPU 812.04 sec
2018-11-26 11:48:18,333 Stage-16 map = 100%, reduce = 91%, Cumulative CPU 818.58 sec
2018-11-26 11:48:24,412 Stage-16 map = 100%, reduce = 93%, Cumulative CPU 825.15 sec
2018-11-26 11:48:30,477 Stage-16 map = 100%, reduce = 95%, Cumulative CPU 831.76 sec
2018-11-26 11:48:36,543 Stage-16 map = 100%, reduce = 97%, Cumulative CPU 838.34 sec
2018-11-26 11:48:42,604 Stage-16 map = 100%, reduce = 98%, Cumulative CPU 845.1 sec
2018-11-26 11:48:48,672 Stage-16 map = 100%, reduce = 100%, Cumulative CPU 851.87 sec
MapReduce Total cumulative CPU time: 14 minutes 11 seconds 870 msec
Ended Job = job_1543224159463_0005
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2018-11-26 11:48:55 Dump the side-table for tag: 1 with group count: 1 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10030/HashTable-Stage-27/MapJoin-mapfile61--.hashtable
Execution completed successfully
MapredLocal task succeeded
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2018-11-26 11:49:01 Dump the side-table for tag: 1 with group count: 1 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10032/HashTable-Stage-31/MapJoin-mapfile91--.hashtable
2018-11-26 11:49:01 Uploaded 1 File to: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10032/HashTable-Stage-31/MapJoin-mapfile91--.hashtable (278 bytes)
2018-11-26 11:49:01 End of local task; Time Taken: 0.98 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 3 out of 18
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0006, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0006/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0006
Hadoop job information for Stage-27: number of mappers: 1; number of reducers: 0
2018-11-26 11:49:05,901 Stage-27 map = 0%, reduce = 0%
2018-11-26 11:49:09,982 Stage-27 map = 100%, reduce = 0%, Cumulative CPU 1.97 sec
MapReduce Total cumulative CPU time: 1 seconds 970 msec
Ended Job = job_1543224159463_0006
Launching Job 4 out of 18
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0007, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0007/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0007
Hadoop job information for Stage-31: number of mappers: 1; number of reducers: 0
2018-11-26 11:49:19,666 Stage-31 map = 0%, reduce = 0%
2018-11-26 11:49:25,749 Stage-31 map = 100%, reduce = 0%, Cumulative CPU 4.93 sec
MapReduce Total cumulative CPU time: 4 seconds 930 msec
Ended Job = job_1543224159463_0007
Stage-39 is filtered out by condition resolver.
Stage-40 is selected by condition resolver.
Stage-2 is filtered out by condition resolver.
Stage-37 is filtered out by condition resolver.
Stage-38 is selected by condition resolver.
Stage-12 is filtered out by condition resolver.
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2018-11-26 11:49:31 Dump the side-table for tag: 0 with group count: 5 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10028/HashTable-Stage-25/MapJoin-mapfile50--.hashtable
2018-11-26 11:49:31 Uploaded 1 File to: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10028/HashTable-Stage-25/MapJoin-mapfile50--.hashtable (485 bytes)
Execution completed successfully
MapredLocal task succeeded
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2018-11-26 11:49:38 Processing rows: 200000 Hashtable size: 199999 Memory usage: 418965208 percentage: 0.055
2018-11-26 11:49:38 Dump the side-table for tag: 0 with group count: 200535 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10024/HashTable-Stage-29/MapJoin-mapfile80--.hashtable
2018-11-26 11:49:38 End of local task; Time Taken: 1.252 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 7 out of 18
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0008, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0008/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0008
Hadoop job information for Stage-25: number of mappers: 1; number of reducers: 0
2018-11-26 11:49:43,206 Stage-25 map = 0%, reduce = 0%
2018-11-26 11:49:50,303 Stage-25 map = 100%, reduce = 0%, Cumulative CPU 6.84 sec
MapReduce Total cumulative CPU time: 6 seconds 840 msec
Ended Job = job_1543224159463_0008
Launching Job 8 out of 18
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0009, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0009/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0009
Hadoop job information for Stage-29: number of mappers: 9; number of reducers: 0
2018-11-26 11:50:00,128 Stage-29 map = 0%, reduce = 0%
2018-11-26 11:50:15,298 Stage-29 map = 6%, reduce = 0%, Cumulative CPU 16.61 sec
2018-11-26 11:50:21,357 Stage-29 map = 10%, reduce = 0%, Cumulative CPU 22.84 sec
2018-11-26 11:50:22,370 Stage-29 map = 11%, reduce = 0%, Cumulative CPU 24.22 sec
2018-11-26 11:50:37,533 Stage-29 map = 16%, reduce = 0%, Cumulative CPU 40.45 sec
2018-11-26 11:50:43,590 Stage-29 map = 21%, reduce = 0%, Cumulative CPU 46.72 sec
2018-11-26 11:50:44,605 Stage-29 map = 22%, reduce = 0%, Cumulative CPU 48.25 sec
2018-11-26 11:50:58,748 Stage-29 map = 29%, reduce = 0%, Cumulative CPU 64.95 sec
2018-11-26 11:51:04,806 Stage-29 map = 32%, reduce = 0%, Cumulative CPU 71.36 sec
2018-11-26 11:51:05,832 Stage-29 map = 33%, reduce = 0%, Cumulative CPU 72.0 sec
2018-11-26 11:51:19,978 Stage-29 map = 40%, reduce = 0%, Cumulative CPU 88.45 sec
2018-11-26 11:51:26,035 Stage-29 map = 43%, reduce = 0%, Cumulative CPU 94.62 sec
2018-11-26 11:51:27,050 Stage-29 map = 44%, reduce = 0%, Cumulative CPU 95.41 sec
2018-11-26 11:51:41,186 Stage-29 map = 51%, reduce = 0%, Cumulative CPU 111.58 sec
2018-11-26 11:51:47,243 Stage-29 map = 54%, reduce = 0%, Cumulative CPU 117.8 sec
2018-11-26 11:51:48,257 Stage-29 map = 56%, reduce = 0%, Cumulative CPU 118.46 sec
2018-11-26 11:52:02,389 Stage-29 map = 60%, reduce = 0%, Cumulative CPU 135.13 sec
2018-11-26 11:52:08,443 Stage-29 map = 65%, reduce = 0%, Cumulative CPU 141.45 sec
2018-11-26 11:52:09,455 Stage-29 map = 67%, reduce = 0%, Cumulative CPU 142.95 sec
2018-11-26 11:52:24,604 Stage-29 map = 73%, reduce = 0%, Cumulative CPU 159.19 sec
2018-11-26 11:52:30,657 Stage-29 map = 76%, reduce = 0%, Cumulative CPU 165.4 sec
2018-11-26 11:52:31,669 Stage-29 map = 78%, reduce = 0%, Cumulative CPU 166.5 sec
2018-11-26 11:52:44,797 Stage-29 map = 83%, reduce = 0%, Cumulative CPU 182.76 sec
2018-11-26 11:52:50,859 Stage-29 map = 87%, reduce = 0%, Cumulative CPU 189.04 sec
2018-11-26 11:52:52,884 Stage-29 map = 89%, reduce = 0%, Cumulative CPU 190.36 sec
2018-11-26 11:53:01,985 Stage-29 map = 100%, reduce = 0%, Cumulative CPU 201.38 sec
MapReduce Total cumulative CPU time: 3 minutes 21 seconds 380 msec
Ended Job = job_1543224159463_0009
Stage-35 is selected by condition resolver.
Stage-36 is filtered out by condition resolver.
Stage-3 is filtered out by condition resolver.
Launching Job 10 out of 18
Number of reduce tasks not specified. Estimated from input data size: 2
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0010, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0010/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0010
Hadoop job information for Stage-13: number of mappers: 2; number of reducers: 2
2018-11-26 11:53:12,104 Stage-13 map = 0%, reduce = 0%
2018-11-26 11:53:27,270 Stage-13 map = 14%, reduce = 0%, Cumulative CPU 16.84 sec
2018-11-26 11:53:33,329 Stage-13 map = 19%, reduce = 0%, Cumulative CPU 24.38 sec
2018-11-26 11:53:39,387 Stage-13 map = 29%, reduce = 0%, Cumulative CPU 31.96 sec
2018-11-26 11:53:45,446 Stage-13 map = 45%, reduce = 0%, Cumulative CPU 39.53 sec
2018-11-26 11:53:46,459 Stage-13 map = 50%, reduce = 0%, Cumulative CPU 40.58 sec
2018-11-26 11:53:56,569 Stage-13 map = 100%, reduce = 0%, Cumulative CPU 50.97 sec
2018-11-26 11:54:09,704 Stage-13 map = 100%, reduce = 50%, Cumulative CPU 65.58 sec
2018-11-26 11:54:21,842 Stage-13 map = 100%, reduce = 100%, Cumulative CPU 79.57 sec
MapReduce Total cumulative CPU time: 1 minutes 19 seconds 570 msec
Ended Job = job_1543224159463_0010
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2018-11-26 11:54:28 Processing rows: 200000 Hashtable size: 199999 Memory usage: 597816664 percentage: 0.078
2018-11-26 11:54:29 Uploaded 1 File to: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10018/HashTable-Stage-21/MapJoin-mapfile21--.hashtable (17391919 bytes)
Execution completed successfully
MapredLocal task succeeded
Launching Job 11 out of 18
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0011, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0011/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0011
Hadoop job information for Stage-21: number of mappers: 1; number of reducers: 0
2018-11-26 11:54:36,037 Stage-21 map = 0%, reduce = 0%
2018-11-26 11:54:42,134 Stage-21 map = 100%, reduce = 0%, Cumulative CPU 7.51 sec
MapReduce Total cumulative CPU time: 7 seconds 510 msec
Ended Job = job_1543224159463_0011
Stage-33 is filtered out by condition resolver.
Stage-34 is selected by condition resolver.
Stage-4 is filtered out by condition resolver.
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2018-11-26 11:54:47 Starting to launch local task to process map join; maximum memory = 76357304322018-11-26 11:54:48 Dump the side-table for tag: 0 with group count: 63458 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_11-25-21_864_8725952102284433838-1/-local-10016/HashTable-Stage-19/MapJoin-mapfile10--.hashtable
2018-11-26 11:54:48 End of local task; Time Taken: 1.576 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 13 out of 18
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0012, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0012/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0012
Hadoop job information for Stage-19: number of mappers: 2; number of reducers: 0
2018-11-26 11:54:55,715 Stage-19 map = 0%, reduce = 0%
2018-11-26 11:55:16,945 Stage-19 map = 27%, reduce = 0%, Cumulative CPU 22.34 sec
2018-11-26 11:55:23,010 Stage-19 map = 50%, reduce = 0%, Cumulative CPU 27.98 sec
2018-11-26 11:55:29,089 Stage-19 map = 100%, reduce = 0%, Cumulative CPU 34.39 sec
MapReduce Total cumulative CPU time: 34 seconds 390 msec
Ended Job = job_1543224159463_0012
Launching Job 14 out of 18
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0013, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0013/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0013
Hadoop job information for Stage-5: number of mappers: 1; number of reducers: 1
2018-11-26 11:55:38,574 Stage-5 map = 0%, reduce = 0%
2018-11-26 11:55:43,656 Stage-5 map = 100%, reduce = 0%, Cumulative CPU 4.1 sec
2018-11-26 11:55:47,735 Stage-5 map = 100%, reduce = 100%, Cumulative CPU 5.81 sec
MapReduce Total cumulative CPU time: 5 seconds 810 msec
Ended Job = job_1543224159463_0013
MapReduce Jobs Launched:
Stage-Stage-8: Map: 56 Reduce: 58 Cumulative CPU: 733.11 sec HDFS Read: 14663688186 HDFS Write: 17767081 SUCCESS
Stage-Stage-16: Map: 56 Reduce: 58 Cumulative CPU: 851.87 sec HDFS Read: 14663630200 HDFS Write: 2376065461 SUCCESS
Stage-Stage-27: Map: 1 Cumulative CPU: 1.97 sec HDFS Read: 11155 HDFS Write: 316 SUCCESS
Stage-Stage-31: Map: 1 Cumulative CPU: 4.93 sec HDFS Read: 142885450 HDFS Write: 4198970 SUCCESS
Stage-Stage-25: Map: 1 Cumulative CPU: 6.84 sec HDFS Read: 142885644 HDFS Write: 36759506 SUCCESS
Stage-Stage-29: Map: 9 Cumulative CPU: 201.38 sec HDFS Read: 2376141958 HDFS Write: 303989920 SUCCESS
Stage-Stage-13: Map: 2 Reduce: 2 Cumulative CPU: 79.57 sec HDFS Read: 304006424 HDFS Write: 303989257 SUCCESS
Stage-Stage-21: Map: 1 Cumulative CPU: 7.51 sec HDFS Read: 36766926 HDFS Write: 13649545 SUCCESS
Stage-Stage-19: Map: 2 Cumulative CPU: 34.39 sec HDFS Read: 304289088 HDFS Write: 9803471 SUCCESS
Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 5.81 sec HDFS Read: 9814528 HDFS Write: 21211 SUCCESS
Total MapReduce CPU Time Spent: 32 minutes 7 seconds 380 msec
OK
9999.91 Supplier#000553792 FRANCE 15053761 Manufacturer#5 tBcEuX13HkIqLfWGlPOByrBQMnt5,NlDwNkR 16-446-683-1939 lar foxes. pending deposits boost furiously. slyly ironic pinto beans
9999.79 Supplier#000712808 UNITED KINGDOM 16212775 Manufacturer#2 xtpkyW4jFq3iJSpC34rTWz 33-559-792-5549 uld have to use carefully slyly even packages. furiously r
9999.66 Supplier#000043638 RUSSIA 1293634 Manufacturer#4 pRnD Nz9fBDFin 32-383-297-3784 rts wake regularly against the quickly silent deposits. slyly
9999.28 Supplier#000494480 ROMANIA 13244466 Manufacturer#1 hNB2qcYmnyeqrd em,o29TxzLwwrl 29-756-312-1779 ly. slyly final pains detect furiously qu
9998.48 Supplier#000073838 FRANCE 7823830 Manufacturer#3 m 5KL6JavtSydTzxsVHZtrmD4Ty1q0FtGbNg 16-551-165-1784 to beans are ruthlessly quick packages. requests cajole blithely furiously sp
9998.38 Supplier#000945885 GERMANY 18445848 Manufacturer#5 p3R630qLsV 17-514-300-6011 usual deposits cajole requests. ironic theodolit
9998.25 Supplier#000419963 RUSSIA 18669908 Manufacturer#3 drB,x4NJ35x,qNm usej 32-697-264-8862 even platelets-- slyly special excuses
9997.58 Supplier#000991966 RUSSIA 12991965 Manufacturer#2 IqRrBGCrKrQ2eZ 32-573-261-5414 pinto beans thrash furiously caref
9997.46 Supplier#000906546 FRANCE 656545 Manufacturer#4 aMZ13ITqaYCGwqW42qXJ92ho 16-249-122-4511 endencies above the furiously even
9997.46 Supplier#000906546 FRANCE 5406535 Manufacturer#3 aMZ13ITqaYCGwqW42qXJ92ho 16-249-122-4511 endencies above the furiously even
9997.12 Supplier#000809701 ROMANIA 8809700 Manufacturer#1 j2bBCxp7sp0qXqBy1fIIUuo42HbmUcGzjLCaX 29-896-280-6122 urts wake. unusual, express deposits use quickly. unusual, ironic requests sleep carefully ab
9997.06 Supplier#000725417 GERMANY 3225410 Manufacturer#2 10iRwq3,LeyTjZx8G 17-577-582-2010 arefully dogged accounts. final, pending asymptote
9996.49 Supplier#000521116 ROMANIA 18771061 Manufacturer#4 Z5WUy6xrdp5veQ 29-579-640-5514 cajole alongside of the platelets! even pinto beans cajole. blithely express
9995.96 Supplier#000629520 FRANCE 379519 Manufacturer#5 kjmenylD2Ut5JJxh1n8P1,cl,XqR 16-785-547-3814 foxes detect blithely. even, ironic courts a
9995.93 Supplier#000621875 ROMANIA 10621874 Manufacturer#4 9oDM1ywnyg3ss 29-380-685-3475 equests play-- unusual theodolites above the ironic packages are finally along the expre
9995.52 Supplier#000933993 FRANCE 19433954 Manufacturer#3 1S3y1tCyznItAnRnYoKGtUo3FF 16-618-664-9796 carefully across the quickly ironic instructions. requests use carefully blithely even depo
9995.52 Supplier#000933993 FRANCE 19933992 Manufacturer#1 1S3y1tCyznItAnRnYoKGtUo3FF 16-618-664-9796 carefully across the quickly ironic instructions. requests use carefully blithely even depo
9995.34 Supplier#000190688 UNITED KINGDOM 19690649 Manufacturer#3 pSlhz0825vaM,Ly8Y4VLCtdGzFNpsDh35U 33-424-942-1566 side of the carefully special pinto
9995.22 Supplier#000047386 ROMANIA 1797384 Manufacturer#3 yuT9CtXmek 6qG6myKjMZt 29-218-916-4495 instructions. blithely ironic theodo
9994.94 Supplier#000679327 ROMANIA 5679326 Manufacturer#4 pfuIVpzrwWJID 29-746-305-6710 ess accounts against the blithely ironic foxes
9994.85 Supplier#000581928 GERMANY 6331921 Manufacturer#5 UbBVSalX1YIGla9W564VvUVr9F8a83qdz 17-230-642-6940 e evenly slyly special asymptotes. blithely pending packages cajole slyly unusu
9994.85 Supplier#000581928 GERMANY 7831906 Manufacturer#4 UbBVSalX1YIGla9W564VvUVr9F8a83qdz 17-230-642-6940 e evenly slyly special asymptotes. blithely pending packages cajole slyly unusu
9994.58 Supplier#000284153 GERMANY 1284152 Manufacturer#1 bSzier0F7Ho6iDe2w4 17-181-297-2119 ckly special deposits. final requests boost furiously slyly regular request
9994.37 Supplier#000030084 GERMANY 1780082 Manufacturer#1 gBEvSkyW o1uHea0CV,oHtkTTVW 17-519-171-6883 pinto beans sleep fluffily alongside of the slyly special deposits. slyly pendi
9994.35 Supplier#000656604 UNITED KINGDOM 6156591 Manufacturer#1 BIRI28lWVnFlCNOgHATLsW4NsPIjHy RyI 33-319-512-1645 pending theodolites. slyly ironic instructions hinder deposits. fluffily s
9993.98 Supplier#000831504 ROMANIA 581503 Manufacturer#2 U iIIVbjEg 29-775-353-7830 en packages haggle furiously. foxes cajole
9993.97 Supplier#000566725 FRANCE 1316723 Manufacturer#4 xRvrlNXz9DfV 16-151-920-7745 long the fluffily special instructions. dogged dolphi
9993.64 Supplier#000087248 GERMANY 10837237 Manufacturer#2 1OEtMvFhbyIQFb,LVXuhQOBfnpcMo3AUDutKs 17-883-366-3990 s. carefully brave theodolites according to the carefully ironic packages run enticingly ruthle
9993.32 Supplier#000119310 FRANCE 9619291 Manufacturer#4 U,0uDux9oyddMH 16-369-729-8742 xes sleep slyly according to the carefully pending deposits. regular, regular instructions wake car
9992.68 Supplier#000961580 UNITED KINGDOM 14461551 Manufacturer#1 me5yS3yU9nKFozDfEW 33-273-192-1544 y about the slyly bold excuses.
9992.66 Supplier#000541492 ROMANIA 18541491 Manufacturer#5 S97sT1WIquqWxhV7tS hJR 29-172-548-6729 heodolites run final accounts. silent, ironic packages haggle. express frets integrate
9992.64 Supplier#000500316 RUSSIA 2250313 Manufacturer#1 4pQ5 cL78TSbYGs5gP 32-479-249-8378 express deposits use never furiously even deposits. carefully ironic depths h
9991.84 Supplier#000462341 UNITED KINGDOM 4712328 Manufacturer#1 U4vG9,xUIV sbvG 4SR8wlbTZd4WDiSkbKTDDi7 33-607-871-1127 carefully bold theodolites among the slyly ironic asymptotes wake unus
9991.80 Supplier#000176854 ROMANIA 14926839 Manufacturer#4 i8hTQeqlqb 29-356-560-2367 the ironic requests. furiously even instructions a
9991.48 Supplier#000519595 FRANCE 19594 Manufacturer#2 2kGFZcjnkUkz 16-317-132-3151 ts sleep among the carefully bold du
9991.36 Supplier#000686364 RUSSIA 11436352 Manufacturer#3 VKj3fW38SILlYACTVnfx29AicQZynz 32-660-977-2923 al asymptotes. ironic, fi
9990.90 Supplier#000501829 UNITED KINGDOM 6751810 Manufacturer#3 O,JerEuNnVslc9YdBh6UMh7SuTm9or01l1 33-559-653-2157 play quickly carefully final packages. furiously silent forges sleep slyly slyly regular fret
9990.87 Supplier#000358954 ROMANIA 9108944 Manufacturer#4 9VIZZ6cY986 29-345-542-5594 ly bold requests are slyly along the care
9990.65 Supplier#000661325 RUSSIA 9661324 Manufacturer#4 Jqvf7BMv3Q 32-312-563-2260 le under the bold accounts. quickly ironic requests wake blithe
9990.35 Supplier#000973997 ROMANIA 8223972 Manufacturer#3 DHoJf2mGnufyASrJ3ZxyiA 29-492-125-3082 ost blithely unusual courts. slyly even instructions print across the carefully pendi
9990.05 Supplier#000008890 ROMANIA 9008889 Manufacturer#2 6lmM3OrUukwhKXY0zqypO2qEsgj 29-208-398-4306 ts. unusual deposits haggle furiously along the even
9990.00 Supplier#000868386 ROMANIA 5618380 Manufacturer#1 jwy2mc4cOPmkvAJoRk3Si6jo 29-623-365-9495 to beans sleep carefully
9989.49 Supplier#000310168 ROMANIA 4560155 Manufacturer#3 5FRMWpZz8xWnkvf8wiv 29-127-942-3074 thely regular excuses. quickly unusual requests above the accounts hagg
9989.46 Supplier#000951681 FRANCE 15201635 Manufacturer#2 pRmOVAkAQ4b7YrjiZ di2buDGsgybh199hHQUrS9 16-796-936-7679 out the carefully regular sheaves cajole quickly carefully even instructions. final d
9988.40 Supplier#000465785 GERMANY 13215771 Manufacturer#2 FTnx Wxs7TvUd 17-981-295-1619 pendencies are closely ironic pinto beans. special requests boost carefully
9988.29 Supplier#000273304 ROMANIA 11773281 Manufacturer#1 rVw7T,PbkYA35J, 29-238-280-6619 r accounts. fluffily ironic foxes against the carefully eve
9988.29 Supplier#000180265 UNITED KINGDOM 18930246 Manufacturer#4 MlAO48VNi6chVRM06Q,ahYTf0ATuRck 33-637-426-6508 ly slyly ironic instructions.
9988.23 Supplier#000834777 ROMANIA 4834776 Manufacturer#2 F71Lhq4ZQoxSQZ4224PixZKe 29-851-710-7841 usly pending packages affix busy sentiments. furiou
9987.87 Supplier#000938166 UNITED KINGDOM 938165 Manufacturer#2 hXE4h6PfytF9RBcS7umIYiv0DYvEqoZV48ZEDn4 33-707-278-6306 busily bold platelets. final, slow deposits are. quickly regular Tiresias haggle regular, silent d
9987.37 Supplier#000654604 FRANCE 10154583 Manufacturer#3 SwTO7goWY9BvpPAtZRizvO7zIKD 16-157-531-6909 kages shall sleep. accounts are furiously. slyly final packages nag regul
9987.14 Supplier#000748330 RUSSIA 14748329 Manufacturer#2 QtylBmh4eykG 32-425-724-1398 quickly brave dolphins haggle about the silent pains. blithely expre
9986.72 Supplier#000440480 FRANCE 4440479 Manufacturer#2 qXa1,RdYwc,NVGyYZvtW TaV,t 16-152-525-6670 its are upon the quickly unusual theodolites. slyly bold accounts run caref
9986.46 Supplier#000402551 FRANCE 4652538 Manufacturer#4 eovjUORglIN AYFkr 16-353-107-7318 oss the bold requests. regular, regular accounts u
9985.94 Supplier#000664668 GERMANY 7414660 Manufacturer#2 kYlSiBNmC3itZhkZ4TDXUJwuimtTAaRewdyw6nb 17-104-332-4251 eas haggle slyly. furiously final dugouts cajole slyly across the pending e
9985.85 Supplier#000462601 FRANCE 5462600 Manufacturer#1 OScT0ALeXB9hR7jrRdXMk 16-737-557-8867 special pinto beans. bold, regular packages cajole blithel
9985.85 Supplier#000462601 FRANCE 13712561 Manufacturer#1 OScT0ALeXB9hR7jrRdXMk 16-737-557-8867 special pinto beans. bold, regular packages cajole blithel
9985.77 Supplier#000221145 FRANCE 8471120 Manufacturer#1 9U1ezsK3jUkkuWWR7Dm,i 16-648-985-3872 t notornis haggle slyly never final requ
9985.68 Supplier#000652843 RUSSIA 9652842 Manufacturer#4 UZpkWQtgOGSv 32-752-915-2662 ought to boost quickly carefully regular th
9985.54 Supplier#000147207 GERMANY 5647196 Manufacturer#2 rDqExccHg3IBN4k 17-345-679-3036 olites are carefully after the ironic instructi
9985.54 Supplier#000147207 GERMANY 15897191 Manufacturer#3 rDqExccHg3IBN4k 17-345-679-3036 olites are carefully after the ironic instructi
9985.28 Supplier#000781278 FRANCE 17781277 Manufacturer#5 HmdRocEYH3Ci9Lk3CPn 16-548-672-9730 thely unusual requests-- fluffily pending theodolites sleep. unusual hockey players wake blithely b
9984.95 Supplier#000888199 ROMANIA 15388168 Manufacturer#2 9Ueet9zZD4ImX1A6cIgxBX1bFoNeHR 29-392-570-4019 its. theodolites grow carefully. blithely express
9984.75 Supplier#000158451 UNITED KINGDOM 12908438 Manufacturer#5 QYb8yJDmZjqFrFrj7s6 ml,0nlLXS4wx 33-823-727-4570 riously even theodolites. bold pinto beans wake furiously express, pending
9984.21 Supplier#000035131 RUSSIA 5035130 Manufacturer#2 8ef9GOD3X6JLav5H4X2XqOLICJW 32-993-135-7694 ithely even deposits across the quickly pending foxes are sp
9984.21 Supplier#000035131 RUSSIA 7285109 Manufacturer#1 8ef9GOD3X6JLav5H4X2XqOLICJW 32-993-135-7694 ithely even deposits across the quickly pending foxes are sp
9982.98 Supplier#000570639 UNITED KINGDOM 9820611 Manufacturer#3 lsUjbqvVhmYb 33-866-160-1846 about the ruthlessly ironic accounts. f
9982.68 Supplier#000817393 ROMANIA 7317378 Manufacturer#2 N40wT6nCOr7ruAfE toVaXZYI71CrL 29-393-956-9323 counts are furiously slowly silent theodol
9982.60 Supplier#000166549 GERMANY 16166548 Manufacturer#3 iR5M8F,KRj2kPQeoPRXm7OLv7m,W7BlQRqJsRG 17-538-702-3077 ic requests might cajole slyly. blithely pending deposits alongside of the express, unusual deposits
9982.44 Supplier#000127488 UNITED KINGDOM 16627455 Manufacturer#4 29x5,nFa22BWQ 33-300-145-5438 ar platelets around the blithely express pinto beans wake quickly fluffily regu
9982.44 Supplier#000127488 UNITED KINGDOM 18877469 Manufacturer#1 29x5,nFa22BWQ 33-300-145-5438 ar platelets around the blithely express pinto beans wake quickly fluffily regu
9982.12 Supplier#000115580 RUSSIA 5365564 Manufacturer#2 B8gpWJE0s cO8K 32-270-216-1537 y regular ideas: excuses serve slyly. furiously special excuses wake. fur
9982.08 Supplier#000412545 GERMANY 15912514 Manufacturer#1 7dBjYswVlAoyrStC6vw7 PX7 LweQzuzUQJN 17-941-599-3043 breach bravely. quickly ironic foxes c
9981.78 Supplier#000678938 FRANCE 3928928 Manufacturer#4 9hTlqW1qGC4WI0TL3Kr43T,yfrTlPKh 16-647-872-9073 ular dependencies are carefully carefully unus
9981.77 Supplier#000863798 FRANCE 19363759 Manufacturer#3 9HzFJWbs2FV5ZSBwbZiPcAW6y0Qx8acxxG5G 16-431-734-7330 platelets; carefully regul
9980.78 Supplier#000568158 UNITED KINGDOM 11318146 Manufacturer#4 tXGVtLtfFuSzc3C2aqnTh0Zlb21 33-837-813-1301 quickly pending packages. express deposits integrate furiously furiously u
9980.75 Supplier#000663992 FRANCE 16663991 Manufacturer#2 8RmFzsMsl8SYTRTTCXEm0b8KxWJ,av 16-397-496-7983 e quickly. express waters cajole. even d
9980.75 Supplier#000663992 FRANCE 19663991 Manufacturer#2 8RmFzsMsl8SYTRTTCXEm0b8KxWJ,av 16-397-496-7983 e quickly. express waters cajole. even d
9980.32 Supplier#000983116 GERMANY 4733111 Manufacturer#3 aIAuv5nN,pNg 17-549-765-5101 p fluffily blithely ironic foxes. slyly unusual accounts nag bravel
9980.23 Supplier#000677829 GERMANY 16177796 Manufacturer#1 13rA 40NxvQnIhwWq9yaZaU 17-994-618-4051 requests nag slyly blithely silent deposits. carefully careful r
9979.99 Supplier#000305456 GERMANY 7055448 Manufacturer#3 5,Fuu,MnF8 y3b 17-298-484-7217 alongside of the furiously ironic pack
9978.98 Supplier#000605960 ROMANIA 11605959 Manufacturer#4 ItdfsL7Louj925RW4lRzdDZoggcJBfuqhH1tXWgh 29-168-639-5614 quickly silent foxes haggle slyly sheaves. silent packages
9978.85 Supplier#000677865 UNITED KINGDOM 8927840 Manufacturer#5 Yej1oazVxYKe m 33-371-855-5232 theodolites above the carefully special pinto beans nag regular, special ideas. blithely
9978.67 Supplier#000678241 RUSSIA 13178214 Manufacturer#3 CcK1IzGlhz6Rrkuhpns85aD3hL6IBfF04f 32-249-278-7500 ons alongside of the blithely even patterns detect after the furiously ironic excuses. quickly final
9978.34 Supplier#000995269 RUSSIA 4995268 Manufacturer#1 SUFtRF3ZIE 32-119-822-3985 ly. quickly ironic foxes sleep carefully against the care
9978.03 Supplier#000435024 UNITED KINGDOM 16934991 Manufacturer#2 kAIU9VPtMigbIfX4fAw0TtiiHHPyKRxTS 33-860-826-6112 across the blithely bold accounts play carefully across the packages. ironic courts wake
9977.74 Supplier#000021233 RUSSIA 5771227 Manufacturer#3 NdWiYW,vXPi8bSWczEbm 32-745-113-6577 yly carefully special theodolites. courts sleep slyly even platelets
9977.62 Supplier#000524020 ROMANIA 15023989 Manufacturer#4 Otr5VYKIKC779nzx4Epi 29-384-107-4107 wake blithely after the pinto beans. fluffily unusual theodolites sleep. fluffily bold packages hag
9977.18 Supplier#000268615 ROMANIA 10768594 Manufacturer#5 j4btyGpi5rbsgz9Ipi 29-699-155-6773 furiously blithely even dep
9976.46 Supplier#000114393 ROMANIA 16364344 Manufacturer#2 OoaLhUmNSZrgp9HKUB7nue0Uey3mFrbbT5dhdXy2 29-339-128-4329 en requests use pending pearls. bold requests detect against the furiously bold deposits. acco
9976.18 Supplier#000704226 ROMANIA 7704225 Manufacturer#2 5hLo q,jGwi 29-160-200-2666 unts. even accounts according to the deposits wake ironically instead
9975.61 Supplier#000198104 RUSSIA 12198103 Manufacturer#4 Xq,1,A0WQzNVjrAYEL 32-105-406-2475 usly regular requests. blithely regular ac
9975.55 Supplier#000688926 ROMANIA 6438919 Manufacturer#5 lLXFejNOTLiUqCmMdQ7alg 29-387-293-3093 tructions cajole. dogged packages affix slyly about t
9975.47 Supplier#000871156 GERMANY 621155 Manufacturer#4 FOOcEQG1UyiHpEI12Upj8UZfPu89d 17-434-616-6758 quickly about the carefully unusual accounts. ironic, even packages boost special theodoli
9974.96 Supplier#000151189 UNITED KINGDOM 9151188 Manufacturer#2 A4 x,AfJ77S8W0 33-567-998-4910 excuses. requests detect. s
9974.93 Supplier#000072260 ROMANIA 5572249 Manufacturer#4 ijrpECIzgkK,qi83qjKtf0qhMX4O3i17i 29-745-482-4154 its above the special deposits engage slyly among the unusual, even asymptotes. carefull
9974.75 Supplier#000820629 RUSSIA 12070592 Manufacturer#1 xUGwDWocDvzGzdPRoZkkkO 32-632-470-6017 ep after the blithe, unusual packages. idly ironic deposits affix furiously carefully regular p
9974.72 Supplier#000969577 FRANCE 6969576 Manufacturer#2 MU018IXYj2TeeM 16-216-897-6924 nstructions do eat ironic accounts. slyly express instructions sublate after the furiously quick re
9974.71 Supplier#000293943 FRANCE 5543927 Manufacturer#5 Vo,kx8AyifEL,Rd8TXxAh3PURIi5k8cQLUhs0s6o 16-654-990-6349 eep after the carefully special ideas. blithel
9974.69 Supplier#000876775 FRANCE 16376742 Manufacturer#3 bB,zykm0iq 16-616-977-8648 wly ironic ideas cajole atop the furiously even courts. furiously even acc
9974.13 Supplier#000534756 ROMANIA 16034723 Manufacturer#5 yyZqhbKst3dNzAFH4ioIDocN 29-172-184-8030 . fluffily ironic requests could
Time taken: 1826.913 seconds, Fetched: 100 row(s)
Copy
<script type="js">
// Q02
// It's not really a grid query as all is resolved in dimension tables
// This query should be run in coordinator !
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
s_acctbal,
s_name,
n_name,
p_partkey,
p_mfgr,
s_address,
s_phone,
s_comment
FROM part, supplier, partsupp, nation, region
WHERE
p_partkey = ps_partkey
AND s_suppkey = ps_suppkey
AND p_size = 15
AND p_type LIKE '%BRASS'
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'EUROPE'
AND ps_supplycost = (
SELECT
MIN(ps_supplycost)
FROM
partsupp,
supplier,
nation,
region
WHERE
p_partkey = ps_partkey
AND s_suppkey = ps_suppkey
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'EUROPE'
)
`
,
`
select FIRST 100
s_acctbal,
s_name,
n_name,
p_partkey,
p_mfgr,
s_address,
s_phone,
s_comment
FROM \${temp}
ORDER BY s_acctbal DESC, n_name, s_name, p_partkey
`
);
</script>
2.3 Substitution Parameters
Values for the following substitution parameter must be generated and used to build the executable query text:
- SIZE is randomly selected within [1. 50]
- TYPE is randomly selected within the list Syllable 3 defined for Types in TPCH Clause 4.2.2.13
- REGION is randomly selected within the list of values defined for R_NAME in 4.2.3.
2.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- SIZE = 15
- TYPE = BRASS
- REGION = EUROPE
Sample Output
| s_acctbal | 9938.53 |
| s_name | Supplier#000005359 |
| n_name | UNITED KINGDOM |
| p_partkey | 185358 |
| p_mfgr | Manufacturer#4 |
| s_address | QKuHYh,vZGiwu2FWEJoLDx04 |
| s_phone | 33-429-790-6131 |
| s_comment | uriously regular requests hag |
3 Q3 - Shipping Priority Query
This query retrieves the 10 unshipped orders with the highest value.
3.1 Business Question
The Shipping Priority Query retrieves the shipping priority and potential revenue, defined as the sum of l_extendedprice * (1-l_discount), of the orders having the largest revenue among those that had not been shipped as of a given date. Orders are listed in decreasing order of revenue. If more than 10 unshipped orders exist, only the 10 orders with the largest revenue are listed.
3.2 Functional Query Definition
Return the first 10 selected rows
Copy
SELECT FIRST 10
l_orderkey,
SUM(l_extendedprice * (1 - l_discount)) AS revenue,
o_orderdate,
o_shippriority
FROM
customer,
orders,
lineitem
WHERE c_mktsegment = 'BUILDING'
AND c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate < MDY(3, 15, 1995)
AND l_shipdate > MDY(3, 15, 1995)
GROUP BY
l_orderkey,
o_orderdate,
o_shippriority
ORDER BY
revenue DESC,
o_orderdate
Copy
select
l_orderkey,
sum(l_extendedprice * (1 - l_discount)) as revenue,
o_orderdate,
o_shippriority
from
customer,
orders,
lineitem
where
c_mktsegment = 'BUILDING'
and c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate < '1995-03-22'
and l_shipdate > '1995-03-22'
group by
l_orderkey,
o_orderdate,
o_shippriority
order by
revenue desc,
o_orderdate
limit 10;
Copy
Estimated Cost: 66686904
Estimated # of Rows Returned: 3333572
Temporary Files Required For: Order By Group By
1) informix.customer: SEQUENTIAL SCAN
Filters: informix.customer.c_mktsegment = 'BUILDING'
2) informix.orders: SEQUENTIAL SCAN
Filters:
Table Scan Filters: informix.orders.o_orderdate < 15-03-1995
DYNAMIC HASH JOIN (Build Outer)
Dynamic Hash Filters: informix.customer.c_custkey = informix.orders.o_custkey
3) informix.lineitem: INDEX PATH
Filters: informix.lineitem.l_shipdate > 15-03-1995
(1) Index Name: informix.lineitem_pk
Index Keys: l_orderkey l_linenumber (Serial, fragments: ALL)
Lower Index Filter: informix.lineitem.l_orderkey = informix.orders.o_orderkey
NESTED LOOP JOIN
Copy
hive>
> select
> l_orderkey,
> sum(l_extendedprice * (1 - l_discount)) as revenue,
> o_orderdate,
> o_shippriority
> from
> customer,
> orders,
> lineitem
> where
> c_mktsegment = 'BUILDING'
> and c_custkey = o_custkey
> and l_orderkey = o_orderkey
> and o_orderdate < '1995-03-22'
> and l_shipdate > '1995-03-22'
> group by
> l_orderkey,
> o_orderdate,
> o_shippriority
> order by
> revenue desc,
> o_orderdate
> limit 10;
No Stats for tpch_100@customer, Columns: c_custkey, c_mktsegment
No Stats for tpch_100@orders, Columns: o_orderdate, o_shippriority, o_custkey, o_orderkey
No Stats for tpch_100@lineitem, Columns: l_orderkey, l_extendedprice, l_shipdate, l_discount
Query ID = hadoop_20181126120000_ed613bf4-8baf-4240-8e87-fe59ef56105f
Total jobs = 6
Launching Job 1 out of 6
Number of reduce tasks not specified. Estimated from input data size: 80
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0014, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0014/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0014
Hadoop job information for Stage-1: number of mappers: 77; number of reducers: 80
2018-11-26 12:00:05,182 Stage-1 map = 0%, reduce = 0%
2018-11-26 12:00:15,321 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 10.03 sec
2018-11-26 12:00:22,429 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 19.96 sec
2018-11-26 12:00:30,539 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 29.71 sec
2018-11-26 12:00:38,635 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 39.86 sec
2018-11-26 12:00:46,727 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 49.92 sec
2018-11-26 12:00:54,819 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 59.71 sec
2018-11-26 12:01:02,902 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 70.59 sec
2018-11-26 12:01:10,989 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 80.44 sec
2018-11-26 12:01:19,093 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 90.14 sec
2018-11-26 12:01:27,174 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 100.21 sec
2018-11-26 12:01:35,268 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 110.49 sec
2018-11-26 12:01:43,356 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 120.7 sec
2018-11-26 12:01:50,445 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 130.17 sec
2018-11-26 12:01:58,557 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 140.02 sec
2018-11-26 12:02:06,668 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 150.33 sec
2018-11-26 12:02:14,756 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 160.08 sec
2018-11-26 12:02:22,840 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 170.14 sec
2018-11-26 12:02:30,940 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 180.74 sec
2018-11-26 12:02:39,027 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 190.9 sec
2018-11-26 12:02:47,118 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 200.73 sec
2018-11-26 12:02:55,200 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 210.34 sec
2018-11-26 12:03:03,291 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 220.35 sec
2018-11-26 12:03:11,372 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 230.2 sec
2018-11-26 12:03:18,438 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 240.31 sec
2018-11-26 12:03:26,536 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 250.3 sec
2018-11-26 12:03:34,629 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 259.94 sec
2018-11-26 12:03:42,733 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 270.16 sec
2018-11-26 12:03:50,816 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 280.69 sec
2018-11-26 12:03:58,894 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 290.31 sec
2018-11-26 12:04:06,994 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 300.07 sec
2018-11-26 12:04:15,076 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 310.12 sec
2018-11-26 12:04:23,160 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 319.93 sec
2018-11-26 12:04:31,239 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 330.73 sec
2018-11-26 12:04:39,324 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 340.36 sec
2018-11-26 12:04:46,399 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 350.28 sec
2018-11-26 12:04:55,501 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 360.86 sec
2018-11-26 12:05:02,585 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 371.13 sec
2018-11-26 12:05:11,675 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 381.64 sec
2018-11-26 12:05:18,768 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 391.82 sec
2018-11-26 12:05:26,861 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 402.13 sec
2018-11-26 12:05:34,958 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 412.39 sec
2018-11-26 12:05:43,035 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 422.86 sec
2018-11-26 12:05:51,113 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 433.12 sec
2018-11-26 12:05:59,193 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 443.67 sec
2018-11-26 12:06:07,293 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 453.84 sec
2018-11-26 12:06:15,384 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 464.37 sec
2018-11-26 12:06:23,470 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 474.28 sec
2018-11-26 12:06:31,557 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 483.79 sec
2018-11-26 12:06:39,646 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 493.43 sec
2018-11-26 12:06:47,720 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 503.36 sec
2018-11-26 12:06:55,805 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 513.64 sec
2018-11-26 12:07:02,877 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 523.56 sec
2018-11-26 12:07:10,982 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 533.17 sec
2018-11-26 12:07:19,079 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 543.42 sec
2018-11-26 12:07:27,162 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 553.19 sec
2018-11-26 12:07:35,250 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 562.88 sec
2018-11-26 12:07:43,335 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 572.59 sec
2018-11-26 12:07:51,423 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 582.56 sec
2018-11-26 12:07:59,520 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 592.53 sec
2018-11-26 12:08:07,604 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 603.03 sec
2018-11-26 12:08:15,703 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 612.93 sec
2018-11-26 12:08:22,772 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 622.6 sec
2018-11-26 12:08:30,868 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 632.44 sec
2018-11-26 12:08:39,969 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 642.24 sec
2018-11-26 12:08:47,058 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 652.15 sec
2018-11-26 12:08:55,138 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 661.77 sec
2018-11-26 12:09:01,206 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 670.11 sec
2018-11-26 12:09:07,284 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 677.81 sec
2018-11-26 12:09:13,370 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 685.49 sec
2018-11-26 12:09:19,452 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 693.0 sec
2018-11-26 12:09:25,534 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 700.63 sec
2018-11-26 12:09:31,615 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 707.98 sec
2018-11-26 12:09:37,679 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 716.03 sec
2018-11-26 12:09:43,739 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 724.07 sec
2018-11-26 12:09:49,813 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 731.6 sec
2018-11-26 12:09:54,867 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 737.83 sec
2018-11-26 12:09:58,936 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 742.88 sec
2018-11-26 12:10:03,991 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 747.84 sec
2018-11-26 12:10:08,029 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 752.56 sec
2018-11-26 12:10:12,067 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 757.47 sec
2018-11-26 12:10:16,110 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 762.4 sec
2018-11-26 12:10:20,156 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 767.01 sec
2018-11-26 12:10:24,196 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 772.18 sec
2018-11-26 12:10:28,239 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 776.98 sec
2018-11-26 12:10:32,284 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 782.02 sec
2018-11-26 12:10:35,317 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 786.94 sec
2018-11-26 12:10:40,378 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 791.66 sec
2018-11-26 12:10:43,411 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 796.45 sec
2018-11-26 12:10:47,482 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 801.26 sec
2018-11-26 12:10:52,542 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 806.44 sec
2018-11-26 12:10:55,580 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 811.3 sec
2018-11-26 12:10:59,638 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 816.17 sec
2018-11-26 12:11:03,681 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 821.17 sec
2018-11-26 12:11:07,736 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 826.12 sec
2018-11-26 12:11:11,778 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 830.93 sec
2018-11-26 12:11:15,820 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 835.91 sec
2018-11-26 12:11:19,861 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 840.81 sec
2018-11-26 12:11:23,899 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 845.81 sec
2018-11-26 12:11:27,959 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 850.68 sec
2018-11-26 12:11:32,016 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 855.79 sec
2018-11-26 12:11:36,072 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 860.75 sec
2018-11-26 12:11:40,115 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 866.23 sec
2018-11-26 12:11:45,175 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 871.04 sec
2018-11-26 12:11:49,223 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 875.94 sec
2018-11-26 12:11:53,267 Stage-1 map = 100%, reduce = 35%, Cumulative CPU 880.64 sec
2018-11-26 12:11:57,304 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 885.46 sec
2018-11-26 12:12:01,350 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 890.45 sec
2018-11-26 12:12:05,389 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 895.13 sec
2018-11-26 12:12:09,436 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 900.11 sec
2018-11-26 12:12:12,473 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 904.85 sec
2018-11-26 12:12:17,521 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 909.58 sec
2018-11-26 12:12:20,553 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 914.54 sec
2018-11-26 12:12:25,601 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 919.51 sec
2018-11-26 12:12:28,654 Stage-1 map = 100%, reduce = 46%, Cumulative CPU 924.35 sec
2018-11-26 12:12:32,706 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 929.4 sec
2018-11-26 12:12:36,763 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 934.57 sec
2018-11-26 12:12:40,827 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 939.27 sec
2018-11-26 12:12:44,873 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 944.12 sec
2018-11-26 12:12:48,936 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 949.09 sec
2018-11-26 12:12:52,994 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 953.85 sec
2018-11-26 12:12:57,044 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 958.63 sec
2018-11-26 12:13:01,091 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 963.58 sec
2018-11-26 12:13:05,135 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 968.5 sec
2018-11-26 12:13:09,196 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 973.33 sec
2018-11-26 12:13:13,243 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 978.51 sec
2018-11-26 12:13:17,281 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 983.27 sec
2018-11-26 12:13:21,324 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 988.42 sec
2018-11-26 12:13:25,366 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 993.45 sec
2018-11-26 12:13:29,404 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 998.33 sec
2018-11-26 12:13:33,446 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 1003.23 sec
2018-11-26 12:13:37,498 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 1007.98 sec
2018-11-26 12:13:41,545 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 1012.94 sec
2018-11-26 12:13:45,590 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 1017.96 sec
2018-11-26 12:13:48,626 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 1022.75 sec
2018-11-26 12:13:53,688 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 1027.77 sec
2018-11-26 12:13:56,735 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 1032.8 sec
2018-11-26 12:14:00,781 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 1037.7 sec
2018-11-26 12:14:04,843 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 1042.45 sec
2018-11-26 12:14:08,887 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 1047.41 sec
2018-11-26 12:14:12,929 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 1052.41 sec
2018-11-26 12:14:16,990 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 1057.41 sec
2018-11-26 12:14:21,034 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 1062.68 sec
2018-11-26 12:14:25,091 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 1067.48 sec
2018-11-26 12:14:29,132 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 1072.47 sec
2018-11-26 12:14:33,191 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 1077.38 sec
2018-11-26 12:14:37,246 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 1082.0 sec
2018-11-26 12:14:41,304 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 1086.78 sec
2018-11-26 12:14:45,356 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 1091.67 sec
2018-11-26 12:14:49,394 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 1096.44 sec
2018-11-26 12:14:53,435 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 1101.32 sec
2018-11-26 12:14:57,479 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 1106.01 sec
2018-11-26 12:15:01,522 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 1110.95 sec
2018-11-26 12:15:05,565 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 1115.74 sec
2018-11-26 12:15:09,609 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 1120.82 sec
2018-11-26 12:15:13,648 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 1125.68 sec
2018-11-26 12:15:17,691 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1130.35 sec
2018-11-26 12:15:21,748 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1135.31 sec
MapReduce Total cumulative CPU time: 18 minutes 55 seconds 310 msec
Ended Job = job_1543224159463_0014
Stage-10 is filtered out by condition resolver.
Stage-11 is filtered out by condition resolver.
Stage-2 is selected by condition resolver.
Launching Job 2 out of 6
Number of reduce tasks not specified. Estimated from input data size: 313
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0015, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0015/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0015
Hadoop job information for Stage-2: number of mappers: 299; number of reducers: 313
2018-11-26 12:15:31,489 Stage-2 map = 0%, reduce = 0%
2018-11-26 12:15:46,709 Stage-2 map = 1%, reduce = 0%, Cumulative CPU 17.67 sec
2018-11-26 12:16:08,003 Stage-2 map = 2%, reduce = 0%, Cumulative CPU 43.8 sec
2018-11-26 12:16:29,273 Stage-2 map = 3%, reduce = 0%, Cumulative CPU 70.02 sec
2018-11-26 12:16:50,611 Stage-2 map = 4%, reduce = 0%, Cumulative CPU 96.96 sec
2018-11-26 12:17:11,888 Stage-2 map = 5%, reduce = 0%, Cumulative CPU 123.89 sec
2018-11-26 12:17:33,123 Stage-2 map = 6%, reduce = 0%, Cumulative CPU 150.11 sec
2018-11-26 12:17:53,340 Stage-2 map = 7%, reduce = 0%, Cumulative CPU 176.66 sec
2018-11-26 12:18:14,607 Stage-2 map = 8%, reduce = 0%, Cumulative CPU 203.42 sec
2018-11-26 12:18:35,863 Stage-2 map = 9%, reduce = 0%, Cumulative CPU 229.88 sec
2018-11-26 12:18:57,118 Stage-2 map = 10%, reduce = 0%, Cumulative CPU 256.56 sec
2018-11-26 12:19:19,360 Stage-2 map = 11%, reduce = 0%, Cumulative CPU 282.94 sec
2018-11-26 12:19:40,585 Stage-2 map = 12%, reduce = 0%, Cumulative CPU 309.54 sec
2018-11-26 12:20:01,861 Stage-2 map = 13%, reduce = 0%, Cumulative CPU 336.42 sec
2018-11-26 12:20:23,105 Stage-2 map = 14%, reduce = 0%, Cumulative CPU 363.4 sec
2018-11-26 12:20:45,338 Stage-2 map = 15%, reduce = 0%, Cumulative CPU 389.5 sec
2018-11-26 12:21:07,574 Stage-2 map = 16%, reduce = 0%, Cumulative CPU 415.95 sec
2018-11-26 12:21:28,830 Stage-2 map = 17%, reduce = 0%, Cumulative CPU 441.81 sec
2018-11-26 12:21:50,062 Stage-2 map = 18%, reduce = 0%, Cumulative CPU 467.78 sec
2018-11-26 12:22:12,325 Stage-2 map = 19%, reduce = 0%, Cumulative CPU 494.36 sec
2018-11-26 12:22:33,546 Stage-2 map = 20%, reduce = 0%, Cumulative CPU 520.94 sec
2018-11-26 12:22:54,778 Stage-2 map = 21%, reduce = 0%, Cumulative CPU 547.04 sec
2018-11-26 12:23:17,014 Stage-2 map = 22%, reduce = 0%, Cumulative CPU 574.52 sec
2018-11-26 12:23:38,271 Stage-2 map = 23%, reduce = 0%, Cumulative CPU 600.77 sec
2018-11-26 12:23:59,516 Stage-2 map = 24%, reduce = 0%, Cumulative CPU 627.94 sec
2018-11-26 12:24:21,770 Stage-2 map = 25%, reduce = 0%, Cumulative CPU 655.07 sec
2018-11-26 12:24:42,989 Stage-2 map = 26%, reduce = 0%, Cumulative CPU 682.81 sec
2018-11-26 12:25:04,286 Stage-2 map = 27%, reduce = 0%, Cumulative CPU 708.89 sec
2018-11-26 12:25:25,532 Stage-2 map = 28%, reduce = 0%, Cumulative CPU 736.28 sec
2018-11-26 12:25:47,776 Stage-2 map = 29%, reduce = 0%, Cumulative CPU 763.61 sec
2018-11-26 12:26:09,027 Stage-2 map = 30%, reduce = 0%, Cumulative CPU 790.61 sec
2018-11-26 12:26:30,320 Stage-2 map = 31%, reduce = 0%, Cumulative CPU 817.65 sec
2018-11-26 12:26:51,592 Stage-2 map = 32%, reduce = 0%, Cumulative CPU 843.82 sec
2018-11-26 12:27:13,849 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 870.18 sec
2018-11-26 12:27:35,092 Stage-2 map = 34%, reduce = 0%, Cumulative CPU 897.13 sec
2018-11-26 12:27:55,290 Stage-2 map = 35%, reduce = 0%, Cumulative CPU 924.0 sec
2018-11-26 12:28:16,527 Stage-2 map = 36%, reduce = 0%, Cumulative CPU 949.94 sec
2018-11-26 12:28:37,797 Stage-2 map = 37%, reduce = 0%, Cumulative CPU 977.06 sec
2018-11-26 12:29:00,029 Stage-2 map = 38%, reduce = 0%, Cumulative CPU 1004.17 sec
2018-11-26 12:29:21,270 Stage-2 map = 39%, reduce = 0%, Cumulative CPU 1030.81 sec
2018-11-26 12:29:41,516 Stage-2 map = 40%, reduce = 0%, Cumulative CPU 1057.63 sec
2018-11-26 12:30:03,775 Stage-2 map = 41%, reduce = 0%, Cumulative CPU 1085.44 sec
2018-11-26 12:30:26,013 Stage-2 map = 42%, reduce = 0%, Cumulative CPU 1111.51 sec
2018-11-26 12:30:47,257 Stage-2 map = 43%, reduce = 0%, Cumulative CPU 1138.16 sec
2018-11-26 12:31:07,503 Stage-2 map = 44%, reduce = 0%, Cumulative CPU 1164.71 sec
2018-11-26 12:31:29,771 Stage-2 map = 45%, reduce = 0%, Cumulative CPU 1191.24 sec
2018-11-26 12:31:51,018 Stage-2 map = 46%, reduce = 0%, Cumulative CPU 1218.18 sec
2018-11-26 12:32:12,224 Stage-2 map = 47%, reduce = 0%, Cumulative CPU 1244.5 sec
2018-11-26 12:32:33,473 Stage-2 map = 48%, reduce = 0%, Cumulative CPU 1270.49 sec
2018-11-26 12:32:55,725 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 1297.18 sec
2018-11-26 12:33:16,964 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 1323.66 sec
2018-11-26 12:33:31,133 Stage-2 map = 51%, reduce = 0%, Cumulative CPU 1342.02 sec
2018-11-26 12:33:52,356 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 1367.34 sec
2018-11-26 12:34:13,586 Stage-2 map = 53%, reduce = 0%, Cumulative CPU 1393.78 sec
2018-11-26 12:34:35,830 Stage-2 map = 54%, reduce = 0%, Cumulative CPU 1420.96 sec
2018-11-26 12:34:57,085 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 1447.21 sec
2018-11-26 12:35:19,352 Stage-2 map = 56%, reduce = 0%, Cumulative CPU 1473.63 sec
2018-11-26 12:35:40,590 Stage-2 map = 57%, reduce = 0%, Cumulative CPU 1499.31 sec
2018-11-26 12:36:01,802 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 1524.99 sec
2018-11-26 12:36:22,049 Stage-2 map = 59%, reduce = 0%, Cumulative CPU 1552.09 sec
2018-11-26 12:36:45,355 Stage-2 map = 60%, reduce = 0%, Cumulative CPU 1578.93 sec
2018-11-26 12:37:07,611 Stage-2 map = 61%, reduce = 0%, Cumulative CPU 1605.01 sec
2018-11-26 12:37:30,864 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 1632.47 sec
2018-11-26 12:37:51,125 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 1659.12 sec
2018-11-26 12:38:12,366 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 1686.39 sec
2018-11-26 12:38:33,592 Stage-2 map = 65%, reduce = 0%, Cumulative CPU 1712.53 sec
2018-11-26 12:38:55,827 Stage-2 map = 66%, reduce = 0%, Cumulative CPU 1739.66 sec
2018-11-26 12:39:18,052 Stage-2 map = 67%, reduce = 0%, Cumulative CPU 1766.85 sec
2018-11-26 12:39:38,299 Stage-2 map = 68%, reduce = 0%, Cumulative CPU 1793.84 sec
2018-11-26 12:40:00,574 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 1821.05 sec
2018-11-26 12:40:22,822 Stage-2 map = 70%, reduce = 0%, Cumulative CPU 1848.7 sec
2018-11-26 12:40:45,061 Stage-2 map = 71%, reduce = 0%, Cumulative CPU 1875.08 sec
2018-11-26 12:41:06,307 Stage-2 map = 72%, reduce = 0%, Cumulative CPU 1902.48 sec
2018-11-26 12:41:28,595 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 1929.9 sec
2018-11-26 12:41:50,849 Stage-2 map = 74%, reduce = 0%, Cumulative CPU 1956.76 sec
2018-11-26 12:42:12,097 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 1982.8 sec
2018-11-26 12:42:35,366 Stage-2 map = 76%, reduce = 0%, Cumulative CPU 2010.06 sec
2018-11-26 12:42:56,662 Stage-2 map = 77%, reduce = 0%, Cumulative CPU 2036.3 sec
2018-11-26 12:43:17,918 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 2063.42 sec
2018-11-26 12:43:41,192 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 2090.15 sec
2018-11-26 12:44:02,432 Stage-2 map = 80%, reduce = 0%, Cumulative CPU 2116.22 sec
2018-11-26 12:44:23,694 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 2142.66 sec
2018-11-26 12:44:45,951 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 2169.89 sec
2018-11-26 12:45:07,201 Stage-2 map = 83%, reduce = 0%, Cumulative CPU 2196.78 sec
2018-11-26 12:45:28,431 Stage-2 map = 84%, reduce = 0%, Cumulative CPU 2223.24 sec
2018-11-26 12:45:49,710 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 2249.64 sec
2018-11-26 12:46:12,001 Stage-2 map = 86%, reduce = 0%, Cumulative CPU 2276.44 sec
2018-11-26 12:46:33,214 Stage-2 map = 87%, reduce = 0%, Cumulative CPU 2303.16 sec
2018-11-26 12:46:55,458 Stage-2 map = 88%, reduce = 0%, Cumulative CPU 2330.4 sec
2018-11-26 12:47:15,679 Stage-2 map = 89%, reduce = 0%, Cumulative CPU 2356.28 sec
2018-11-26 12:47:36,950 Stage-2 map = 90%, reduce = 0%, Cumulative CPU 2382.19 sec
2018-11-26 12:47:59,189 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 2408.27 sec
2018-11-26 12:48:20,425 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 2435.25 sec
2018-11-26 12:48:41,648 Stage-2 map = 93%, reduce = 0%, Cumulative CPU 2462.35 sec
2018-11-26 12:49:02,926 Stage-2 map = 94%, reduce = 0%, Cumulative CPU 2488.51 sec
2018-11-26 12:49:24,205 Stage-2 map = 95%, reduce = 0%, Cumulative CPU 2515.59 sec
2018-11-26 12:49:46,479 Stage-2 map = 96%, reduce = 0%, Cumulative CPU 2542.47 sec
2018-11-26 12:50:07,717 Stage-2 map = 97%, reduce = 0%, Cumulative CPU 2569.85 sec
2018-11-26 12:50:28,943 Stage-2 map = 98%, reduce = 0%, Cumulative CPU 2597.53 sec
2018-11-26 12:50:50,212 Stage-2 map = 99%, reduce = 0%, Cumulative CPU 2624.52 sec
2018-11-26 12:51:50,858 Stage-2 map = 99%, reduce = 0%, Cumulative CPU 2695.05 sec
2018-11-26 12:51:58,959 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 2706.63 sec
2018-11-26 12:52:09,078 Stage-2 map = 100%, reduce = 1%, Cumulative CPU 2716.49 sec
2018-11-26 12:52:21,263 Stage-2 map = 100%, reduce = 2%, Cumulative CPU 2731.07 sec
2018-11-26 12:52:34,450 Stage-2 map = 100%, reduce = 3%, Cumulative CPU 2745.87 sec
2018-11-26 12:52:47,608 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 2760.94 sec
2018-11-26 12:53:05,820 Stage-2 map = 100%, reduce = 5%, Cumulative CPU 2780.78 sec
2018-11-26 12:53:19,994 Stage-2 map = 100%, reduce = 6%, Cumulative CPU 2795.88 sec
2018-11-26 12:53:33,144 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 2810.53 sec
2018-11-26 12:53:45,294 Stage-2 map = 100%, reduce = 8%, Cumulative CPU 2825.48 sec
2018-11-26 12:53:58,491 Stage-2 map = 100%, reduce = 9%, Cumulative CPU 2840.49 sec
2018-11-26 12:54:11,663 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 2855.38 sec
2018-11-26 12:54:25,849 Stage-2 map = 100%, reduce = 11%, Cumulative CPU 2870.29 sec
2018-11-26 12:54:39,005 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 2885.45 sec
2018-11-26 12:54:58,252 Stage-2 map = 100%, reduce = 13%, Cumulative CPU 2905.89 sec
2018-11-26 12:55:11,425 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 2920.81 sec
2018-11-26 12:55:25,603 Stage-2 map = 100%, reduce = 15%, Cumulative CPU 2935.76 sec
2018-11-26 12:55:38,760 Stage-2 map = 100%, reduce = 16%, Cumulative CPU 2950.79 sec
2018-11-26 12:55:51,917 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 2965.24 sec
2018-11-26 12:56:05,085 Stage-2 map = 100%, reduce = 18%, Cumulative CPU 2979.97 sec
2018-11-26 12:56:18,238 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 2994.81 sec
2018-11-26 12:56:37,481 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 3015.43 sec
2018-11-26 12:56:50,680 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 3030.45 sec
2018-11-26 12:57:03,894 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 3045.31 sec
2018-11-26 12:57:18,063 Stage-2 map = 100%, reduce = 23%, Cumulative CPU 3060.43 sec
2018-11-26 12:57:32,235 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 3075.61 sec
2018-11-26 12:57:45,393 Stage-2 map = 100%, reduce = 25%, Cumulative CPU 3091.04 sec
2018-11-26 12:57:56,515 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 3105.93 sec
2018-11-26 12:58:09,701 Stage-2 map = 100%, reduce = 27%, Cumulative CPU 3120.82 sec
2018-11-26 12:58:27,989 Stage-2 map = 100%, reduce = 28%, Cumulative CPU 3140.27 sec
2018-11-26 12:58:42,185 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 3155.71 sec
2018-11-26 12:58:57,374 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 3171.12 sec
2018-11-26 12:59:11,549 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 3186.32 sec
2018-11-26 12:59:23,744 Stage-2 map = 100%, reduce = 32%, Cumulative CPU 3200.75 sec
2018-11-26 12:59:37,936 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 3215.67 sec
2018-11-26 12:59:50,111 Stage-2 map = 100%, reduce = 34%, Cumulative CPU 3230.54 sec
2018-11-26 13:00:04,291 Stage-2 map = 100%, reduce = 35%, Cumulative CPU 3245.37 sec
2018-11-26 13:00:22,515 Stage-2 map = 100%, reduce = 36%, Cumulative CPU 3264.91 sec
2018-11-26 13:00:36,687 Stage-2 map = 100%, reduce = 37%, Cumulative CPU 3279.97 sec
2018-11-26 13:00:48,861 Stage-2 map = 100%, reduce = 38%, Cumulative CPU 3295.14 sec
2018-11-26 13:01:03,098 Stage-2 map = 100%, reduce = 39%, Cumulative CPU 3310.54 sec
2018-11-26 13:01:17,297 Stage-2 map = 100%, reduce = 40%, Cumulative CPU 3325.66 sec
2018-11-26 13:01:31,478 Stage-2 map = 100%, reduce = 41%, Cumulative CPU 3340.48 sec
2018-11-26 13:01:44,634 Stage-2 map = 100%, reduce = 42%, Cumulative CPU 3355.6 sec
2018-11-26 13:02:01,836 Stage-2 map = 100%, reduce = 43%, Cumulative CPU 3376.01 sec
2018-11-26 13:02:15,025 Stage-2 map = 100%, reduce = 44%, Cumulative CPU 3390.75 sec
2018-11-26 13:02:28,236 Stage-2 map = 100%, reduce = 45%, Cumulative CPU 3405.81 sec
2018-11-26 13:02:41,405 Stage-2 map = 100%, reduce = 46%, Cumulative CPU 3420.92 sec
2018-11-26 13:02:55,574 Stage-2 map = 100%, reduce = 47%, Cumulative CPU 3435.91 sec
2018-11-26 13:03:09,743 Stage-2 map = 100%, reduce = 48%, Cumulative CPU 3451.17 sec
2018-11-26 13:03:22,928 Stage-2 map = 100%, reduce = 49%, Cumulative CPU 3465.67 sec
2018-11-26 13:03:36,119 Stage-2 map = 100%, reduce = 50%, Cumulative CPU 3480.89 sec
2018-11-26 13:03:53,374 Stage-2 map = 100%, reduce = 51%, Cumulative CPU 3500.73 sec
2018-11-26 13:04:06,545 Stage-2 map = 100%, reduce = 52%, Cumulative CPU 3515.28 sec
2018-11-26 13:04:19,719 Stage-2 map = 100%, reduce = 53%, Cumulative CPU 3530.5 sec
2018-11-26 13:04:33,922 Stage-2 map = 100%, reduce = 54%, Cumulative CPU 3546.1 sec
2018-11-26 13:04:48,118 Stage-2 map = 100%, reduce = 55%, Cumulative CPU 3561.12 sec
2018-11-26 13:05:00,294 Stage-2 map = 100%, reduce = 56%, Cumulative CPU 3575.14 sec
2018-11-26 13:05:13,485 Stage-2 map = 100%, reduce = 57%, Cumulative CPU 3590.34 sec
2018-11-26 13:05:25,628 Stage-2 map = 100%, reduce = 58%, Cumulative CPU 3604.84 sec
2018-11-26 13:05:41,828 Stage-2 map = 100%, reduce = 59%, Cumulative CPU 3624.52 sec
2018-11-26 13:05:55,992 Stage-2 map = 100%, reduce = 60%, Cumulative CPU 3639.75 sec
2018-11-26 13:06:09,149 Stage-2 map = 100%, reduce = 61%, Cumulative CPU 3654.94 sec
2018-11-26 13:06:22,353 Stage-2 map = 100%, reduce = 62%, Cumulative CPU 3670.17 sec
2018-11-26 13:06:35,580 Stage-2 map = 100%, reduce = 63%, Cumulative CPU 3685.74 sec
2018-11-26 13:06:48,729 Stage-2 map = 100%, reduce = 64%, Cumulative CPU 3700.99 sec
2018-11-26 13:07:01,886 Stage-2 map = 100%, reduce = 65%, Cumulative CPU 3715.93 sec
2018-11-26 13:07:20,111 Stage-2 map = 100%, reduce = 66%, Cumulative CPU 3735.63 sec
2018-11-26 13:07:33,272 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 3750.67 sec
2018-11-26 13:07:46,441 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 3765.97 sec
2018-11-26 13:08:00,654 Stage-2 map = 100%, reduce = 69%, Cumulative CPU 3781.49 sec
2018-11-26 13:08:13,820 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 3796.58 sec
2018-11-26 13:08:28,014 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 3811.57 sec
2018-11-26 13:08:42,207 Stage-2 map = 100%, reduce = 72%, Cumulative CPU 3826.53 sec
2018-11-26 13:08:55,369 Stage-2 map = 100%, reduce = 73%, Cumulative CPU 3841.38 sec
2018-11-26 13:09:12,631 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 3861.53 sec
2018-11-26 13:09:24,810 Stage-2 map = 100%, reduce = 75%, Cumulative CPU 3876.42 sec
2018-11-26 13:09:37,990 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 3891.38 sec
2018-11-26 13:09:51,148 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 3906.23 sec
2018-11-26 13:10:05,338 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 3921.49 sec
2018-11-26 13:10:17,504 Stage-2 map = 100%, reduce = 79%, Cumulative CPU 3936.38 sec
2018-11-26 13:10:31,700 Stage-2 map = 100%, reduce = 80%, Cumulative CPU 3951.84 sec
2018-11-26 13:10:46,890 Stage-2 map = 100%, reduce = 81%, Cumulative CPU 3967.35 sec
2018-11-26 13:11:05,149 Stage-2 map = 100%, reduce = 82%, Cumulative CPU 3987.94 sec
2018-11-26 13:11:20,350 Stage-2 map = 100%, reduce = 83%, Cumulative CPU 4002.8 sec
2018-11-26 13:11:33,509 Stage-2 map = 100%, reduce = 84%, Cumulative CPU 4018.17 sec
2018-11-26 13:11:45,711 Stage-2 map = 100%, reduce = 85%, Cumulative CPU 4032.8 sec
2018-11-26 13:11:57,860 Stage-2 map = 100%, reduce = 86%, Cumulative CPU 4047.49 sec
2018-11-26 13:12:11,050 Stage-2 map = 100%, reduce = 87%, Cumulative CPU 4062.52 sec
2018-11-26 13:12:25,243 Stage-2 map = 100%, reduce = 88%, Cumulative CPU 4077.8 sec
2018-11-26 13:12:42,478 Stage-2 map = 100%, reduce = 89%, Cumulative CPU 4097.96 sec
2018-11-26 13:12:55,639 Stage-2 map = 100%, reduce = 90%, Cumulative CPU 4113.47 sec
2018-11-26 13:13:07,851 Stage-2 map = 100%, reduce = 91%, Cumulative CPU 4128.59 sec
2018-11-26 13:13:21,040 Stage-2 map = 100%, reduce = 92%, Cumulative CPU 4143.35 sec
2018-11-26 13:13:33,213 Stage-2 map = 100%, reduce = 93%, Cumulative CPU 4158.28 sec
2018-11-26 13:13:47,416 Stage-2 map = 100%, reduce = 94%, Cumulative CPU 4173.3 sec
2018-11-26 13:14:00,579 Stage-2 map = 100%, reduce = 95%, Cumulative CPU 4188.29 sec
2018-11-26 13:14:13,763 Stage-2 map = 100%, reduce = 96%, Cumulative CPU 4203.52 sec
2018-11-26 13:14:32,012 Stage-2 map = 100%, reduce = 97%, Cumulative CPU 4224.17 sec
2018-11-26 13:14:46,204 Stage-2 map = 100%, reduce = 98%, Cumulative CPU 4238.89 sec
2018-11-26 13:14:59,372 Stage-2 map = 100%, reduce = 99%, Cumulative CPU 4253.5 sec
2018-11-26 13:15:16,608 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 4273.3 sec
MapReduce Total cumulative CPU time: 0 days 1 hours 11 minutes 13 seconds 300 msec
Ended Job = job_1543224159463_0015
Launching Job 3 out of 6
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0016, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0016/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0016
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
2018-11-26 13:15:26,412 Stage-3 map = 0%, reduce = 0%
2018-11-26 13:15:32,502 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 7.23 sec
2018-11-26 13:15:36,564 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 9.26 sec
MapReduce Total cumulative CPU time: 9 seconds 260 msec
Ended Job = job_1543224159463_0016
Launching Job 4 out of 6
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0017, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0017/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0017
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 1
2018-11-26 13:15:46,270 Stage-4 map = 0%, reduce = 0%
2018-11-26 13:15:50,347 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 2.66 sec
2018-11-26 13:15:55,420 Stage-4 map = 100%, reduce = 100%, Cumulative CPU 4.26 sec
MapReduce Total cumulative CPU time: 4 seconds 260 msec
Ended Job = job_1543224159463_0017
MapReduce Jobs Launched:
Stage-Stage-1: Map: 77 Reduce: 80 Cumulative CPU: 1135.31 sec HDFS Read: 20258379800 HDFS Write: 380710965 SUCCESS
Stage-Stage-2: Map: 299 Reduce: 313 Cumulative CPU: 4273.3 sec HDFS Read: 79967861548 HDFS Write: 147110 SUCCESS
Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 9.26 sec HDFS Read: 235722 HDFS Write: 117178 SUCCESS
Stage-Stage-4: Map: 1 Reduce: 1 Cumulative CPU: 4.26 sec HDFS Read: 125757 HDFS Write: 555 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 30 minutes 22 seconds 130 msec
OK
357786404 418745.6262 1995-03-18 0
75938756 392364.3930 1995-02-19 0
597700673 379690.3573 1995-03-08 0
229784007 370268.7439 1995-03-18 0
384349158 363748.1683 1995-03-06 0
430908804 362791.4899 1995-03-15 0
462347972 356307.6317 1995-03-03 0
68469056 345470.7325 1995-02-25 0
209537569 345249.0956 1995-02-13 0
112713351 344661.1525 1995-03-14 0
Time taken: 4556.122 seconds, Fetched: 10 row(s)
Copy
<script type="js">
// Q03
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT l_orderkey,
SUM(l_extendedprice * (1 - l_discount)) AS revenue,
o_orderdate,
o_shippriority
FROM
customer,
orders,
lineitem
WHERE c_mktsegment = 'BUILDING'
AND c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate < MDY(3, 15, 1995)
AND l_shipdate > MDY(3, 15, 1995)
GROUP BY
l_orderkey,
o_orderdate,
o_shippriority
`
,
`
select FIRST 10
l_orderkey,
SUM(revenue) AS revenue,
o_orderdate,
o_shippriority
FROM \${temp}
GROUP BY
l_orderkey,
o_orderdate,
o_shippriority
ORDER BY
revenue DESC,
o_orderdate
`
);
</script>
3.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- SEGMENT is randomly selected within the list of values defined for Segments in Clause 4.2.2.13
- DATE is a randomly selected day within [1995-03-01 .. 1995-03-31].
3.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- SEGMENT = BUILDING
- DATE = 1995-03-15
Sample Output
| l_orderkey | 2456423 |
| revenue | 406181,011100000 |
| o_orderdate | 05-03-1995 |
| p_paro_shipprioritytkey | 0 |
4 Q4 - Order Priority Checking Query
This query determines how well the order priority system is working and gives an assessment of customer satisfaction.
4.1 Business Question
The Order Priority Checking Query counts the number of orders ordered in a given quarter of a given year in which at least one lineitem was received by the customer later than its committed date. The query lists the count of such orders for each order priority sorted in ascending priority order.
4.2 Functional Query Definition
Copy
SELECT o_orderpriority,
COUNT(*) AS order_count
FROM orders
WHERE o_orderdate >= MDY (7, 1, 1993)
AND o_orderdate < MDY (7, 1, 1993) + 3 UNITS MONTH
AND EXISTS (
SELECT *
FROM lineitem
WHERE l_orderkey = o_orderkey
AND l_commitdate < l_receiptdate
)
GROUP BY o_orderpriority
ORDER BY o_orderpriority
Copy
select
o_orderpriority,
count(*) as order_count
from
orders as o
where
o_orderdate >= '1993-07-01'
and o_orderdate < '1993-10-01'
and exists (
select
*
from
lineitem
where
l_orderkey = o.o_orderkey
and l_commitdate < l_receiptdate
)
group by
o_orderpriority
order by
o_orderpriority;
Copy
hive>
>
>
> select
> o_orderpriority,
> count(*) as order_count
> from
> orders as o
> where
> o_orderdate >= '1993-07-01'
> and o_orderdate < '1993-10-01'
> and exists (
> select
> *
> from
> lineitem
> where
> l_orderkey = o.o_orderkey
> and l_commitdate < l_receiptdate
> )
> group by
> o_orderpriority
> order by
> o_orderpriority;
Query ID = hadoop_20181126132310_9d488ec4-7df0-45f8-8450-9fd001d24ed5
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks not specified. Estimated from input data size: 381
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0018, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0018/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0018
Hadoop job information for Stage-1: number of mappers: 364; number of reducers: 381
2018-11-26 13:23:14,488 Stage-1 map = 0%, reduce = 0%
2018-11-26 13:23:27,725 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 15.87 sec
2018-11-26 13:23:52,091 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 48.08 sec
2018-11-26 13:24:16,392 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 80.34 sec
2018-11-26 13:24:33,597 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 106.04 sec
2018-11-26 13:24:57,921 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 139.26 sec
2018-11-26 13:25:22,258 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 170.71 sec
2018-11-26 13:25:40,483 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 195.24 sec
2018-11-26 13:26:03,746 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 228.16 sec
2018-11-26 13:26:22,002 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 252.31 sec
2018-11-26 13:26:46,290 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 284.45 sec
2018-11-26 13:27:10,610 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 318.12 sec
2018-11-26 13:27:28,833 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 343.48 sec
2018-11-26 13:27:53,100 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 377.86 sec
2018-11-26 13:28:16,391 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 411.08 sec
2018-11-26 13:28:34,622 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 435.28 sec
2018-11-26 13:28:58,897 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 468.06 sec
2018-11-26 13:29:22,183 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 500.59 sec
2018-11-26 13:29:40,436 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 525.48 sec
2018-11-26 13:30:07,785 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 563.64 sec
2018-11-26 13:30:32,129 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 594.44 sec
2018-11-26 13:31:04,537 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 636.94 sec
2018-11-26 13:31:35,907 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 678.93 sec
2018-11-26 13:31:59,211 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 709.28 sec
2018-11-26 13:32:31,625 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 750.9 sec
2018-11-26 13:33:04,019 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 792.81 sec
2018-11-26 13:33:27,289 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 824.34 sec
2018-11-26 13:33:59,682 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 866.88 sec
2018-11-26 13:34:32,050 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 909.13 sec
2018-11-26 13:34:55,360 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 940.99 sec
2018-11-26 13:35:27,757 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 982.93 sec
2018-11-26 13:36:00,168 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 1025.17 sec
2018-11-26 13:36:23,463 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 1055.87 sec
2018-11-26 13:36:55,844 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 1097.55 sec
2018-11-26 13:37:20,136 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 1128.98 sec
2018-11-26 13:37:51,517 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 1171.1 sec
2018-11-26 13:38:23,947 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 1214.35 sec
2018-11-26 13:38:48,226 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 1245.25 sec
2018-11-26 13:39:20,591 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 1286.71 sec
2018-11-26 13:39:51,962 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 1328.3 sec
2018-11-26 13:40:16,257 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 1358.71 sec
2018-11-26 13:40:48,641 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 1401.01 sec
2018-11-26 13:41:20,057 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 1442.72 sec
2018-11-26 13:41:44,345 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 1473.31 sec
2018-11-26 13:42:15,707 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 1514.5 sec
2018-11-26 13:42:39,002 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 1544.88 sec
2018-11-26 13:43:11,397 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 1587.33 sec
2018-11-26 13:43:42,761 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 1627.83 sec
2018-11-26 13:44:06,039 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1659.63 sec
2018-11-26 13:44:38,436 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 1700.53 sec
2018-11-26 13:45:10,868 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1742.31 sec
2018-11-26 13:45:34,168 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 1773.11 sec
2018-11-26 13:46:06,549 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 1815.03 sec
2018-11-26 13:46:38,924 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 1855.97 sec
2018-11-26 13:47:02,246 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 1887.22 sec
2018-11-26 13:47:34,664 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 1928.49 sec
2018-11-26 13:48:07,068 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 1970.68 sec
2018-11-26 13:48:30,361 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 2002.52 sec
2018-11-26 13:49:02,767 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 2043.61 sec
2018-11-26 13:49:27,054 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 2075.07 sec
2018-11-26 13:49:58,422 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 2116.64 sec
2018-11-26 13:50:30,846 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 2158.25 sec
2018-11-26 13:50:55,145 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 2190.04 sec
2018-11-26 13:51:26,542 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 2232.27 sec
2018-11-26 13:51:58,907 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 2272.66 sec
2018-11-26 13:52:23,233 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 2303.9 sec
2018-11-26 13:52:55,601 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 2345.94 sec
2018-11-26 13:53:27,016 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 2387.5 sec
2018-11-26 13:53:51,326 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 2417.9 sec
2018-11-26 13:54:23,720 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 2459.15 sec
2018-11-26 13:54:46,024 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 2490.12 sec
2018-11-26 13:55:18,441 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 2531.96 sec
2018-11-26 13:55:50,815 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 2574.92 sec
2018-11-26 13:56:15,078 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 2606.21 sec
2018-11-26 13:56:46,461 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 2648.08 sec
2018-11-26 13:57:17,826 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 2689.24 sec
2018-11-26 13:57:41,108 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 2720.46 sec
2018-11-26 13:58:13,521 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 2762.03 sec
2018-11-26 13:58:45,903 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 2803.03 sec
2018-11-26 13:59:09,202 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 2834.33 sec
2018-11-26 13:59:41,610 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 2875.95 sec
2018-11-26 14:00:14,019 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 2918.2 sec
2018-11-26 14:00:37,292 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 2949.5 sec
2018-11-26 14:01:09,689 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 2989.59 sec
2018-11-26 14:01:33,993 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 3020.75 sec
2018-11-26 14:02:05,389 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 3063.21 sec
2018-11-26 14:02:37,784 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 3105.33 sec
2018-11-26 14:03:02,082 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 3136.71 sec
2018-11-26 14:03:34,506 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 3178.91 sec
2018-11-26 14:04:05,893 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 3220.09 sec
2018-11-26 14:04:30,187 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 3250.45 sec
2018-11-26 14:05:00,534 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 3291.71 sec
2018-11-26 14:05:32,946 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 3333.47 sec
2018-11-26 14:05:57,304 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 3365.22 sec
2018-11-26 14:06:28,696 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 3406.87 sec
2018-11-26 14:06:53,018 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 3437.68 sec
2018-11-26 14:07:25,445 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 3480.07 sec
2018-11-26 14:07:56,820 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 3521.42 sec
2018-11-26 14:08:21,176 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 3552.76 sec
2018-11-26 14:08:53,594 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3594.89 sec
2018-11-26 14:09:27,015 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3639.02 sec
2018-11-26 14:09:33,124 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 3646.68 sec
2018-11-26 14:09:45,287 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 3662.17 sec
2018-11-26 14:09:57,448 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 3677.76 sec
2018-11-26 14:10:09,604 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 3693.26 sec
2018-11-26 14:10:22,779 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 3708.74 sec
2018-11-26 14:10:31,944 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 3720.72 sec
2018-11-26 14:10:44,145 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 3736.13 sec
2018-11-26 14:10:57,355 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 3751.66 sec
2018-11-26 14:11:10,551 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 3767.41 sec
2018-11-26 14:11:22,715 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 3783.14 sec
2018-11-26 14:11:34,875 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 3798.36 sec
2018-11-26 14:11:45,036 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 3810.48 sec
2018-11-26 14:11:57,246 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 3826.43 sec
2018-11-26 14:12:09,416 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 3842.4 sec
2018-11-26 14:12:22,609 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 3857.96 sec
2018-11-26 14:12:34,769 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 3873.76 sec
2018-11-26 14:12:42,865 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 3885.62 sec
2018-11-26 14:12:55,032 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 3901.62 sec
2018-11-26 14:13:08,197 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 3917.57 sec
2018-11-26 14:13:20,384 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 3933.51 sec
2018-11-26 14:13:32,566 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 3949.39 sec
2018-11-26 14:13:41,685 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 3961.17 sec
2018-11-26 14:13:54,866 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 3977.44 sec
2018-11-26 14:14:07,023 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 3993.04 sec
2018-11-26 14:14:19,185 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 4008.97 sec
2018-11-26 14:14:32,393 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 4025.4 sec
2018-11-26 14:14:41,553 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 4037.16 sec
2018-11-26 14:14:53,716 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 4052.36 sec
2018-11-26 14:15:06,914 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 4067.95 sec
2018-11-26 14:15:19,071 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 4083.68 sec
2018-11-26 14:15:30,192 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 4099.76 sec
2018-11-26 14:15:42,374 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 4115.65 sec
2018-11-26 14:15:51,531 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 4126.92 sec
2018-11-26 14:16:03,734 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 4142.59 sec
2018-11-26 14:16:15,909 Stage-1 map = 100%, reduce = 35%, Cumulative CPU 4158.38 sec
2018-11-26 14:16:28,069 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 4174.42 sec
2018-11-26 14:16:39,215 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 4189.85 sec
2018-11-26 14:16:48,326 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 4201.8 sec
2018-11-26 14:17:00,507 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 4217.68 sec
2018-11-26 14:17:12,711 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 4233.38 sec
2018-11-26 14:17:24,900 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 4248.85 sec
2018-11-26 14:17:37,074 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 4264.25 sec
2018-11-26 14:17:46,202 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 4275.53 sec
2018-11-26 14:17:58,355 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 4291.31 sec
2018-11-26 14:18:10,533 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 4306.79 sec
2018-11-26 14:18:22,721 Stage-1 map = 100%, reduce = 46%, Cumulative CPU 4322.51 sec
2018-11-26 14:18:35,956 Stage-1 map = 100%, reduce = 47%, Cumulative CPU 4339.13 sec
2018-11-26 14:18:46,087 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 4351.02 sec
2018-11-26 14:18:58,264 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 4366.98 sec
2018-11-26 14:19:10,424 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 4382.55 sec
2018-11-26 14:19:21,578 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 4398.41 sec
2018-11-26 14:19:33,805 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 4414.34 sec
2018-11-26 14:19:46,013 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 4430.53 sec
2018-11-26 14:19:56,143 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 4442.55 sec
2018-11-26 14:20:08,311 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 4458.38 sec
2018-11-26 14:20:20,474 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 4474.14 sec
2018-11-26 14:20:32,644 Stage-1 map = 100%, reduce = 57%, Cumulative CPU 4489.81 sec
2018-11-26 14:20:44,803 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 4505.77 sec
2018-11-26 14:20:53,941 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 4518.21 sec
2018-11-26 14:21:07,159 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 4534.0 sec
2018-11-26 14:21:19,318 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 4549.46 sec
2018-11-26 14:21:31,478 Stage-1 map = 100%, reduce = 62%, Cumulative CPU 4565.15 sec
2018-11-26 14:21:43,645 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 4580.39 sec
2018-11-26 14:21:52,779 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 4592.06 sec
2018-11-26 14:22:04,986 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 4608.1 sec
2018-11-26 14:22:17,229 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 4623.6 sec
2018-11-26 14:22:29,403 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 4639.62 sec
2018-11-26 14:22:41,580 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 4655.24 sec
2018-11-26 14:22:50,710 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 4666.63 sec
2018-11-26 14:23:02,865 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 4682.46 sec
2018-11-26 14:23:15,074 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 4698.31 sec
2018-11-26 14:23:27,307 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 4714.09 sec
2018-11-26 14:23:39,503 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 4729.53 sec
2018-11-26 14:23:51,661 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 4746.11 sec
2018-11-26 14:24:00,790 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 4758.1 sec
2018-11-26 14:24:13,986 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 4774.23 sec
2018-11-26 14:24:26,201 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 4789.41 sec
2018-11-26 14:24:38,444 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 4804.95 sec
2018-11-26 14:24:51,647 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 4820.92 sec
2018-11-26 14:25:00,773 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 4832.93 sec
2018-11-26 14:25:12,946 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 4848.72 sec
2018-11-26 14:25:26,129 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 4865.01 sec
2018-11-26 14:25:37,302 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 4880.36 sec
2018-11-26 14:25:49,472 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 4896.03 sec
2018-11-26 14:25:58,624 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 4907.7 sec
2018-11-26 14:26:10,785 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 4923.48 sec
2018-11-26 14:26:23,980 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 4939.83 sec
2018-11-26 14:26:36,132 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 4955.35 sec
2018-11-26 14:26:48,318 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 4970.8 sec
2018-11-26 14:26:57,462 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 4983.08 sec
2018-11-26 14:27:09,690 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 4998.66 sec
2018-11-26 14:27:21,865 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 5014.58 sec
2018-11-26 14:27:34,050 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 5030.21 sec
2018-11-26 14:27:46,208 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 5045.83 sec
2018-11-26 14:27:57,352 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 5061.36 sec
2018-11-26 14:28:06,522 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 5073.22 sec
2018-11-26 14:28:18,725 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 5088.82 sec
2018-11-26 14:28:30,922 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 5104.77 sec
2018-11-26 14:28:43,090 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 5120.64 sec
2018-11-26 14:28:58,302 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5140.96 sec
MapReduce Total cumulative CPU time: 0 days 1 hours 25 minutes 40 seconds 960 msec
Ended Job = job_1543224159463_0018
Launching Job 2 out of 3
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0019, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0019/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0019
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2018-11-26 14:29:08,019 Stage-2 map = 0%, reduce = 0%
2018-11-26 14:29:14,164 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 7.86 sec
2018-11-26 14:29:18,217 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 9.62 sec
MapReduce Total cumulative CPU time: 9 seconds 620 msec
Ended Job = job_1543224159463_0019
Launching Job 3 out of 3
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0020, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0020/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0020
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
2018-11-26 14:29:27,902 Stage-3 map = 0%, reduce = 0%
2018-11-26 14:29:31,973 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.66 sec
2018-11-26 14:29:36,052 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 3.27 sec
MapReduce Total cumulative CPU time: 3 seconds 270 msec
Ended Job = job_1543224159463_0020
MapReduce Jobs Launched:
Stage-Stage-1: Map: 364 Reduce: 381 Cumulative CPU: 5140.96 sec HDFS Read: 97381851725 HDFS Write: 105156 SUCCESS
Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 9.62 sec HDFS Read: 209986 HDFS Write: 281 SUCCESS
Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 3.27 sec HDFS Read: 7890 HDFS Write: 267 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 25 minutes 53 seconds 850 msec
OK
1-URGENT 1051801
2-HIGH 1051366
3-MEDIUM 1051587
4-NOT SPECIFIED 1050950
5-LOW 1051725
Time taken: 3986.452 seconds, Fetched: 5 row(s)
Copy
<script type="js">
// Q04
var grid = new Ax.Grid("grid8");
grid.setTimeout(900);
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT o_orderpriority,
COUNT(*) AS order_count
FROM orders
WHERE o_orderdate >= MDY (7, 1, 1993)
AND o_orderdate < MDY (7, 1, 1993) + 3 UNITS MONTH
AND EXISTS (
SELECT *
FROM lineitem
WHERE l_orderkey = o_orderkey
AND l_commitdate < l_receiptdate
)
GROUP BY o_orderpriority
`
,
`
select o_orderpriority, sum(order_count) as order_count
FROM \${temp}
GROUP BY o_orderpriority
ORDER BY o_orderpriority
`
);
</script>
4.3 Substitution Parameters
Values for the following substitution parameter must be generated and used to build the executable query text:
- DATE is the first day of a randomly selected month between the first month of 1993 and the 10th month of 1997
4.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- DATE = 1993-07-01
Sample Output
Copy
o_orderpriority order_count 1-URGENT 1051801 2-HIGH 1051366 3-MEDIUM 1051587 4-NOT SPECIFIED 1050950 5-LOW 1051725 5 row(s) retrieved.
5 Q5 - Local Supplier Volume Query
This query lists the revenue volume done through local suppliers.
5.1 Business Question
The Local Supplier Volume Query lists for each nation in a region the revenue volume that resulted from lineitem transactions in which the customer ordering parts and the supplier filling them were both within that nation. The query is run in order to determine whether to institute local distribution centers in a given region. The query considers only parts ordered in a given year. The query displays the nations and revenue volume in descending order by revenue. Revenue volume for all qualifying lineitems in a particular nation is defined as sum(l_extendedprice * (1 - l_discount)).
5.2 Functional Query Definition
Copy
SELECT n_name, SUM (l_extendedprice * (1 - l_discount)) AS revenue FROM customer, orders, lineitem, supplier, nation, region WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND l_suppkey = s_suppkey AND c_nationkey = s_nationkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'ASIA' AND o_orderdate >= MDY(1,1,1994) AND o_orderdate < MDY(1,1,1994) + 1 UNITS YEAR GROUP BY n_name ORDER BY revenue DESC
Copy
SELECT n_name, SUM (l_extendedprice * (1 - l_discount)) AS revenue FROM customer, orders, lineitem, supplier, nation, region WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND l_suppkey = s_suppkey AND c_nationkey = s_nationkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'ASIA' AND o_orderdate >= '1994-01-01' AND o_orderdate < '1995-01-01' GROUP BY n_name ORDER BY revenue DESC
Copy
hive>
>
>
>
> SELECT n_name,
> SUM (l_extendedprice * (1 - l_discount)) AS revenue
> FROM customer, orders, lineitem, supplier, nation, region
> WHERE c_custkey = o_custkey
> AND l_orderkey = o_orderkey
> AND l_suppkey = s_suppkey
> AND c_nationkey = s_nationkey
> AND s_nationkey = n_nationkey
> AND n_regionkey = r_regionkey
> AND r_name = 'ASIA'
> AND o_orderdate >= '1994-01-01'
> AND o_orderdate < '1995-01-01'
> GROUP BY n_name
> ORDER BY revenue DESC;
No Stats for tpch_100@customer, Columns: c_custkey, c_nationkey
No Stats for tpch_100@orders, Columns: o_orderdate, o_custkey, o_orderkey
No Stats for tpch_100@lineitem, Columns: l_orderkey, l_suppkey, l_extendedprice, l_discount
No Stats for tpch_100@supplier, Columns: s_nationkey, s_suppkey
No Stats for tpch_100@nation, Columns: n_nationkey, n_regionkey, n_name
No Stats for tpch_100@region, Columns: r_regionkey, r_name
Query ID = hadoop_20181126153152_d04fbd8a-5c39-477d-b729-b993a92e3c33
Total jobs = 10
Launching Job 1 out of 10
Number of reduce tasks not specified. Estimated from input data size: 80
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0021, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0021/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0021
Hadoop job information for Stage-1: number of mappers: 77; number of reducers: 80
2018-11-26 15:31:56,524 Stage-1 map = 0%, reduce = 0%
2018-11-26 15:32:04,661 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 8.75 sec
2018-11-26 15:32:11,750 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 17.86 sec
2018-11-26 15:32:18,839 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 26.62 sec
2018-11-26 15:32:25,925 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 35.27 sec
2018-11-26 15:32:32,015 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 44.55 sec
2018-11-26 15:32:38,094 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 53.26 sec
2018-11-26 15:32:45,186 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 61.86 sec
2018-11-26 15:32:51,268 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 70.35 sec
2018-11-26 15:32:58,354 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 79.26 sec
2018-11-26 15:33:05,432 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 88.35 sec
2018-11-26 15:33:12,512 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 97.22 sec
2018-11-26 15:33:19,595 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 106.64 sec
2018-11-26 15:33:26,681 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 115.52 sec
2018-11-26 15:33:33,761 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 124.0 sec
2018-11-26 15:33:39,820 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 133.01 sec
2018-11-26 15:33:45,888 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 142.01 sec
2018-11-26 15:33:52,960 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 150.37 sec
2018-11-26 15:33:59,023 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 158.98 sec
2018-11-26 15:34:05,086 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 167.56 sec
2018-11-26 15:34:11,153 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 176.05 sec
2018-11-26 15:34:17,214 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 184.94 sec
2018-11-26 15:34:24,295 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 194.14 sec
2018-11-26 15:34:30,356 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 203.42 sec
2018-11-26 15:34:37,456 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 212.47 sec
2018-11-26 15:34:43,525 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 221.62 sec
2018-11-26 15:34:50,615 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 230.52 sec
2018-11-26 15:34:57,691 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 238.96 sec
2018-11-26 15:35:04,759 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 248.02 sec
2018-11-26 15:35:11,839 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 256.83 sec
2018-11-26 15:35:17,919 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 265.73 sec
2018-11-26 15:35:25,003 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 274.73 sec
2018-11-26 15:35:32,084 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 284.2 sec
2018-11-26 15:35:39,169 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 292.78 sec
2018-11-26 15:35:46,242 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 301.79 sec
2018-11-26 15:35:53,319 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 311.06 sec
2018-11-26 15:35:59,400 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 319.53 sec
2018-11-26 15:36:05,467 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 328.29 sec
2018-11-26 15:36:12,565 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 337.4 sec
2018-11-26 15:36:18,652 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 346.41 sec
2018-11-26 15:36:25,731 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 354.91 sec
2018-11-26 15:36:31,814 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 363.68 sec
2018-11-26 15:36:37,881 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 372.38 sec
2018-11-26 15:36:44,971 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 381.33 sec
2018-11-26 15:36:52,065 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 390.45 sec
2018-11-26 15:36:59,141 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 399.57 sec
2018-11-26 15:37:06,231 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 408.48 sec
2018-11-26 15:37:13,301 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 417.06 sec
2018-11-26 15:37:19,361 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 425.74 sec
2018-11-26 15:37:26,438 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 434.86 sec
2018-11-26 15:37:33,514 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 443.65 sec
2018-11-26 15:37:40,585 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 452.97 sec
2018-11-26 15:37:46,644 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 461.41 sec
2018-11-26 15:37:53,738 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 470.63 sec
2018-11-26 15:38:00,822 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 479.32 sec
2018-11-26 15:38:07,923 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 488.5 sec
2018-11-26 15:38:14,997 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 497.3 sec
2018-11-26 15:38:21,061 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 506.04 sec
2018-11-26 15:38:27,131 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 515.0 sec
2018-11-26 15:38:34,209 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 523.99 sec
2018-11-26 15:38:40,272 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 532.69 sec
2018-11-26 15:38:47,349 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 541.86 sec
2018-11-26 15:38:54,446 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 550.71 sec
2018-11-26 15:39:00,526 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 558.89 sec
2018-11-26 15:39:07,601 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 568.05 sec
2018-11-26 15:39:13,670 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 576.98 sec
2018-11-26 15:39:19,737 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 585.47 sec
2018-11-26 15:39:27,823 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 594.59 sec
2018-11-26 15:39:34,903 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 603.36 sec
2018-11-26 15:39:41,977 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 612.15 sec
2018-11-26 15:39:49,043 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 621.3 sec
2018-11-26 15:39:57,121 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 629.94 sec
2018-11-26 15:40:04,194 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 638.67 sec
2018-11-26 15:40:11,265 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 647.76 sec
2018-11-26 15:40:18,331 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 656.57 sec
2018-11-26 15:40:24,392 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 664.84 sec
2018-11-26 15:40:29,477 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 670.85 sec
2018-11-26 15:40:33,527 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 676.13 sec
2018-11-26 15:40:37,572 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 680.11 sec
2018-11-26 15:40:41,616 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 684.23 sec
2018-11-26 15:40:45,659 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 688.44 sec
2018-11-26 15:40:49,707 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 692.72 sec
2018-11-26 15:40:53,754 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 696.89 sec
2018-11-26 15:40:57,814 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 700.98 sec
2018-11-26 15:41:01,865 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 705.0 sec
2018-11-26 15:41:05,912 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 709.12 sec
2018-11-26 15:41:08,956 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 709.12 sec
2018-11-26 15:41:13,018 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 717.22 sec
2018-11-26 15:41:17,084 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 721.35 sec
2018-11-26 15:41:21,135 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 725.34 sec
2018-11-26 15:41:25,184 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 729.34 sec
2018-11-26 15:41:29,256 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 733.42 sec
2018-11-26 15:41:33,308 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 737.73 sec
2018-11-26 15:41:37,362 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 742.05 sec
2018-11-26 15:41:41,413 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 746.05 sec
2018-11-26 15:41:45,466 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 750.1 sec
2018-11-26 15:41:49,528 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 754.34 sec
2018-11-26 15:41:53,584 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 758.44 sec
2018-11-26 15:41:57,635 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 762.54 sec
2018-11-26 15:42:01,701 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 766.7 sec
2018-11-26 15:42:05,750 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 770.74 sec
2018-11-26 15:42:09,790 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 774.87 sec
2018-11-26 15:42:13,834 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 779.25 sec
2018-11-26 15:42:17,876 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 783.41 sec
2018-11-26 15:42:21,919 Stage-1 map = 100%, reduce = 34%, Cumulative CPU 787.94 sec
2018-11-26 15:42:25,970 Stage-1 map = 100%, reduce = 35%, Cumulative CPU 792.1 sec
2018-11-26 15:42:30,028 Stage-1 map = 100%, reduce = 36%, Cumulative CPU 796.08 sec
2018-11-26 15:42:34,070 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 800.34 sec
2018-11-26 15:42:37,105 Stage-1 map = 100%, reduce = 39%, Cumulative CPU 804.36 sec
2018-11-26 15:42:41,160 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 808.58 sec
2018-11-26 15:42:46,225 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 812.51 sec
2018-11-26 15:42:49,279 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 816.74 sec
2018-11-26 15:42:53,346 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 821.04 sec
2018-11-26 15:42:57,402 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 824.97 sec
2018-11-26 15:43:01,461 Stage-1 map = 100%, reduce = 46%, Cumulative CPU 829.34 sec
2018-11-26 15:43:05,513 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 833.35 sec
2018-11-26 15:43:09,561 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 837.28 sec
2018-11-26 15:43:13,614 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 841.38 sec
2018-11-26 15:43:17,668 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 845.48 sec
2018-11-26 15:43:21,742 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 849.57 sec
2018-11-26 15:43:25,789 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 853.83 sec
2018-11-26 15:43:29,836 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 857.79 sec
2018-11-26 15:43:33,885 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 861.94 sec
2018-11-26 15:43:37,931 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 866.15 sec
2018-11-26 15:43:41,977 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 870.53 sec
2018-11-26 15:43:46,020 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 874.65 sec
2018-11-26 15:43:50,065 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 878.63 sec
2018-11-26 15:43:54,109 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 882.73 sec
2018-11-26 15:43:58,151 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 886.98 sec
2018-11-26 15:44:01,181 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 890.96 sec
2018-11-26 15:44:05,243 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 894.94 sec
2018-11-26 15:44:09,309 Stage-1 map = 100%, reduce = 68%, Cumulative CPU 899.11 sec
2018-11-26 15:44:13,354 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 903.23 sec
2018-11-26 15:44:17,424 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 907.3 sec
2018-11-26 15:44:21,477 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 911.32 sec
2018-11-26 15:44:25,521 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 915.42 sec
2018-11-26 15:44:29,584 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 919.89 sec
2018-11-26 15:44:33,635 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 923.94 sec
2018-11-26 15:44:37,690 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 928.09 sec
2018-11-26 15:44:41,743 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 932.37 sec
2018-11-26 15:44:45,789 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 936.5 sec
2018-11-26 15:44:49,837 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 940.51 sec
2018-11-26 15:44:53,906 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 944.41 sec
2018-11-26 15:44:57,951 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 948.49 sec
2018-11-26 15:45:01,996 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 952.65 sec
2018-11-26 15:45:06,038 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 956.8 sec
2018-11-26 15:45:10,081 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 960.69 sec
2018-11-26 15:45:14,130 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 964.77 sec
2018-11-26 15:45:18,174 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 968.77 sec
2018-11-26 15:45:22,223 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 972.73 sec
2018-11-26 15:45:26,272 Stage-1 map = 100%, reduce = 91%, Cumulative CPU 976.9 sec
2018-11-26 15:45:30,318 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 981.08 sec
2018-11-26 15:45:33,347 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 985.1 sec
2018-11-26 15:45:37,403 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 989.08 sec
2018-11-26 15:45:41,466 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 993.22 sec
2018-11-26 15:45:45,537 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 997.25 sec
2018-11-26 15:45:49,598 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 1001.35 sec
2018-11-26 15:45:53,673 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1005.52 sec
MapReduce Total cumulative CPU time: 16 minutes 45 seconds 520 msec
Ended Job = job_1543224159463_0021
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2018-11-26 15:45:59 Starting to launch local task to process map join; maximum memory = 7635730432
2018-11-26 15:46:00 Dump the side-table for tag: 1 with group count: 1 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_15-31-52_026_8968393652617690011-1/-local-10018/HashTable-Stage-18/MapJoin-mapfile171--.hashtable
2018-11-26 15:46:00 Uploaded 1 File to: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_15-31-52_026_8968393652617690011-1/-local-10018/HashTable-Stage-18/MapJoin-mapfile171--.hashtable (278 bytes)
2018-11-26 15:46:00 End of local task; Time Taken: 1.043 sec.
Execution completed successfully
MapredLocal task succeeded
Stage-22 is filtered out by condition resolver.
Stage-23 is filtered out by condition resolver.
Stage-2 is selected by condition resolver.
Launching Job 2 out of 10
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0022, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0022/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0022
Hadoop job information for Stage-18: number of mappers: 1; number of reducers: 0
2018-11-26 15:46:05,375 Stage-18 map = 0%, reduce = 0%
2018-11-26 15:46:11,485 Stage-18 map = 100%, reduce = 0%, Cumulative CPU 5.94 sec
MapReduce Total cumulative CPU time: 5 seconds 940 msec
Ended Job = job_1543224159463_0022
Launching Job 3 out of 10
Number of reduce tasks not specified. Estimated from input data size: 313
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0023, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0023/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0023
Hadoop job information for Stage-2: number of mappers: 299; number of reducers: 313
2018-11-26 15:46:21,019 Stage-2 map = 0%, reduce = 0%
2018-11-26 15:46:40,360 Stage-2 map = 1%, reduce = 0%, Cumulative CPU 22.43 sec
2018-11-26 15:47:07,716 Stage-2 map = 2%, reduce = 0%, Cumulative CPU 55.29 sec
2018-11-26 15:47:34,021 Stage-2 map = 3%, reduce = 0%, Cumulative CPU 88.59 sec
2018-11-26 15:48:01,390 Stage-2 map = 4%, reduce = 0%, Cumulative CPU 122.15 sec
2018-11-26 15:48:28,717 Stage-2 map = 5%, reduce = 0%, Cumulative CPU 155.74 sec
2018-11-26 15:48:56,015 Stage-2 map = 6%, reduce = 0%, Cumulative CPU 188.8 sec
2018-11-26 15:49:22,299 Stage-2 map = 7%, reduce = 0%, Cumulative CPU 222.58 sec
2018-11-26 15:49:49,606 Stage-2 map = 8%, reduce = 0%, Cumulative CPU 256.05 sec
2018-11-26 15:50:16,930 Stage-2 map = 9%, reduce = 0%, Cumulative CPU 289.9 sec
2018-11-26 15:50:44,206 Stage-2 map = 10%, reduce = 0%, Cumulative CPU 323.53 sec
2018-11-26 15:51:10,528 Stage-2 map = 11%, reduce = 0%, Cumulative CPU 355.57 sec
2018-11-26 15:51:37,849 Stage-2 map = 12%, reduce = 0%, Cumulative CPU 388.44 sec
2018-11-26 15:52:05,163 Stage-2 map = 13%, reduce = 0%, Cumulative CPU 421.39 sec
2018-11-26 15:52:32,464 Stage-2 map = 14%, reduce = 0%, Cumulative CPU 454.96 sec
2018-11-26 15:52:58,757 Stage-2 map = 15%, reduce = 0%, Cumulative CPU 487.77 sec
2018-11-26 15:53:26,084 Stage-2 map = 16%, reduce = 0%, Cumulative CPU 520.87 sec
2018-11-26 15:53:53,365 Stage-2 map = 17%, reduce = 0%, Cumulative CPU 554.57 sec
2018-11-26 15:54:19,653 Stage-2 map = 18%, reduce = 0%, Cumulative CPU 588.66 sec
2018-11-26 15:54:46,964 Stage-2 map = 19%, reduce = 0%, Cumulative CPU 620.76 sec
2018-11-26 15:55:14,268 Stage-2 map = 20%, reduce = 0%, Cumulative CPU 654.98 sec
2018-11-26 15:55:40,547 Stage-2 map = 21%, reduce = 0%, Cumulative CPU 687.79 sec
2018-11-26 15:56:07,857 Stage-2 map = 22%, reduce = 0%, Cumulative CPU 720.64 sec
2018-11-26 15:56:35,165 Stage-2 map = 23%, reduce = 0%, Cumulative CPU 753.56 sec
2018-11-26 15:57:02,494 Stage-2 map = 24%, reduce = 0%, Cumulative CPU 786.99 sec
2018-11-26 15:57:28,783 Stage-2 map = 25%, reduce = 0%, Cumulative CPU 820.56 sec
2018-11-26 15:57:56,112 Stage-2 map = 26%, reduce = 0%, Cumulative CPU 853.54 sec
2018-11-26 15:58:23,440 Stage-2 map = 27%, reduce = 0%, Cumulative CPU 886.99 sec
2018-11-26 15:58:50,744 Stage-2 map = 28%, reduce = 0%, Cumulative CPU 920.02 sec
2018-11-26 15:59:17,067 Stage-2 map = 29%, reduce = 0%, Cumulative CPU 952.59 sec
2018-11-26 15:59:44,387 Stage-2 map = 30%, reduce = 0%, Cumulative CPU 985.31 sec
2018-11-26 16:00:11,676 Stage-2 map = 31%, reduce = 0%, Cumulative CPU 1018.0 sec
2018-11-26 16:00:37,943 Stage-2 map = 32%, reduce = 0%, Cumulative CPU 1050.6 sec
2018-11-26 16:01:05,266 Stage-2 map = 33%, reduce = 0%, Cumulative CPU 1083.29 sec
2018-11-26 16:01:32,574 Stage-2 map = 34%, reduce = 0%, Cumulative CPU 1115.88 sec
2018-11-26 16:01:59,869 Stage-2 map = 35%, reduce = 0%, Cumulative CPU 1150.15 sec
2018-11-26 16:02:26,166 Stage-2 map = 36%, reduce = 0%, Cumulative CPU 1183.35 sec
2018-11-26 16:02:53,490 Stage-2 map = 37%, reduce = 0%, Cumulative CPU 1217.33 sec
2018-11-26 16:03:20,817 Stage-2 map = 38%, reduce = 0%, Cumulative CPU 1250.54 sec
2018-11-26 16:03:48,102 Stage-2 map = 39%, reduce = 0%, Cumulative CPU 1283.66 sec
2018-11-26 16:04:14,417 Stage-2 map = 40%, reduce = 0%, Cumulative CPU 1317.37 sec
2018-11-26 16:04:41,708 Stage-2 map = 41%, reduce = 0%, Cumulative CPU 1350.3 sec
2018-11-26 16:05:09,013 Stage-2 map = 42%, reduce = 0%, Cumulative CPU 1383.33 sec
2018-11-26 16:05:35,299 Stage-2 map = 43%, reduce = 0%, Cumulative CPU 1415.5 sec
2018-11-26 16:06:02,637 Stage-2 map = 44%, reduce = 0%, Cumulative CPU 1448.19 sec
2018-11-26 16:06:29,950 Stage-2 map = 45%, reduce = 0%, Cumulative CPU 1480.24 sec
2018-11-26 16:06:57,264 Stage-2 map = 46%, reduce = 0%, Cumulative CPU 1513.25 sec
2018-11-26 16:07:23,583 Stage-2 map = 47%, reduce = 0%, Cumulative CPU 1546.14 sec
2018-11-26 16:07:50,878 Stage-2 map = 48%, reduce = 0%, Cumulative CPU 1578.81 sec
2018-11-26 16:08:18,205 Stage-2 map = 49%, reduce = 0%, Cumulative CPU 1611.84 sec
2018-11-26 16:08:45,489 Stage-2 map = 50%, reduce = 0%, Cumulative CPU 1645.17 sec
2018-11-26 16:09:02,703 Stage-2 map = 51%, reduce = 0%, Cumulative CPU 1666.85 sec
2018-11-26 16:09:30,014 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 1699.34 sec
2018-11-26 16:09:57,298 Stage-2 map = 53%, reduce = 0%, Cumulative CPU 1732.51 sec
2018-11-26 16:10:24,595 Stage-2 map = 54%, reduce = 0%, Cumulative CPU 1765.47 sec
2018-11-26 16:10:50,873 Stage-2 map = 55%, reduce = 0%, Cumulative CPU 1797.94 sec
2018-11-26 16:11:18,240 Stage-2 map = 56%, reduce = 0%, Cumulative CPU 1830.46 sec
2018-11-26 16:11:45,561 Stage-2 map = 57%, reduce = 0%, Cumulative CPU 1862.96 sec
2018-11-26 16:12:11,842 Stage-2 map = 58%, reduce = 0%, Cumulative CPU 1895.67 sec
2018-11-26 16:12:39,138 Stage-2 map = 59%, reduce = 0%, Cumulative CPU 1928.41 sec
2018-11-26 16:13:06,436 Stage-2 map = 60%, reduce = 0%, Cumulative CPU 1962.18 sec
2018-11-26 16:13:33,728 Stage-2 map = 61%, reduce = 0%, Cumulative CPU 1995.31 sec
2018-11-26 16:14:00,021 Stage-2 map = 62%, reduce = 0%, Cumulative CPU 2028.02 sec
2018-11-26 16:14:27,321 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 2060.29 sec
2018-11-26 16:14:54,635 Stage-2 map = 64%, reduce = 0%, Cumulative CPU 2094.0 sec
2018-11-26 16:15:20,908 Stage-2 map = 65%, reduce = 0%, Cumulative CPU 2126.79 sec
2018-11-26 16:15:48,216 Stage-2 map = 66%, reduce = 0%, Cumulative CPU 2159.31 sec
2018-11-26 16:16:15,538 Stage-2 map = 67%, reduce = 0%, Cumulative CPU 2192.84 sec
2018-11-26 16:16:42,852 Stage-2 map = 68%, reduce = 0%, Cumulative CPU 2226.67 sec
2018-11-26 16:17:09,124 Stage-2 map = 69%, reduce = 0%, Cumulative CPU 2259.37 sec
2018-11-26 16:17:36,447 Stage-2 map = 70%, reduce = 0%, Cumulative CPU 2292.39 sec
2018-11-26 16:18:03,754 Stage-2 map = 71%, reduce = 0%, Cumulative CPU 2324.85 sec
2018-11-26 16:18:31,050 Stage-2 map = 72%, reduce = 0%, Cumulative CPU 2357.68 sec
2018-11-26 16:18:57,353 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 2390.69 sec
2018-11-26 16:19:24,709 Stage-2 map = 74%, reduce = 0%, Cumulative CPU 2423.83 sec
2018-11-26 16:19:52,010 Stage-2 map = 75%, reduce = 0%, Cumulative CPU 2457.25 sec
2018-11-26 16:20:18,293 Stage-2 map = 76%, reduce = 0%, Cumulative CPU 2490.52 sec
2018-11-26 16:20:45,611 Stage-2 map = 77%, reduce = 0%, Cumulative CPU 2523.21 sec
2018-11-26 16:21:12,910 Stage-2 map = 78%, reduce = 0%, Cumulative CPU 2555.81 sec
2018-11-26 16:21:40,221 Stage-2 map = 79%, reduce = 0%, Cumulative CPU 2588.26 sec
2018-11-26 16:22:06,546 Stage-2 map = 80%, reduce = 0%, Cumulative CPU 2621.75 sec
2018-11-26 16:22:33,828 Stage-2 map = 81%, reduce = 0%, Cumulative CPU 2654.17 sec
2018-11-26 16:23:01,139 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 2687.34 sec
2018-11-26 16:23:28,448 Stage-2 map = 83%, reduce = 0%, Cumulative CPU 2721.09 sec
2018-11-26 16:23:54,797 Stage-2 map = 84%, reduce = 0%, Cumulative CPU 2753.36 sec
2018-11-26 16:24:22,121 Stage-2 map = 85%, reduce = 0%, Cumulative CPU 2786.52 sec
2018-11-26 16:24:49,444 Stage-2 map = 86%, reduce = 0%, Cumulative CPU 2819.94 sec
2018-11-26 16:25:15,710 Stage-2 map = 87%, reduce = 0%, Cumulative CPU 2853.89 sec
2018-11-26 16:25:43,053 Stage-2 map = 88%, reduce = 0%, Cumulative CPU 2885.99 sec
2018-11-26 16:26:10,362 Stage-2 map = 89%, reduce = 0%, Cumulative CPU 2918.86 sec
2018-11-26 16:26:36,649 Stage-2 map = 90%, reduce = 0%, Cumulative CPU 2952.0 sec
2018-11-26 16:27:03,994 Stage-2 map = 91%, reduce = 0%, Cumulative CPU 2984.68 sec
2018-11-26 16:27:31,308 Stage-2 map = 92%, reduce = 0%, Cumulative CPU 3017.31 sec
2018-11-26 16:27:58,615 Stage-2 map = 93%, reduce = 0%, Cumulative CPU 3049.37 sec
2018-11-26 16:28:24,903 Stage-2 map = 94%, reduce = 0%, Cumulative CPU 3082.49 sec
2018-11-26 16:28:52,233 Stage-2 map = 95%, reduce = 0%, Cumulative CPU 3114.93 sec
2018-11-26 16:29:19,545 Stage-2 map = 96%, reduce = 0%, Cumulative CPU 3147.49 sec
2018-11-26 16:29:46,869 Stage-2 map = 97%, reduce = 0%, Cumulative CPU 3180.84 sec
2018-11-26 16:30:13,167 Stage-2 map = 98%, reduce = 0%, Cumulative CPU 3214.14 sec
2018-11-26 16:30:40,515 Stage-2 map = 99%, reduce = 0%, Cumulative CPU 3247.36 sec
2018-11-26 16:31:41,150 Stage-2 map = 99%, reduce = 0%, Cumulative CPU 3305.35 sec
2018-11-26 16:32:15,533 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 3360.27 sec
2018-11-26 16:32:24,644 Stage-2 map = 100%, reduce = 1%, Cumulative CPU 3371.13 sec
2018-11-26 16:32:39,865 Stage-2 map = 100%, reduce = 2%, Cumulative CPU 3387.19 sec
2018-11-26 16:32:55,060 Stage-2 map = 100%, reduce = 3%, Cumulative CPU 3403.17 sec
2018-11-26 16:33:10,225 Stage-2 map = 100%, reduce = 4%, Cumulative CPU 3419.08 sec
2018-11-26 16:33:29,478 Stage-2 map = 100%, reduce = 5%, Cumulative CPU 3441.01 sec
2018-11-26 16:33:44,694 Stage-2 map = 100%, reduce = 6%, Cumulative CPU 3456.76 sec
2018-11-26 16:33:59,926 Stage-2 map = 100%, reduce = 7%, Cumulative CPU 3473.02 sec
2018-11-26 16:34:15,121 Stage-2 map = 100%, reduce = 8%, Cumulative CPU 3489.32 sec
2018-11-26 16:34:30,288 Stage-2 map = 100%, reduce = 9%, Cumulative CPU 3505.45 sec
2018-11-26 16:34:45,453 Stage-2 map = 100%, reduce = 10%, Cumulative CPU 3521.81 sec
2018-11-26 16:34:59,648 Stage-2 map = 100%, reduce = 11%, Cumulative CPU 3537.56 sec
2018-11-26 16:35:14,885 Stage-2 map = 100%, reduce = 12%, Cumulative CPU 3553.03 sec
2018-11-26 16:35:35,180 Stage-2 map = 100%, reduce = 13%, Cumulative CPU 3574.85 sec
2018-11-26 16:35:50,347 Stage-2 map = 100%, reduce = 14%, Cumulative CPU 3591.06 sec
2018-11-26 16:36:05,507 Stage-2 map = 100%, reduce = 15%, Cumulative CPU 3607.61 sec
2018-11-26 16:36:20,703 Stage-2 map = 100%, reduce = 16%, Cumulative CPU 3623.77 sec
2018-11-26 16:36:34,900 Stage-2 map = 100%, reduce = 17%, Cumulative CPU 3640.1 sec
2018-11-26 16:36:50,094 Stage-2 map = 100%, reduce = 18%, Cumulative CPU 3656.31 sec
2018-11-26 16:37:05,296 Stage-2 map = 100%, reduce = 19%, Cumulative CPU 3673.16 sec
2018-11-26 16:37:25,549 Stage-2 map = 100%, reduce = 20%, Cumulative CPU 3695.21 sec
2018-11-26 16:37:39,707 Stage-2 map = 100%, reduce = 21%, Cumulative CPU 3711.73 sec
2018-11-26 16:37:54,930 Stage-2 map = 100%, reduce = 22%, Cumulative CPU 3728.21 sec
2018-11-26 16:38:10,116 Stage-2 map = 100%, reduce = 23%, Cumulative CPU 3744.41 sec
2018-11-26 16:38:25,320 Stage-2 map = 100%, reduce = 24%, Cumulative CPU 3760.35 sec
2018-11-26 16:38:40,511 Stage-2 map = 100%, reduce = 25%, Cumulative CPU 3776.5 sec
2018-11-26 16:38:55,673 Stage-2 map = 100%, reduce = 26%, Cumulative CPU 3792.75 sec
2018-11-26 16:39:09,858 Stage-2 map = 100%, reduce = 27%, Cumulative CPU 3808.84 sec
2018-11-26 16:39:30,154 Stage-2 map = 100%, reduce = 28%, Cumulative CPU 3830.62 sec
2018-11-26 16:39:45,365 Stage-2 map = 100%, reduce = 29%, Cumulative CPU 3846.64 sec
2018-11-26 16:40:00,542 Stage-2 map = 100%, reduce = 30%, Cumulative CPU 3863.01 sec
2018-11-26 16:40:15,713 Stage-2 map = 100%, reduce = 31%, Cumulative CPU 3879.15 sec
2018-11-26 16:40:30,883 Stage-2 map = 100%, reduce = 32%, Cumulative CPU 3895.2 sec
2018-11-26 16:40:45,091 Stage-2 map = 100%, reduce = 33%, Cumulative CPU 3911.21 sec
2018-11-26 16:41:00,294 Stage-2 map = 100%, reduce = 34%, Cumulative CPU 3927.19 sec
2018-11-26 16:41:15,507 Stage-2 map = 100%, reduce = 35%, Cumulative CPU 3943.67 sec
2018-11-26 16:41:35,761 Stage-2 map = 100%, reduce = 36%, Cumulative CPU 3965.58 sec
2018-11-26 16:41:50,925 Stage-2 map = 100%, reduce = 37%, Cumulative CPU 3981.79 sec
2018-11-26 16:42:06,117 Stage-2 map = 100%, reduce = 38%, Cumulative CPU 3998.47 sec
2018-11-26 16:42:20,316 Stage-2 map = 100%, reduce = 39%, Cumulative CPU 4014.34 sec
2018-11-26 16:42:35,524 Stage-2 map = 100%, reduce = 40%, Cumulative CPU 4030.45 sec
2018-11-26 16:42:50,715 Stage-2 map = 100%, reduce = 41%, Cumulative CPU 4047.01 sec
2018-11-26 16:43:05,902 Stage-2 map = 100%, reduce = 42%, Cumulative CPU 4063.2 sec
2018-11-26 16:43:25,134 Stage-2 map = 100%, reduce = 43%, Cumulative CPU 4084.85 sec
2018-11-26 16:43:40,355 Stage-2 map = 100%, reduce = 44%, Cumulative CPU 4101.06 sec
2018-11-26 16:43:55,581 Stage-2 map = 100%, reduce = 45%, Cumulative CPU 4117.21 sec
2018-11-26 16:44:10,766 Stage-2 map = 100%, reduce = 46%, Cumulative CPU 4133.86 sec
2018-11-26 16:44:25,948 Stage-2 map = 100%, reduce = 47%, Cumulative CPU 4150.42 sec
2018-11-26 16:44:41,114 Stage-2 map = 100%, reduce = 48%, Cumulative CPU 4166.75 sec
2018-11-26 16:44:55,291 Stage-2 map = 100%, reduce = 49%, Cumulative CPU 4183.34 sec
2018-11-26 16:45:10,512 Stage-2 map = 100%, reduce = 50%, Cumulative CPU 4200.33 sec
2018-11-26 16:45:30,802 Stage-2 map = 100%, reduce = 51%, Cumulative CPU 4221.75 sec
2018-11-26 16:45:45,986 Stage-2 map = 100%, reduce = 52%, Cumulative CPU 4238.74 sec
2018-11-26 16:46:01,148 Stage-2 map = 100%, reduce = 53%, Cumulative CPU 4254.66 sec
2018-11-26 16:46:15,328 Stage-2 map = 100%, reduce = 54%, Cumulative CPU 4271.3 sec
2018-11-26 16:46:30,578 Stage-2 map = 100%, reduce = 55%, Cumulative CPU 4287.67 sec
2018-11-26 16:46:45,776 Stage-2 map = 100%, reduce = 56%, Cumulative CPU 4303.87 sec
2018-11-26 16:47:01,982 Stage-2 map = 100%, reduce = 57%, Cumulative CPU 4320.18 sec
2018-11-26 16:47:17,154 Stage-2 map = 100%, reduce = 58%, Cumulative CPU 4336.72 sec
2018-11-26 16:47:37,365 Stage-2 map = 100%, reduce = 59%, Cumulative CPU 4358.34 sec
2018-11-26 16:47:51,593 Stage-2 map = 100%, reduce = 60%, Cumulative CPU 4374.81 sec
2018-11-26 16:48:06,800 Stage-2 map = 100%, reduce = 61%, Cumulative CPU 4391.33 sec
2018-11-26 16:48:21,995 Stage-2 map = 100%, reduce = 62%, Cumulative CPU 4407.87 sec
2018-11-26 16:48:37,197 Stage-2 map = 100%, reduce = 63%, Cumulative CPU 4424.29 sec
2018-11-26 16:48:52,361 Stage-2 map = 100%, reduce = 64%, Cumulative CPU 4440.23 sec
2018-11-26 16:49:07,549 Stage-2 map = 100%, reduce = 65%, Cumulative CPU 4457.0 sec
2018-11-26 16:49:26,834 Stage-2 map = 100%, reduce = 66%, Cumulative CPU 4479.1 sec
2018-11-26 16:49:42,045 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 4496.0 sec
2018-11-26 16:49:57,235 Stage-2 map = 100%, reduce = 68%, Cumulative CPU 4512.53 sec
2018-11-26 16:50:12,426 Stage-2 map = 100%, reduce = 69%, Cumulative CPU 4528.52 sec
2018-11-26 16:50:27,594 Stage-2 map = 100%, reduce = 70%, Cumulative CPU 4544.91 sec
2018-11-26 16:50:41,766 Stage-2 map = 100%, reduce = 71%, Cumulative CPU 4561.93 sec
2018-11-26 16:50:57,005 Stage-2 map = 100%, reduce = 72%, Cumulative CPU 4578.4 sec
2018-11-26 16:51:12,229 Stage-2 map = 100%, reduce = 73%, Cumulative CPU 4594.69 sec
2018-11-26 16:51:32,506 Stage-2 map = 100%, reduce = 74%, Cumulative CPU 4616.52 sec
2018-11-26 16:51:47,675 Stage-2 map = 100%, reduce = 75%, Cumulative CPU 4632.77 sec
2018-11-26 16:52:01,838 Stage-2 map = 100%, reduce = 76%, Cumulative CPU 4649.17 sec
2018-11-26 16:52:17,059 Stage-2 map = 100%, reduce = 77%, Cumulative CPU 4665.25 sec
2018-11-26 16:52:32,284 Stage-2 map = 100%, reduce = 78%, Cumulative CPU 4681.92 sec
2018-11-26 16:52:47,480 Stage-2 map = 100%, reduce = 79%, Cumulative CPU 4698.31 sec
2018-11-26 16:53:02,647 Stage-2 map = 100%, reduce = 80%, Cumulative CPU 4715.17 sec
2018-11-26 16:53:16,800 Stage-2 map = 100%, reduce = 81%, Cumulative CPU 4731.45 sec
2018-11-26 16:53:37,077 Stage-2 map = 100%, reduce = 82%, Cumulative CPU 4753.35 sec
2018-11-26 16:53:52,295 Stage-2 map = 100%, reduce = 83%, Cumulative CPU 4769.37 sec
2018-11-26 16:54:07,488 Stage-2 map = 100%, reduce = 84%, Cumulative CPU 4785.25 sec
2018-11-26 16:54:22,660 Stage-2 map = 100%, reduce = 85%, Cumulative CPU 4801.67 sec
2018-11-26 16:54:37,817 Stage-2 map = 100%, reduce = 86%, Cumulative CPU 4818.22 sec
2018-11-26 16:54:52,988 Stage-2 map = 100%, reduce = 87%, Cumulative CPU 4834.74 sec
2018-11-26 16:55:07,220 Stage-2 map = 100%, reduce = 88%, Cumulative CPU 4851.12 sec
2018-11-26 16:55:27,510 Stage-2 map = 100%, reduce = 89%, Cumulative CPU 4872.99 sec
2018-11-26 16:55:42,699 Stage-2 map = 100%, reduce = 90%, Cumulative CPU 4889.23 sec
2018-11-26 16:55:57,864 Stage-2 map = 100%, reduce = 91%, Cumulative CPU 4905.61 sec
2018-11-26 16:56:13,036 Stage-2 map = 100%, reduce = 92%, Cumulative CPU 4921.85 sec
2018-11-26 16:56:27,243 Stage-2 map = 100%, reduce = 93%, Cumulative CPU 4938.01 sec
2018-11-26 16:56:42,464 Stage-2 map = 100%, reduce = 94%, Cumulative CPU 4953.84 sec
2018-11-26 16:56:57,669 Stage-2 map = 100%, reduce = 95%, Cumulative CPU 4969.93 sec
2018-11-26 16:57:12,864 Stage-2 map = 100%, reduce = 96%, Cumulative CPU 4985.82 sec
2018-11-26 16:57:32,085 Stage-2 map = 100%, reduce = 97%, Cumulative CPU 5007.45 sec
2018-11-26 16:57:47,300 Stage-2 map = 100%, reduce = 98%, Cumulative CPU 5023.9 sec
2018-11-26 16:58:02,516 Stage-2 map = 100%, reduce = 99%, Cumulative CPU 5040.11 sec
2018-11-26 16:58:22,768 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 5061.68 sec
MapReduce Total cumulative CPU time: 0 days 1 hours 24 minutes 21 seconds 680 msec
Ended Job = job_1543224159463_0023
Stage-20 is filtered out by condition resolver.
Stage-21 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.0.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2018-11-26 16:58:29 Processing rows: 200000 Hashtable size: 199999 Memory usage: 510057072 percentage: 0.0672018-11-26 16:58:30 Dump the side-table for tag: 0 with group count: 289768 into file: file:/tmp/hadoop/java/hadoop/92a8bde2-d9e6-4dce-9d32-40df15e61031/hive_2018-11-26_15-31-52_026_8968393652617690011-1/-local-10012/HashTable-Stage-13/MapJoin-mapfile140--.hashtable
Execution completed successfully
MapredLocal task succeeded
Launching Job 5 out of 10
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1543224159463_0024, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0024/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0024
Hadoop job information for Stage-13: number of mappers: 1; number of reducers: 0
2018-11-26 16:58:35,246 Stage-13 map = 0%, reduce = 0%
2018-11-26 16:58:40,341 Stage-13 map = 100%, reduce = 0%, Cumulative CPU 4.62 sec
MapReduce Total cumulative CPU time: 4 seconds 620 msec
Ended Job = job_1543224159463_0024
Launching Job 6 out of 10
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0025, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0025/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0025
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 1
2018-11-26 16:58:50,115 Stage-4 map = 0%, reduce = 0%
2018-11-26 16:58:54,187 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 1.72 sec
2018-11-26 16:58:58,268 Stage-4 map = 100%, reduce = 100%, Cumulative CPU 3.33 sec
MapReduce Total cumulative CPU time: 3 seconds 330 msec
Ended Job = job_1543224159463_0025
Launching Job 7 out of 10
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0026, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0026/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0026
Hadoop job information for Stage-5: number of mappers: 1; number of reducers: 1
2018-11-26 16:59:07,958 Stage-5 map = 0%, reduce = 0%
2018-11-26 16:59:12,026 Stage-5 map = 100%, reduce = 0%, Cumulative CPU 1.7 sec
2018-11-26 16:59:16,086 Stage-5 map = 100%, reduce = 100%, Cumulative CPU 3.26 sec
MapReduce Total cumulative CPU time: 3 seconds 260 msec
Ended Job = job_1543224159463_0026
MapReduce Jobs Launched:
Stage-Stage-1: Map: 77 Reduce: 80 Cumulative CPU: 1005.52 sec HDFS Read: 20258359557 HDFS Write: 522968193 SUCCESS
Stage-Stage-18: Map: 1 Cumulative CPU: 5.94 sec HDFS Read: 142885794 HDFS Write: 9583392 SUCCESS
Stage-Stage-2: Map: 299 Reduce: 313 Cumulative CPU: 5061.68 sec HDFS Read: 80109493717 HDFS Write: 8755471 SUCCESS
Stage-Stage-13: Map: 1 Cumulative CPU: 4.62 sec HDFS Read: 9592430 HDFS Write: 346 SUCCESS
Stage-Stage-4: Map: 1 Reduce: 1 Cumulative CPU: 3.33 sec HDFS Read: 7244 HDFS Write: 346 SUCCESS
Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 3.26 sec HDFS Read: 8051 HDFS Write: 347 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 41 minutes 24 seconds 350 msec
OK
INDONESIA 18151242.6551
VIETNAM 17014255.3877
INDIA 16487209.5005
CHINA 16412150.3488
JAPAN 16354911.5968
Time taken: 5245.095 seconds, Fetched: 5 row(s)
Copy
<script type="js">
// Q05
var grid = new Ax.Grid("grid8");
grid.setTimeout(900);
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT n_name,
SUM (l_extendedprice * (1 - l_discount)) AS revenue
FROM customer, orders, lineitem, supplier, nation, region
WHERE c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND l_suppkey = s_suppkey
AND c_nationkey = s_nationkey
AND s_nationkey = n_nationkey
AND n_regionkey = r_regionkey
AND r_name = 'ASIA'
AND o_orderdate >= MDY(1,1,1994)
AND o_orderdate < MDY(1,1,1994) + 1 UNITS YEAR
GROUP BY n_name
`
,
`
select n_name, sum(revenue) as revenue
from \${temp}
GROUP BY n_name
ORDER BY revenue DESC
`
);
</script>
5.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- REGION is randomly selected within the list of values defined for R_NAME
- DATE is the first of January of a randomly selected year within [1993 .. 1997]
5.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- REGION = ASIA
- DATE = 1994-01-01
REVIEW DATA
Sample Output
Copy
INDONESIA 55502041.1697000 VIETNAM 55295086.9967000 CHINA 53724494.2566000 INDIA 52035512.0002000 JAPAN 45410175.6954000 5 row(s) retrieved.
6 Q6 - Forecasting Revenue Change Query
This query quantifies the amount of revenue increase that would have resulted from eliminating certain companywide discounts in a given percentage range in a given year. Asking this type of "what if" query can be used to look for ways to increase revenues.
6.1 Business Question
The Forecasting Revenue Change Query considers all the lineitems shipped in a given year with discounts between DISCOUNT-0.01 and DISCOUNT+0.01. The query lists the amount by which the total revenue would have increased if these discounts had been eliminated for lineitems with l_quantity less than quantity. Note that the potential revenue increase is equal to the sum of [l_extendedprice * l_discount] for all lineitems with discounts and quantities in the qualifying range.
6.2 Functional Query Definition
Copy
SELECT
SUM(l_extendedprice * l_discount) AS revenue
FROM
lineitem
WHERE
l_shipdate >= MDY(1,1,1994)
AND l_shipdate < MDY(1,1,1994) + 1 UNITS YEAR
AND l_discount BETWEEN 0.05 and 0.07
AND l_quantity < 24;
Copy
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= '1994-01-01'
and l_shipdate < '1995-01-01'
and l_discount between 0.05 and 0.07
and l_quantity < 24;
Copy
hive>
>
> select
> sum(l_extendedprice * l_discount) as revenue
> from
> lineitem
> where
> l_shipdate >= '1994-01-01'
> and l_shipdate < '1995-01-01'
> and l_discount between 0.06 - 0.01 and 0.06 + 0.01001
> and l_quantity < 25;
Query ID = hadoop_20181126171250_8a3fb74b-bf70-428a-a752-5edfd8531144
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0027, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0027/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0027
Hadoop job information for Stage-1: number of mappers: 297; number of reducers: 1
2018-11-26 17:12:54,592 Stage-1 map = 0%, reduce = 0%
2018-11-26 17:13:05,811 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 14.64 sec
2018-11-26 17:13:22,037 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 35.29 sec
2018-11-26 17:13:37,220 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 56.56 sec
2018-11-26 17:13:52,397 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 76.49 sec
2018-11-26 17:14:07,564 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 97.54 sec
2018-11-26 17:14:23,787 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 119.32 sec
2018-11-26 17:14:39,006 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 140.03 sec
2018-11-26 17:14:54,190 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 160.8 sec
2018-11-26 17:15:09,350 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 181.03 sec
2018-11-26 17:15:24,527 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 201.63 sec
2018-11-26 17:15:40,697 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 223.76 sec
2018-11-26 17:15:55,883 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 245.57 sec
2018-11-26 17:16:13,120 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 267.12 sec
2018-11-26 17:16:29,336 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 287.9 sec
2018-11-26 17:16:44,485 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 309.33 sec
2018-11-26 17:16:59,651 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 330.26 sec
2018-11-26 17:17:14,822 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 351.0 sec
2018-11-26 17:17:23,954 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 364.76 sec
2018-11-26 17:17:39,168 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 385.82 sec
2018-11-26 17:17:54,347 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 406.37 sec
2018-11-26 17:18:09,513 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 427.72 sec
2018-11-26 17:18:24,664 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 448.87 sec
2018-11-26 17:18:39,820 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 469.63 sec
2018-11-26 17:18:54,985 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 490.01 sec
2018-11-26 17:19:10,162 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 511.59 sec
2018-11-26 17:19:25,379 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 532.61 sec
2018-11-26 17:19:40,597 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 552.59 sec
2018-11-26 17:19:55,764 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 573.3 sec
2018-11-26 17:20:10,964 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 594.39 sec
2018-11-26 17:20:27,137 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 614.99 sec
2018-11-26 17:20:41,330 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 635.7 sec
2018-11-26 17:20:57,530 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 656.31 sec
2018-11-26 17:21:13,740 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 677.29 sec
2018-11-26 17:21:29,927 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 698.39 sec
2018-11-26 17:21:45,083 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 719.56 sec
2018-11-26 17:22:00,247 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 740.11 sec
2018-11-26 17:22:14,410 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 761.14 sec
2018-11-26 17:22:29,606 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 781.98 sec
2018-11-26 17:22:45,809 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 802.83 sec
2018-11-26 17:23:01,992 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 824.31 sec
2018-11-26 17:23:19,195 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 845.18 sec
2018-11-26 17:23:34,390 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 865.67 sec
2018-11-26 17:23:49,600 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 886.59 sec
2018-11-26 17:24:04,787 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 906.55 sec
2018-11-26 17:24:19,977 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 927.26 sec
2018-11-26 17:24:36,148 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 949.24 sec
2018-11-26 17:24:51,310 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 970.72 sec
2018-11-26 17:25:07,481 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 992.33 sec
2018-11-26 17:25:22,651 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 1013.5 sec
2018-11-26 17:25:37,845 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1033.5 sec
2018-11-26 17:25:49,005 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 1047.26 sec
2018-11-26 17:26:05,174 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 1069.18 sec
2018-11-26 17:26:22,354 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 1090.11 sec
2018-11-26 17:26:37,519 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 1110.77 sec
2018-11-26 17:26:52,711 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 1131.41 sec
2018-11-26 17:27:08,936 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 1152.4 sec
2018-11-26 17:27:25,133 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 1173.5 sec
2018-11-26 17:27:40,297 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 1194.26 sec
2018-11-26 17:27:55,446 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 1215.83 sec
2018-11-26 17:28:11,631 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 1236.74 sec
2018-11-26 17:28:26,810 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1258.39 sec
2018-11-26 17:28:42,016 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 1278.91 sec
2018-11-26 17:28:58,239 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1300.07 sec
2018-11-26 17:29:13,404 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 1319.98 sec
2018-11-26 17:29:29,597 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 1341.46 sec
2018-11-26 17:29:44,765 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 1362.05 sec
2018-11-26 17:29:59,946 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 1383.68 sec
2018-11-26 17:30:15,163 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 1404.29 sec
2018-11-26 17:30:31,373 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 1424.87 sec
2018-11-26 17:30:46,548 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1445.51 sec
2018-11-26 17:31:02,736 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 1466.64 sec
2018-11-26 17:31:16,884 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 1488.0 sec
2018-11-26 17:31:33,047 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1508.74 sec
2018-11-26 17:31:47,246 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 1529.77 sec
2018-11-26 17:32:02,410 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1551.41 sec
2018-11-26 17:32:17,602 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1572.07 sec
2018-11-26 17:32:33,789 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 1593.56 sec
2018-11-26 17:32:48,947 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1614.02 sec
2018-11-26 17:33:03,134 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 1634.16 sec
2018-11-26 17:33:18,317 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1654.56 sec
2018-11-26 17:33:35,560 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 1676.13 sec
2018-11-26 17:33:50,731 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 1696.84 sec
2018-11-26 17:34:05,897 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 1717.72 sec
2018-11-26 17:34:18,032 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 1732.16 sec
2018-11-26 17:34:32,214 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1752.67 sec
2018-11-26 17:34:47,429 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 1773.51 sec
2018-11-26 17:35:03,613 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 1794.83 sec
2018-11-26 17:35:18,792 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 1816.73 sec
2018-11-26 17:35:34,961 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 1838.03 sec
2018-11-26 17:35:51,140 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 1859.39 sec
2018-11-26 17:36:06,300 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 1880.23 sec
2018-11-26 17:36:21,473 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 1901.52 sec
2018-11-26 17:36:36,680 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 1922.71 sec
2018-11-26 17:36:51,878 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1944.35 sec
2018-11-26 17:37:08,045 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 1966.06 sec
2018-11-26 17:37:24,232 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 1986.99 sec
2018-11-26 17:37:39,449 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 2007.51 sec
2018-11-26 17:37:54,662 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 2028.29 sec
2018-11-26 17:38:10,894 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 2049.86 sec
2018-11-26 17:38:31,124 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2075.59 sec
2018-11-26 17:38:33,149 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2077.84 sec
MapReduce Total cumulative CPU time: 34 minutes 37 seconds 840 msec
Ended Job = job_1543224159463_0027
MapReduce Jobs Launched:
Stage-Stage-1: Map: 297 Reduce: 1 Cumulative CPU: 2077.84 sec HDFS Read: 79585504033 HDFS Write: 102 SUCCESS
Total MapReduce CPU Time Spent: 34 minutes 37 seconds 840 msec
OK
NULL
Time taken: 1544.943 seconds, Fetched: 1 row(s)
hive>
>
>
> select
> sum(l_extendedprice * l_discount) as revenue
> from
> lineitem
> where
> l_shipdate >= '1994-01-01'
> and l_shipdate < '1995-01-01'
> and l_discount between 0.05 and 0.07
> and l_quantity < 24;
Query ID = hadoop_20181126181905_526b521e-0732-4356-852e-5f1ca38bfd70
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1543224159463_0028, Tracking URL = http://nuc10:8088/proxy/application_1543224159463_0028/
Kill Command = /home/hadoop/bin/mapred job -kill job_1543224159463_0028
Hadoop job information for Stage-1: number of mappers: 297; number of reducers: 1
2018-11-26 18:19:10,158 Stage-1 map = 0%, reduce = 0%
2018-11-26 18:19:22,351 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 14.31 sec
2018-11-26 18:19:37,537 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 35.96 sec
2018-11-26 18:19:53,759 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 56.95 sec
2018-11-26 18:20:10,977 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 78.77 sec
2018-11-26 18:20:27,168 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 99.83 sec
2018-11-26 18:20:42,399 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 122.13 sec
2018-11-26 18:20:57,612 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 142.55 sec
2018-11-26 18:21:14,820 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 164.76 sec
2018-11-26 18:21:29,982 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 186.03 sec
2018-11-26 18:21:47,185 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 207.92 sec
2018-11-26 18:22:03,368 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 229.55 sec
2018-11-26 18:22:19,589 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 249.96 sec
2018-11-26 18:22:34,768 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 271.15 sec
2018-11-26 18:22:49,956 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 292.76 sec
2018-11-26 18:23:06,146 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 313.91 sec
2018-11-26 18:23:21,314 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 335.21 sec
2018-11-26 18:23:36,485 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 356.72 sec
2018-11-26 18:23:46,631 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 371.09 sec
2018-11-26 18:24:01,837 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 392.84 sec
2018-11-26 18:24:17,042 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 413.79 sec
2018-11-26 18:24:32,216 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 434.37 sec
2018-11-26 18:24:47,382 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 455.25 sec
2018-11-26 18:25:02,545 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 476.84 sec
2018-11-26 18:25:18,750 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 498.7 sec
2018-11-26 18:25:33,951 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 519.65 sec
2018-11-26 18:25:49,142 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 541.68 sec
2018-11-26 18:26:04,308 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 563.33 sec
2018-11-26 18:26:19,460 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 584.73 sec
2018-11-26 18:26:34,614 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 606.01 sec
2018-11-26 18:26:49,781 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 627.32 sec
2018-11-26 18:27:04,962 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 648.95 sec
2018-11-26 18:27:22,171 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 670.18 sec
2018-11-26 18:27:37,342 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 691.06 sec
2018-11-26 18:27:52,504 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 712.4 sec
2018-11-26 18:28:08,679 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 734.33 sec
2018-11-26 18:28:23,844 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 755.31 sec
2018-11-26 18:28:38,998 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 776.09 sec
2018-11-26 18:28:55,184 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 797.98 sec
2018-11-26 18:29:11,361 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 818.96 sec
2018-11-26 18:29:27,561 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 840.42 sec
2018-11-26 18:29:42,722 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 860.8 sec
2018-11-26 18:29:58,914 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 881.87 sec
2018-11-26 18:30:13,067 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 903.01 sec
2018-11-26 18:30:30,273 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 924.58 sec
2018-11-26 18:30:46,471 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 946.43 sec
2018-11-26 18:31:02,668 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 968.5 sec
2018-11-26 18:31:18,859 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 990.84 sec
2018-11-26 18:31:34,022 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 1011.48 sec
2018-11-26 18:31:48,178 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 1032.12 sec
2018-11-26 18:32:03,383 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1053.77 sec
2018-11-26 18:32:13,519 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 1067.99 sec
2018-11-26 18:32:28,697 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 1089.6 sec
2018-11-26 18:32:44,877 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 1111.17 sec
2018-11-26 18:33:00,040 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 1131.79 sec
2018-11-26 18:33:16,218 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 1153.41 sec
2018-11-26 18:33:31,398 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 1174.6 sec
2018-11-26 18:33:47,655 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 1195.34 sec
2018-11-26 18:34:04,855 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 1217.12 sec
2018-11-26 18:34:21,011 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 1238.9 sec
2018-11-26 18:34:36,176 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 1259.92 sec
2018-11-26 18:34:51,329 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 1280.74 sec
2018-11-26 18:35:06,550 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 1301.75 sec
2018-11-26 18:35:22,789 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 1323.48 sec
2018-11-26 18:35:38,973 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 1344.65 sec
2018-11-26 18:35:54,136 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 1365.08 sec
2018-11-26 18:36:11,331 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 1387.46 sec
2018-11-26 18:36:27,510 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 1409.15 sec
2018-11-26 18:36:42,700 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 1430.28 sec
2018-11-26 18:36:58,958 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 1450.74 sec
2018-11-26 18:37:16,161 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1472.73 sec
2018-11-26 18:37:31,321 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 1493.41 sec
2018-11-26 18:37:46,509 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 1515.14 sec
2018-11-26 18:38:02,683 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1536.75 sec
2018-11-26 18:38:17,879 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 1558.7 sec
2018-11-26 18:38:33,059 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1579.64 sec
2018-11-26 18:38:48,233 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1600.82 sec
2018-11-26 18:39:04,432 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 1622.17 sec
2018-11-26 18:39:19,599 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1644.05 sec
2018-11-26 18:39:34,781 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 1664.51 sec
2018-11-26 18:39:50,980 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1686.53 sec
2018-11-26 18:40:07,187 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 1707.53 sec
2018-11-26 18:40:22,345 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 1728.69 sec
2018-11-26 18:40:39,531 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 1750.22 sec
2018-11-26 18:40:50,658 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 1764.25 sec
2018-11-26 18:41:05,836 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1784.74 sec
2018-11-26 18:41:22,043 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 1806.4 sec
2018-11-26 18:41:37,248 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 1827.92 sec
2018-11-26 18:41:53,443 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 1849.67 sec
2018-11-26 18:42:09,611 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 1871.22 sec
2018-11-26 18:42:24,759 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 1892.17 sec
2018-11-26 18:42:39,923 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 1914.25 sec
2018-11-26 18:42:55,082 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 1936.09 sec
2018-11-26 18:43:10,273 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 1957.47 sec
2018-11-26 18:43:26,476 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1979.64 sec
2018-11-26 18:43:41,632 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 2001.0 sec
2018-11-26 18:43:57,819 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 2022.62 sec
2018-11-26 18:44:13,992 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 2044.32 sec
2018-11-26 18:44:28,168 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 2064.96 sec
2018-11-26 18:44:44,385 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 2087.02 sec
2018-11-26 18:45:04,623 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2114.71 sec
2018-11-26 18:45:07,661 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2117.34 sec
MapReduce Total cumulative CPU time: 35 minutes 17 seconds 340 msec
Ended Job = job_1543224159463_0028
MapReduce Jobs Launched:
Stage-Stage-1: Map: 297 Reduce: 1 Cumulative CPU: 2117.34 sec HDFS Read: 79585497479 HDFS Write: 116 SUCCESS
Total MapReduce CPU Time Spent: 35 minutes 17 seconds 340 msec
OK
12330426888.4637
Time taken: 1562.907 seconds, Fetched: 1 row(s)
Copy
<script type="js">
// Q06
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
SUM(l_extendedprice * l_discount) AS revenue
FROM
lineitem
WHERE
l_shipdate >= MDY(1,1,1994)
AND l_shipdate < MDY(1,1,1994) + 1 UNITS YEAR
AND l_discount BETWEEN 0.05 and 0.07
AND l_quantity < 24;
`
,
`
select sum(revenue) as revenue
from \${temp}
`
);
</script>
6.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- DATE is the first of January of a randomly selected year within [1993 .. 1997]
- DISCOUNT is randomly selected within [0.02 .. 0.09]
- QUANTITY is randomly selected within [24 .. 25]
6.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- DATE = 1994-01-01
- DISCOUNT = 0.06
- QUANTITY = 24
Sample Output
| revenue | 12330426888.4637 |
7 Q7 - Volume Shipping Query
This query determines the value of goods shipped between certain nations to help in the re-negotiation of shipping contracts.
7.1 Business Question
The Volume Shipping Query finds, for two given nations, the gross discounted revenues derived from lineitems in which parts were shipped from a supplier in either nation to a customer in the other nation during 1995 and 1996. The query lists the supplier nation, the customer nation, the year, and the revenue from shipments that took place in that year. The query orders the answer by Supplier nation, Customer nation, and year (all ascending).
7.2 Functional Query Definition
Copy
SELECT
supp_nation,
cust_nation,
l_year,
SUM(volume) AS revenue
FROM
(
SELECT
n1.n_name AS supp_nation,
n2.n_name AS cust_nation,
YEAR(l_shipdate) AS l_year,
l_extendedprice * (1 - l_discount) AS volume
FROM
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2
WHERE
s_suppkey = l_suppkey
AND o_orderkey = l_orderkey
AND c_custkey = o_custkey
AND s_nationkey = n1.n_nationkey
AND c_nationkey = n2.n_nationkey
AND (
(n1.n_name = 'FRANCE' AND n2.n_name = 'GERMANY')
OR (n1.n_name = 'GERMANY' AND n2.n_name = 'FRANCE')
)
AND l_shipdate BETWEEN MDY(1,1,1995) AND MDY(12,31,1996)
) AS shipping
GROUP BY
supp_nation,
cust_nation,
l_year
ORDER BY
supp_nation,
cust_nation,
l_year
Copy
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
year(l_shipdate) as l_year,
l_extendedprice * (1 - l_discount) as volume
from
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2
where
s_suppkey = l_suppkey
and o_orderkey = l_orderkey
and c_custkey = o_custkey
and s_nationkey = n1.n_nationkey
and c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY')
or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')
)
and l_shipdate between '1995-01-01' and '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year;
Copy
Copy
<script type="js">
// Q07
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
supp_nation,
cust_nation,
l_year,
SUM(volume) AS revenue
FROM
(
SELECT
n1.n_name AS supp_nation,
n2.n_name AS cust_nation,
YEAR(l_shipdate) AS l_year,
l_extendedprice * (1 - l_discount) AS volume
FROM
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2
WHERE
s_suppkey = l_suppkey
AND o_orderkey = l_orderkey
AND c_custkey = o_custkey
AND s_nationkey = n1.n_nationkey
AND c_nationkey = n2.n_nationkey
AND (
(n1.n_name = 'FRANCE' AND n2.n_name = 'GERMANY')
OR (n1.n_name = 'GERMANY' AND n2.n_name = 'FRANCE')
)
AND l_shipdate BETWEEN MDY(1,1,1995) AND MDY(12,31,1996)
) AS shipping
GROUP BY
supp_nation,
cust_nation,
l_year
`
,
`
SELECT
supp_nation,
cust_nation,
l_year,
SUM(revenue) AS revenue
FROM \${temp}
GROUP BY
supp_nation,
cust_nation,
l_year
ORDER BY
supp_nation,
cust_nation,
l_year
`
);
</script>
7.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- NATION1 is randomly selected within the list of values defined for N_NAME in Clause 4.2.3
- NATION2 is randomly selected within the list of values defined for N_NAME in Clause 4.2.3 and must be different from the value selected for NATION1 in item 1 above.
- QUANTITY is randomly selected within [24 .. 25]
7.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- NATION1 = FRANCE
- NATION2 = GERMANY
Sample Output
| supp_nation | FRANCE |
| cust_nation | GERMANY |
| l_year | 1995 |
| revenue | 54639732,7336000 |
8 Q8 - National Market Share Query
This query determines how the market share of a given nation within a given region has changed over two years for a given part type.
TPC-H Q8 is an eight-way join.
In the diagram, each of the 8 tables in the FROM clause of the query is identified by a large circle with the label of the FROM-clause term: N2, S, L, P, O, C, N1 and R. The arcs in the graph represent the estimated cost of computing each term assuming that the origin of the arc is in an outer loop. For example, the cost of running the S loop as an inner loop to L is 2.30 whereas the cost of running the S loop as an outer loop to L is 9.17.
The "cost" here is logarithmic. With nested loops, the work is multiplied, not added. But it is customary to think of graphs with additive weights and so the graph shows the logarithm of the various costs. The graph shows a cost advantage of S being inside of L of about 6.87, but this translates into the query running about 963 times faster when S loop is inside of the L loop rather than being outside of it.
The arrows from the small circles labeled with "*" indicate the cost of running each loop with no dependencies. The outer loop must use this *-cost. Inner loops have the option of using the *-cost or a cost assuming one of the other terms is in an outer loop, whichever gives the best result. One can think of the *-costs as a short-hand notation indicating multiple arcs, one from each of the other nodes in the graph. The graph is therefore "complete", meaning that there are arcs (some explicit and some implied) in both directions between every pair of nodes in the graph.
The problem of finding the best query plan is equivalent to finding a minimum-cost path through the graph that visits each node exactly once.
8.1 Business Question
The market share for a given nation within a given region is defined as the fraction of the revenue, the sum of [l_extendedprice * (1-l_discount)], from the products of a specified type in that region that was supplied by suppliers from the given nation. The query determines this for the years 1995 and 1996 presented in this order.
8.2 Functional Query Definition
Copy
SELECT
o_year,
SUM(CASE
WHEN nation = 'BRAZIL' THEN volume
ELSE 0
END) / SUM(volume) AS mkt_share
FROM
(
SELECT
YEAR(o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) AS volume,
n2.n_name AS nation
FROM
part,
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2,
region
WHERE
p_partkey = l_partkey
AND s_suppkey = l_suppkey
AND l_orderkey = o_orderkey
AND o_custkey = c_custkey
AND c_nationkey = n1.n_nationkey
AND n1.n_regionkey = r_regionkey
AND r_name = 'AMERICA'
AND s_nationkey = n2.n_nationkey
AND o_orderdate BETWEEN CAST ('1995-01-01' AS DATE) AND CAST ('1996-12-31' AS DATE)
AND p_type = 'ECONOMY ANODIZED STEEL'
) AS all_nations
GROUP BY
o_year
ORDER BY
o_year
Copy
select o_year, sum(case when nation = 'PERU' then volume else 0 end) / sum(volume) as mkt_share from ( select year(o_orderdate) as o_year, l_extendedprice * (1 - l_discount) as volume, n2.n_name as nation from part, supplier, lineitem, orders, customer, nation n1, nation n2, region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'AMERICA' and s_nationkey = n2.n_nationkey and o_orderdate between '1995-01-01' and '1996-12-31' and p_type = 'ECONOMY BURNISHED NICKEL' ) as all_nations group by o_year order by o_year;
Copy
Copy
<script type="js">
// Q08
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
o_year,
SUM(CASE
WHEN nation = 'BRAZIL' THEN volume
ELSE 0
END) AS volume,
SUM(volume) AS all_volume
FROM
(
SELECT
YEAR(o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) AS volume,
n2.n_name AS nation
FROM
part,
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2,
region
WHERE
p_partkey = l_partkey
AND s_suppkey = l_suppkey
AND l_orderkey = o_orderkey
AND o_custkey = c_custkey
AND c_nationkey = n1.n_nationkey
AND n1.n_regionkey = r_regionkey
AND r_name = 'AMERICA'
AND s_nationkey = n2.n_nationkey
AND o_orderdate BETWEEN MDY(1, 1, 1995) AND MDY(12, 31, 1996)
AND p_type = 'ECONOMY ANODIZED STEEL'
) AS all_nations
GROUP BY
o_year
`
,
`
SELECT
o_year,
SUM(volume) / SUM(all_volume) AS mkt_share
FROM \${temp}
GROUP BY
o_year
ORDER BY
o_year
`
);
</script>
8.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- NATION is randomly selected within the list of values defined for N_NAME in Clause 4.2.3
- REGION is the value defined in Clause 4.2.3 for R_NAME where R_REGIONKEY corresponds to N_REGIONKEY for the selected NATION in item 1 above.
- TYPE is randomly selected within the list of 3-syllable strings defined for Types in Clause 4.2.2.13
8.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- NATION = BRAZIL
- REGION = AMERICA
- TYPE = ECONOMY ANODIZED STEEL
Sample Output
| o_year | 1995 |
| mkt_share | 0,03443589040665 |
9 Q9 - Product Type Profit Measure Query
This query determines how much profit is made on a given line of parts, broken out by supplier nation and year.
9.1 Business Question
The Product Type Profit Measure Query finds, for each nation and each year, the profit for all parts ordered in that year that contain a specified substring in their names and that were filled by a supplier in that nation. The profit is defined as the sum of [(l_extendedprice*(1-l_discount)) - (ps_supplycost * l_quantity)] for all lineitems describing parts in the specified line. The query lists the nations in ascending alphabetical order and, for each nation, the year and profit in descending order by year (most recent first).
9.2 Functional Query Definition
Copy
SELECT
nation,
o_year,
SUM(amount) AS sum_profit
FROM
(
SELECT
n_name AS nation,
YEAR(o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount
FROM
part,
supplier,
lineitem,
partsupp,
orders,
nation
WHERE
s_suppkey = l_suppkey
AND ps_suppkey = l_suppkey
AND ps_partkey = l_partkey
AND p_partkey = l_partkey
AND o_orderkey = l_orderkey
AND s_nationkey = n_nationkey
AND p_name LIKE '%green%'
) AS profit
GROUP BY
nation,
o_year
ORDER BY
nation,
o_year DESC
Copy
select
nation,
o_year,
sum(amount) as sum_profit
from
(
select
n_name as nation,
year(o_orderdate) as o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
from
part,
supplier,
lineitem,
partsupp,
orders,
nation
where
s_suppkey = l_suppkey
and ps_suppkey = l_suppkey
and ps_partkey = l_partkey
and p_partkey = l_partkey
and o_orderkey = l_orderkey
and s_nationkey = n_nationkey
and p_name like '%plum%'
) as profit
group by
nation,
o_year
order by
nation,
o_year desc;
Copy
Copy
<script type="js">
// Q09
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
nation,
o_year,
SUM(amount) AS sum_profit
FROM
(
SELECT
n_name AS nation,
YEAR(o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount
FROM
part,
supplier,
lineitem,
partsupp,
orders,
nation
WHERE
s_suppkey = l_suppkey
AND ps_suppkey = l_suppkey
AND ps_partkey = l_partkey
AND p_partkey = l_partkey
AND o_orderkey = l_orderkey
AND s_nationkey = n_nationkey
AND p_name LIKE '%green%'
) AS profit
GROUP BY
nation,
o_year
`
,
`
SELECT
nation,
o_year,
SUM(sum_profit) AS sum_profit
FROM \${temp}
GROUP BY
nation,
o_year
ORDER BY
nation,
o_year DESC
`
);
</script>
9.3 Substitution Parameters
Values for the following substitution parameter must be generated and used to build the executable query text:
- COLOR is randomly selected within the list of values defined for the generation of P_NAME in Clause 4.2.3
9.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- COLOR = green
Sample Output
| nation | ALGERIA |
| o_year | 1998 |
| sum_profit | 27136900,1803000 |
10 Q10 - Returned Item Reporting Query
The query identifies customers who might be having problems with the parts that are shipped to them.
10.1 Business Question
The Returned Item Reporting Query finds the top 20 customers, in terms of their effect on lost revenue for a given quarter, who have returned parts. The query considers only parts that were ordered in the specified quarter. The query lists the customer's name, address, nation, phone number, account balance, comment information and revenue lost. The customers are listed in descending order of lost revenue. Revenue lost is defined as sum(l_extendedprice*(1-l_discount)) for all qualifying lineitems.
10.2 Functional Query Definition
Return the first 20 selected rows
Copy
SELECT FIRST 20
c_custkey,
c_name,
SUM(l_extendedprice * (1 - l_discount)) AS revenue,
c_acctbal,
n_name,
c_address,
c_phone,
c_comment
FROM
customer,
orders,
lineitem,
nation
WHERE
c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate >= MDY (10,1,1993)
AND o_orderdate < MDY(10,1,1993) + 3 UNITS MONTH
AND l_returnflag = 'R'
AND c_nationkey = n_nationkey
GROUP BY
c_custkey,
c_name,
c_acctbal,
c_phone,
n_name,
c_address,
c_comment
ORDER BY
revenue DESC
Copy
select
c_custkey,
c_name,
sum(l_extendedprice * (1 - l_discount)) as revenue,
c_acctbal,
n_name,
c_address,
c_phone,
c_comment
from
customer,
orders,
lineitem,
nation
where
c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate >= '1993-07-01'
and o_orderdate < '1993-10-01'
and l_returnflag = 'R'
and c_nationkey = n_nationkey
group by
c_custkey,
c_name,
c_acctbal,
c_phone,
n_name,
c_address,
c_comment
order by
revenue desc
limit 20;
Copy
Copy
<script type="js">
// Q10
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
c_custkey,
c_name,
SUM(l_extendedprice * (1 - l_discount)) AS revenue,
c_acctbal,
n_name,
c_address,
c_phone,
c_comment
FROM
customer,
orders,
lineitem,
nation
WHERE
c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate >= MDY (10,1,1993)
AND o_orderdate < MDY(10,1,1993) + 3 UNITS MONTH
AND l_returnflag = 'R'
AND c_nationkey = n_nationkey
GROUP BY
c_custkey,
c_name,
c_acctbal,
c_phone,
n_name,
c_address,
c_comment
`
,
`
SELECT FIRST 20
c_custkey,
c_name,
SUM(revenue) AS revenue,
c_acctbal,
n_name,
c_address,
c_phone,
c_comment
FROM \${temp}
GROUP BY
c_custkey,
c_name,
c_acctbal,
c_phone,
n_name,
c_address,
c_comment
ORDER BY
revenue DESC
`
);
</script>
10.3 Substitution Parameters
Values for the following substitution parameter must be generated and used to build the executable query text:
- DATE is the first day of a randomly selected month from the second month of 1993 to the first month of 1995
10.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- DATE = 1993-10-01
Sample Output
| c_custkey | 57040 |
| c_name | Customer#000057040 |
| revenue | 734235,245500000 |
| c_acctbal | 632,87 |
| n_name | JAPAN |
| c_address | Eioyzjf4pp |
| c_phone | 22-895-641-3466 |
| c_comment | sits. slyly regular requests sleep alongside of the regular inst |
11 Q11 - Important Stock Identification Query
This query finds the most important subset of suppliers' stock in a given nation.
11.1 Business Question
The Important Stock Identification Query finds, from scanning the available stock of suppliers in a given nation, all the parts that represent a significant percentage of the total value of all available parts. The query displays the part number and the value of those parts in descending order of value.
11.2 Functional Query Definition
Copy
SELECT
FIRST 10
ps_partkey,
SUM(ps_supplycost * ps_availqty) AS value
FROM
partsupp,
supplier,
nation
WHERE
ps_suppkey = s_suppkey
AND s_nationkey = n_nationkey
AND n_name = 'GERMANY'
GROUP BY
ps_partkey HAVING
SUM(ps_supplycost * ps_availqty) > (
SELECT
SUM(ps_supplycost * ps_availqty) * 0.0001000000e-2
FROM
partsupp,
supplier,
nation
WHERE
ps_suppkey = s_suppkey
AND s_nationkey = n_nationkey
AND n_name = 'GERMANY'
)
ORDER BY
value DESC
Copy
drop view q11_part_tmp_cached;
drop view q11_sum_tmp_cached;
create view q11_part_tmp_cached as
select
ps_partkey,
sum(ps_supplycost * ps_availqty) as part_value
from
partsupp,
supplier,
nation
where
ps_suppkey = s_suppkey
and s_nationkey = n_nationkey
and n_name = 'GERMANY'
group by ps_partkey;
create view q11_sum_tmp_cached as
select
sum(part_value) as total_value
from
q11_part_tmp_cached;
select
ps_partkey, part_value as value
from (
select
ps_partkey,
part_value,
total_value
from
q11_part_tmp_cached join q11_sum_tmp_cached
) a
where
part_value > total_value * 0.0001
order by
value desc;
Copy
$ hive
which: no hbase in (/home/hive/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/informix/bin:/home/jdk1.8.0_161/bin:/home/hadoop/bin:/home/hadoop/sbin:/home/hive/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 1eb53ce3-b9fd-4005-9453-1ea6105fc7f9
Logging initialized using configuration in jar:file:/home/apache-hive-3.1.1-bin/lib/hive-common-3.1.1.jar!/hive-log4j2.properties Async: true
Hive Session ID = 9e98faf3-d7be-42c9-8a79-0bfe8bb70daa
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> use tpch_100;
OK
Time taken: 0.972 seconds
hive> drop view q11_part_tmp_cached;
OK
Time taken: 0.79 seconds
hive> drop view q11_sum_tmp_cached;
OK
Time taken: 0.102 seconds
hive>
> create view q11_part_tmp_cached as
> select
> ps_partkey,
> sum(ps_supplycost * ps_availqty) as part_value
> from
> partsupp,
> supplier,
> nation
> where
> ps_suppkey = s_suppkey
> and s_nationkey = n_nationkey
> and n_name = 'GERMANY'
> group by ps_partkey;
No Stats for tpch_100@partsupp, Columns: ps_suppkey, ps_availqty, ps_partkey, ps_supplycost
No Stats for tpch_100@supplier, Columns: s_nationkey, s_suppkey
No Stats for tpch_100@nation, Columns: n_nationkey, n_name
OK
Time taken: 2.953 seconds
hive>
> create view q11_sum_tmp_cached as
> select
> sum(part_value) as total_value
> from
> q11_part_tmp_cached;
No Stats for tpch_100@partsupp, Columns: ps_suppkey, ps_availqty, ps_partkey, ps_supplycost
No Stats for tpch_100@supplier, Columns: s_nationkey, s_suppkey
No Stats for tpch_100@nation, Columns: n_nationkey, n_name
OK
Time taken: 0.42 seconds
hive>
> select
> ps_partkey, part_value as value
> from (
> select
> ps_partkey,
> part_value,
> total_value
> from
> q11_part_tmp_cached join q11_sum_tmp_cached
> ) a
> where
> part_value > total_value * 0.0001
> order by
> value desc;
No Stats for tpch_100@partsupp, Columns: ps_suppkey, ps_availqty, ps_partkey, ps_supplycost
No Stats for tpch_100@supplier, Columns: s_nationkey, s_suppkey
No Stats for tpch_100@nation, Columns: n_nationkey, n_name
Warning: Map Join MAPJOIN[94][bigTable=?] in task 'Stage-15:MAPRED' is a cross product
Warning: Map Join MAPJOIN[85][bigTable=?] in task 'Stage-14:MAPRED' is a cross product
Warning: Shuffle Join JOIN[49][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-4:MAPRED' is a cross product
Query ID = hadoop_20210321185109_41c8098b-e86e-4425-b4f9-d71670bd20b9
Total jobs = 15
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2021-03-21 18:51:18 Starting to launch local task to process map join; maximum memory = 239075328
2021-03-21 18:51:20 End of local task; Time Taken: 2.242 sec.
Execution completed successfully
MapredLocal task succeeded
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021-03-21 18:51:31 Dump the side-table for tag: 1 with group count: 1 into file: file:/tmp/hadoop/java/hadoop/1eb53ce3-b9fd-4005-9453-1ea6105fc7f9/hive_2021-03-21_18-51-09_513_6936179104068942276-1/-local-10026/HashTable-Stage-24/MapJoin-mapfile71--.hashtable2021-03-21 18:51:31 End of local task; Time Taken: 1.761 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1616348886054_0001, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0001/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0001
Hadoop job information for Stage-20: number of mappers: 2; number of reducers: 0
2021-03-21 18:51:43,810 Stage-20 map = 0%, reduce = 0%
2021-03-21 18:51:54,160 Stage-20 map = 50%, reduce = 0%, Cumulative CPU 8.38 sec
2021-03-21 18:51:56,225 Stage-20 map = 100%, reduce = 0%, Cumulative CPU 15.85 sec
MapReduce Total cumulative CPU time: 15 seconds 850 msec
Ended Job = job_1616348886054_0001
Launching Job 2 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1616348886054_0002, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0002/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0002
Hadoop job information for Stage-24: number of mappers: 2; number of reducers: 0
2021-03-21 18:52:05,173 Stage-24 map = 0%, reduce = 0%
2021-03-21 18:52:10,313 Stage-24 map = 50%, reduce = 0%, Cumulative CPU 3.9 sec
2021-03-21 18:52:15,448 Stage-24 map = 100%, reduce = 0%, Cumulative CPU 12.11 sec
MapReduce Total cumulative CPU time: 12 seconds 110 msec
Ended Job = job_1616348886054_0002
Stage-27 is filtered out by condition resolver.
Stage-28 is selected by condition resolver.
Stage-2 is filtered out by condition resolver.
Stage-30 is filtered out by condition resolver.
Stage-31 is selected by condition resolver.
Stage-9 is filtered out by condition resolver.
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021-03-21 18:52:25 End of local task; Time Taken: 1.499 sec.
Execution completed successfully
MapredLocal task succeeded
SLF4J: Found binding in [jar:file:/home/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2021-03-21 18:52:35 Starting to launch local task to process map join; maximum memory = 239075328
2021-03-21 18:52:37 Uploaded 1 File to: file:/tmp/hadoop/java/hadoop/1eb53ce3-b9fd-4005-9453-1ea6105fc7f9/hive_2021-03-21_18-51-09_513_6936179104068942276-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile60--.hashtable (842653 bytes)
Execution completed successfully
MapredLocal task succeeded
Launching Job 5 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1616348886054_0003, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0003/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0003
Hadoop job information for Stage-18: number of mappers: 46; number of reducers: 0
2021-03-21 18:52:45,050 Stage-18 map = 0%, reduce = 0%
2021-03-21 18:52:54,357 Stage-18 map = 4%, reduce = 0%, Cumulative CPU 17.0 sec
2021-03-21 18:52:55,383 Stage-18 map = 7%, reduce = 0%, Cumulative CPU 26.22 sec
2021-03-21 18:52:57,437 Stage-18 map = 15%, reduce = 0%, Cumulative CPU 64.29 sec
2021-03-21 18:52:59,555 Stage-18 map = 17%, reduce = 0%, Cumulative CPU 74.73 sec
2021-03-21 18:53:00,612 Stage-18 map = 28%, reduce = 0%, Cumulative CPU 134.38 sec
2021-03-21 18:53:01,658 Stage-18 map = 46%, reduce = 0%, Cumulative CPU 225.18 sec
2021-03-21 18:53:02,721 Stage-18 map = 61%, reduce = 0%, Cumulative CPU 304.29 sec
2021-03-21 18:53:03,757 Stage-18 map = 67%, reduce = 0%, Cumulative CPU 330.37 sec
2021-03-21 18:53:04,792 Stage-18 map = 72%, reduce = 0%, Cumulative CPU 358.66 sec
2021-03-21 18:53:05,829 Stage-18 map = 76%, reduce = 0%, Cumulative CPU 386.06 sec
2021-03-21 18:53:08,937 Stage-18 map = 80%, reduce = 0%, Cumulative CPU 402.97 sec
2021-03-21 18:53:09,970 Stage-18 map = 85%, reduce = 0%, Cumulative CPU 424.22 sec
2021-03-21 18:53:11,001 Stage-18 map = 91%, reduce = 0%, Cumulative CPU 458.47 sec
2021-03-21 18:53:12,033 Stage-18 map = 98%, reduce = 0%, Cumulative CPU 492.7 sec
2021-03-21 18:53:13,064 Stage-18 map = 100%, reduce = 0%, Cumulative CPU 505.26 sec
MapReduce Total cumulative CPU time: 8 minutes 25 seconds 260 msec
Ended Job = job_1616348886054_0003
Launching Job 6 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1616348886054_0004, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0004/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0004
Hadoop job information for Stage-22: number of mappers: 46; number of reducers: 0
2021-03-21 18:53:27,716 Stage-22 map = 0%, reduce = 0%
2021-03-21 18:53:40,152 Stage-22 map = 15%, reduce = 0%, Cumulative CPU 65.21 sec
2021-03-21 18:53:41,180 Stage-22 map = 20%, reduce = 0%, Cumulative CPU 86.25 sec
2021-03-21 18:53:42,209 Stage-22 map = 46%, reduce = 0%, Cumulative CPU 222.06 sec
2021-03-21 18:53:43,246 Stage-22 map = 50%, reduce = 0%, Cumulative CPU 247.09 sec
2021-03-21 18:53:44,279 Stage-22 map = 57%, reduce = 0%, Cumulative CPU 284.39 sec
2021-03-21 18:53:45,305 Stage-22 map = 61%, reduce = 0%, Cumulative CPU 308.86 sec
2021-03-21 18:53:46,331 Stage-22 map = 63%, reduce = 0%, Cumulative CPU 322.99 sec
2021-03-21 18:53:47,370 Stage-22 map = 67%, reduce = 0%, Cumulative CPU 346.53 sec
2021-03-21 18:53:48,403 Stage-22 map = 70%, reduce = 0%, Cumulative CPU 355.98 sec
2021-03-21 18:53:49,430 Stage-22 map = 74%, reduce = 0%, Cumulative CPU 374.98 sec
2021-03-21 18:53:50,457 Stage-22 map = 83%, reduce = 0%, Cumulative CPU 414.18 sec
2021-03-21 18:53:52,508 Stage-22 map = 87%, reduce = 0%, Cumulative CPU 436.84 sec
2021-03-21 18:53:53,538 Stage-22 map = 96%, reduce = 0%, Cumulative CPU 481.65 sec
2021-03-21 18:53:54,567 Stage-22 map = 100%, reduce = 0%, Cumulative CPU 503.34 sec
MapReduce Total cumulative CPU time: 8 minutes 23 seconds 340 msec
Ended Job = job_1616348886054_0004
Launching Job 7 out of 15
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1616348886054_0005, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0005/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0005
Hadoop job information for Stage-3: number of mappers: 8; number of reducers: 1
2021-03-21 18:54:06,279 Stage-3 map = 0%, reduce = 0%
2021-03-21 18:54:12,460 Stage-3 map = 13%, reduce = 0%, Cumulative CPU 5.27 sec
2021-03-21 18:54:14,514 Stage-3 map = 63%, reduce = 0%, Cumulative CPU 34.02 sec
2021-03-21 18:54:15,540 Stage-3 map = 75%, reduce = 0%, Cumulative CPU 43.18 sec
2021-03-21 18:54:16,566 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 62.41 sec
2021-03-21 18:54:28,884 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 80.35 sec
MapReduce Total cumulative CPU time: 1 minutes 20 seconds 350 msec
Ended Job = job_1616348886054_0005
Launching Job 8 out of 15
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1616348886054_0006, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0006/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0006
Hadoop job information for Stage-10: number of mappers: 8; number of reducers: 1
2021-03-21 18:54:46,033 Stage-10 map = 0%, reduce = 0%
2021-03-21 18:54:53,263 Stage-10 map = 25%, reduce = 0%, Cumulative CPU 11.4 sec
2021-03-21 18:54:54,291 Stage-10 map = 38%, reduce = 0%, Cumulative CPU 18.16 sec
2021-03-21 18:54:55,319 Stage-10 map = 88%, reduce = 0%, Cumulative CPU 53.86 sec
2021-03-21 18:54:58,421 Stage-10 map = 100%, reduce = 0%, Cumulative CPU 64.27 sec
2021-03-21 18:55:05,606 Stage-10 map = 100%, reduce = 100%, Cumulative CPU 75.98 sec
MapReduce Total cumulative CPU time: 1 minutes 15 seconds 980 msec
Ended Job = job_1616348886054_0006
Launching Job 9 out of 15
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1616348886054_0007, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0007/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0007
Hadoop job information for Stage-11: number of mappers: 1; number of reducers: 1
2021-03-21 18:55:16,029 Stage-11 map = 0%, reduce = 0%
2021-03-21 18:55:23,222 Stage-11 map = 100%, reduce = 0%, Cumulative CPU 2.32 sec
2021-03-21 18:55:29,378 Stage-11 map = 100%, reduce = 100%, Cumulative CPU 4.77 sec
MapReduce Total cumulative CPU time: 4 seconds 770 msec
Ended Job = job_1616348886054_0007
Stage-25 is selected by condition resolver.
Stage-26 is filtered out by condition resolver.
Stage-4 is filtered out by condition resolver.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021-03-21 18:55:39 Uploaded 1 File to: file:/tmp/hadoop/java/hadoop/1eb53ce3-b9fd-4005-9453-1ea6105fc7f9/hive_2021-03-21_18-51-09_513_6936179104068942276-1/-local-10012/HashTable-Stage-14/MapJoin-mapfile01--.hashtable (286 bytes)
2021-03-21 18:55:39 End of local task; Time Taken: 1.201 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 11 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1616348886054_0008, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0008/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0008
Hadoop job information for Stage-14: number of mappers: 1; number of reducers: 0
2021-03-21 18:55:49,310 Stage-14 map = 0%, reduce = 0%
2021-03-21 18:56:05,697 Stage-14 map = 100%, reduce = 0%, Cumulative CPU 18.29 sec
MapReduce Total cumulative CPU time: 18 seconds 290 msec
Ended Job = job_1616348886054_0008
Launching Job 12 out of 15
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1616348886054_0009, Tracking URL = http://nuc10:8088/proxy/application_1616348886054_0009/
Kill Command = /home/hadoop/bin/mapred job -kill job_1616348886054_0009
Hadoop job information for Stage-5: number of mappers: 1; number of reducers: 1
2021-03-21 18:56:18,560 Stage-5 map = 0%, reduce = 0%
2021-03-21 18:56:22,783 Stage-5 map = 100%, reduce = 0%, Cumulative CPU 1.38 sec
2021-03-21 18:56:30,993 Stage-5 map = 100%, reduce = 100%, Cumulative CPU 4.6 sec
MapReduce Total cumulative CPU time: 4 seconds 600 msec
Ended Job = job_1616348886054_0009
MapReduce Jobs Launched:
Stage-Stage-20: Map: 2 Cumulative CPU: 15.85 sec HDFS Read: 142892875 HDFS Write: 838632 SUCCESS
Stage-Stage-24: Map: 2 Cumulative CPU: 12.11 sec HDFS Read: 142892949 HDFS Write: 838632 SUCCESS
Stage-Stage-18: Map: 46 Cumulative CPU: 505.26 sec HDFS Read: 12210156320 HDFS Write: 81519008 SUCCESS
Stage-Stage-22: Map: 46 Cumulative CPU: 503.34 sec HDFS Read: 12210156688 HDFS Write: 81519008 SUCCESS
Stage-Stage-3: Map: 8 Reduce: 1 Cumulative CPU: 80.35 sec HDFS Read: 81560691 HDFS Write: 81515128 SUCCESS
Stage-Stage-10: Map: 8 Reduce: 1 Cumulative CPU: 75.98 sec HDFS Read: 81561014 HDFS Write: 122 SUCCESS
Stage-Stage-11: Map: 1 Reduce: 1 Cumulative CPU: 4.77 sec HDFS Read: 6422 HDFS Write: 122 SUCCESS
Stage-Stage-14: Map: 1 Cumulative CPU: 18.29 sec HDFS Read: 81522390 HDFS Write: 96 SUCCESS
Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 4.6 sec HDFS Read: 7547 HDFS Write: 87 SUCCESS
Total MapReduce CPU Time Spent: 20 minutes 20 seconds 550 msec
OK
Time taken: 322.559 seconds
hive>
Copy
<script type="js">
// Q11
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
ps_partkey,
SUM(ps_supplycost * ps_availqty) AS value
FROM
partsupp,
supplier,
nation
WHERE
ps_suppkey = s_suppkey
AND s_nationkey = n_nationkey
AND n_name = 'GERMANY'
GROUP BY
ps_partkey
HAVING SUM(ps_supplycost * ps_availqty) > (
SELECT
SUM(ps_supplycost * ps_availqty) * 0.0001000000e-2
FROM
partsupp,
supplier,
nation
WHERE
ps_suppkey = s_suppkey
AND s_nationkey = n_nationkey
AND n_name = 'GERMANY'
)
`
,
`
SELECT FIRST 10
ps_partkey,
SUM(value) AS value
FROM \${temp}
GROUP BY
ps_partkey
ORDER BY
value DESC
`
);
</script>
11.3 Substitution Parameters
Values for the following substitution parameter must be generated and used to build the executable query text:
- NATION is randomly selected within the list of values defined for N_NAME in Clause 4.2.3
- FRACTION is chosen as 0.0001 / SF
11.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- NATION = GERMANY
- FRACTION = 0.0001
Sample Output
| ps_partkey | 129760 |
| value | 17538456,86 |
12 Q12 - Shipping Modes and Order Priority Query
This query determines whether selecting less expensive modes of shipping is negatively affecting the critical-priority orders by causing more parts to be received by customers after the committed date.
12.1 Business Question
The Shipping Modes and Order Priority Query counts, by ship mode, for lineitems actually received by customers in a given year, the number of lineitems belonging to orders for which the l_receiptdate exceeds the l_commitdate for two different specified ship modes. Only lineitems that were actually shipped before the l_commitdate are considered. The late lineitems are partitioned into two groups, those with priority URGENT or HIGH, and those with a priority other than URGENT or HIGH.
12.2 Functional Query Definition
Copy
SELECT
l_shipmode,
SUM(CASE
WHEN o_orderpriority = '1-URGENT'
OR o_orderpriority = '2-HIGH'
THEN 1
ELSE 0
END) AS high_line_count,
SUM(CASE
WHEN o_orderpriority <> '1-URGENT'
AND o_orderpriority <> '2-HIGH'
THEN 1
ELSE 0
END) AS low_line_count
FROM
orders,
lineitem
WHERE
o_orderkey = l_orderkey
AND l_shipmode IN ('MAIL', 'SHIP')
AND l_commitdate < l_receiptdate
AND l_shipdate < l_commitdate
AND l_receiptdate >= MDY(1,1, 1994)
AND l_receiptdate < MDY(1,1,1994) + 1 UNITS YEAR
GROUP BY
l_shipmode
ORDER BY
l_shipmode
Copy
select
l_shipmode,
sum(case
when o_orderpriority = '1-URGENT'
or o_orderpriority = '2-HIGH'
then 1
else 0
end) as high_line_count,
sum(case
when o_orderpriority <> '1-URGENT'
and o_orderpriority <> '2-HIGH'
then 1
else 0
end) as low_line_count
from
orders,
lineitem
where
o_orderkey = l_orderkey
and l_shipmode in ('REG AIR', 'MAIL')
and l_commitdate < l_receiptdate
and l_shipdate < l_commitdate
and l_receiptdate >= '1995-01-01'
and l_receiptdate < '1996-01-01'
group by
l_shipmode
order by
l_shipmode;
Copy
Copy
<script type="js">
// Q12
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
l_shipmode,
SUM(CASE
WHEN o_orderpriority = '1-URGENT'
OR o_orderpriority = '2-HIGH'
THEN 1
ELSE 0
END) AS high_line_count,
SUM(CASE
WHEN o_orderpriority <> '1-URGENT'
AND o_orderpriority <> '2-HIGH'
THEN 1
ELSE 0
END) AS low_line_count
FROM
orders,
lineitem
WHERE
o_orderkey = l_orderkey
AND l_shipmode IN ('MAIL', 'SHIP')
AND l_commitdate < l_receiptdate
AND l_shipdate < l_commitdate
AND l_receiptdate >= MDY(1,1, 1994)
AND l_receiptdate < MDY(1,1,1994) + 1 UNITS YEAR
GROUP BY
l_shipmode
`
,
`
SELECT l_shipmode, SUM(high_line_count) AS high_line_count, SUM(low_line_count) AS low_line_count
FROM \${temp}
GROUP BY
l_shipmode
ORDER BY
l_shipmode
`
);
</script>
12.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- SHIPMODE1 is randomly selected within the list of values defined for Modes in Clause 4.2.2.13
- SHIPMODE2 is randomly selected within the list of values defined for Modes in Clause 4.2.2.13 and must be different from the value selected for SHIPMODE1 in item 1
- DATE is the first of January of a randomly selected year within [1993 .. 1997]
12.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- SHIPMODE1 = MAIL
- SHIPMODE2 = SHIP
- DATE = 1994-01-01
Sample Output
| l_shipmode | |
| high_line_count | 6202 |
| low_line_count | 9324 |
13 Q13 - Customer Distribution Query
This query seeks relationships between customers and the size of their orders.
13.1 Business Question
This query determines the distribution of customers by the number of orders they have made, including customers who have no record of orders, past or present. It counts and reports how many customers have no orders, how many have 1, 2, 3, etc. A check is made to ensure that the orders counted do not fall into one of several special categories of orders. Special categories are identified in the order comment column by looking for a particular pattern.
13.2 Functional Query Definition
Copy
SELECT
c_count,
COUNT(*) AS custdist
FROM (
SELECT
c_custkey,
COUNT(o_orderkey) AS c_count
FROM
(SELECT * FROM customer
LEFT OUTER JOIN orders ON
c_custkey = o_custkey AND
o_comment NOT LIKE '%special%requests%') c_customer
GROUP BY
c_custkey
) c_orders
GROUP BY
c_count
ORDER BY
custdist DESC,
c_count DESC
Copy
select
c_count,
count(*) as custdist
from
(
select
c_custkey,
count(o_orderkey) as c_count
from
customer left outer join orders on
c_custkey = o_custkey
and o_comment not like '%special%requests%'
group by
c_custkey
) c_orders
group by
c_count
order by
custdist desc,
c_count desc;
Copy
Copy
<script type="js">
// Q13
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
c_count,
COUNT(*) AS custdist
FROM (
SELECT
c_custkey,
COUNT(o_orderkey) AS c_count
FROM
(SELECT * FROM customer
LEFT OUTER JOIN orders ON
c_custkey = o_custkey AND
o_comment NOT LIKE '%special%requests%') c_customer
GROUP BY
c_custkey
) c_orders
GROUP BY
c_count
`
,
`
SELECT c_count, SUM(custdist) AS custdist
FROM \${temp}
GROUP BY c_count
ORDER BY
custdist DESC,
c_count DESC
`
);
</script>
13.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- WORD1 is randomly selected from 4 possible values: special, pending, unusual, express
- WORD2 is randomly selected from 4 possible values: packages, requests, accounts, deposits
13.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- WORD1 = special
- WORD2 = requests
Sample Output
| c_count | 9 |
| custdist | 6641 |
14 Q14 - Promotion Effect Query
This query monitors the market response to a promotion such as TV advertisements or a special campaign
14.1 Business Question
The Promotion Effect Query determines what percentage of the revenue in a given year and month was derived from promotional parts. The query considers only parts actually shipped in that month and gives the percentage. Revenue is defined as (l_extendedprice * (1-l_discount)).
14.2 Functional Query Definition
Copy
SELECT
100.00 * SUM(CASE
WHEN p_type LIKE 'PROMO%'
THEN l_extendedprice * (1 - l_discount)
ELSE 0
END) / SUM(l_extendedprice * (1 - l_discount)) AS promo_revenue
FROM
lineitem,
part
WHERE
l_partkey = p_partkey
AND l_shipdate >= MDY(9,1,1995)
AND l_shipdate < MDY(9,1,1995) + 1 UNITS MONTH
Copy
select 100.00 * sum(case when p_type like 'PROMO%' then l_extendedprice * (1 - l_discount) else 0 end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue from lineitem, part where l_partkey = p_partkey and l_shipdate >= '1995-08-01' and l_shipdate < '1995-09-01';
Copy
Copy
<script type="js">
// Q14
var grid = new Ax.Grid("grid8");
grid.setTimeout(900);
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
100.00 * SUM(CASE
WHEN p_type LIKE 'PROMO%'
THEN l_extendedprice * (1 - l_discount)
ELSE 0
END) AS promo_amount,
SUM(l_extendedprice * (1 - l_discount)) AS all_amount
FROM
lineitem,
part
WHERE
l_partkey = p_partkey
AND l_shipdate >= MDY(9,1,1995)
AND l_shipdate < MDY(9,1,1995) + 1 UNITS MONTH
`
,
`
SELECT SUM(promo_amount) / SUM(all_amount) AS promo_revenue
FROM \${temp}
`
);
</script>
Copy
SELECT SUM(top) / SUM(bottom)
FROM ((
SELECT
100.00 * SUM(CASE
WHEN p_type LIKE 'PROMO%'
THEN l_extendedprice * (1 - l_discount)
ELSE 0
END),
SUM(l_extendedprice * (1 - l_discount)) AS promo_revenue
FROM
lineitem,
part
GRID ALL 'grid_all'
WHERE
l_partkey = p_partkey
AND l_shipdate >= MDY(9,1,1995)
AND l_shipdate < MDY(9,1,1995) + 1 UNITS MONTH)) AS tab(top, bottom);
14.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- DATE is the first day of a month randomly selected from a random year within [1993 .. 1997]
14.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- DATE = 1995-09-01
Sample Output
| promo_revenue | 16,3807786263955 |
15 Q15
Copy
create view revenue0 (supplier_no, total_revenue) as
select
l_suppkey,
sum(l_extendedprice * (1 - l_discount))
from
lineitem
where
l_shipdate >= mdy (1, 1, 1996 )
and l_shipdate < mdy(4, 1, 1996)
group by l_suppkey;
select
s_suppkey,
s_name,
s_address,
s_phone,
total_revenue
from
supplier,
revenue0
where
s_suppkey = supplier_no
and total_revenue = (
select
max(total_revenue)
from
revenue0
)
order by
s_suppkey;
drop view revenue0;
Copy
drop view revenue_cached; drop view max_revenue_cached; create view revenue_cached as select l_suppkey as supplier_no, sum(l_extendedprice * (1 - l_discount)) as total_revenue from lineitem where l_shipdate >= '1996-01-01' and l_shipdate < '1996-04-01' group by l_suppkey; create view max_revenue_cached as select max(total_revenue) as max_revenue from revenue_cached; select s_suppkey, s_name, s_address, s_phone, total_revenue from supplier, revenue_cached, max_revenue_cached where s_suppkey = supplier_no and total_revenue = max_revenue order by s_suppkey;
16 Q16 - Parts/Supplier Relationship Query
This query finds out how many suppliers can supply parts with given attributes. It might be used, for example, to determine whether there is a sufficient number of suppliers for heavily ordered parts
16.1 Business Question
The Parts/Supplier Relationship Query counts the number of suppliers who can supply parts that satisfy a particular customer's requirements. The customer is interested in parts of eight different sizes as long as they are not of a given type, not of a given brand, and not from a supplier who has had complaints registered at the Better Business Bureau. Results must be presented in descending count and ascending brand, type, and size.
16.2 Functional Query Definition
Copy
SELECT
p_brand,
p_type,
p_size,
COUNT(DISTINCT ps_suppkey) AS supplier_cnt
FROM
partsupp,
part
WHERE
p_partkey = ps_partkey
AND p_brand <> 'Brand#45'
AND p_type NOT LIKE 'MEDIUM POLISHED%'
AND p_size IN (49, 14, 23, 45, 19, 3, 36, 9)
AND ps_suppkey NOT IN (
SELECT
s_suppkey
FROM
supplier
WHERE
s_comment LIKE '%Customer%Complaints%'
)
GROUP BY
p_brand,
p_type,
p_size
ORDER BY
supplier_cnt DESC,
p_brand,
p_type,
p_size
Copy
select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp, part where p_partkey = ps_partkey and p_brand <> 'Brand#34' and p_type not like 'ECONOMY BRUSHED%' and p_size in (22, 14, 27, 49, 21, 33, 35, 28) and partsupp.ps_suppkey not in ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size;
Copy
Copy
<script type="js">
// Q16
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
p_brand,
p_type,
p_size,
COUNT(DISTINCT ps_suppkey) AS supplier_cnt
FROM
partsupp,
part
WHERE
p_partkey = ps_partkey
AND p_brand <> 'Brand#45'
AND p_type NOT LIKE 'MEDIUM POLISHED%'
AND p_size IN (49, 14, 23, 45, 19, 3, 36, 9)
AND ps_suppkey NOT IN (
SELECT s_suppkey
FROM supplier
WHERE s_comment LIKE '%Customer%Complaints%'
)
GROUP BY
p_brand,
p_type,
p_size
`
,
`
SELECT p_brand,
p_type,
p_size,
SUM(supplier_cnt) AS supplier_cnt
FROM \${temp}
GROUP BY
p_brand,
p_type,
p_size
ORDER BY
supplier_cnt DESC,
p_brand,
p_type,
p_size
`
);
</script>
16.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- BRAND = Brand#MN where M and N are two single character strings representing two numbers randomly and independently selected within [1 .. 5];
- TYPE is made of the first 2 syllables of a string randomly selected within the list of 3-syllable strings defined for Types in Clause 4.2.2.13;
- SIZE1 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE2 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE3 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE4 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE5 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE6 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE7 is randomly selected as a set of eight different values within [1 .. 50];
- SIZE8 is randomly selected as a set of eight different values within [1 .. 50].
16.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- BRAND = Brand#45
- TYPE = MEDIUM POLISHED .
- SIZE1 = 49
- SIZE2 = 14
- SIZE3 = 23
- SIZE4 = 45
- SIZE5 = 19
- SIZE6 = 3
- SIZE7 = 36
- SIZE8 = 9.
Sample Output
| p_brand | Brand#41 |
| p_type | MEDIUM BRUSHED TIN |
| p_size | 3 |
| supplier_cnt | 28 |
17 Q17 - Small-Quantity-Order Revenue Query
This query determines how much average yearly revenue would be lost if orders were no longer filled for small quantities of certain parts. This may reduce overhead expenses by concentrating sales on larger shipments.
17.1 Business Question
The Small-Quantity-Order Revenue Query considers parts of a given brand and with a given container type and determines the average lineitem quantity of such parts ordered for all orders (past and pending) in the 7-year database. What would be the average yearly gross (undiscounted) loss in revenue if orders for these parts with a quantity of less than 20% of this average were no longer taken?
17.2 Functional Query Definition
Copy
SELECT
SUM(l_extendedprice) / 7.0 AS avg_yearly
FROM
lineitem,
part
WHERE
p_partkey = l_partkey
AND p_brand = 'Brand#23'
AND p_container = 'MED BOX'
AND l_quantity < (
SELECT
2e-1 * AVG(l_quantity)
FROM
lineitem
WHERE
l_partkey = p_partkey
)
Copy
with q17_part as ( select p_partkey from part where p_brand = 'Brand#23' and p_container = 'MED BOX' ), q17_avg as ( select l_partkey as t_partkey, 0.2 * avg(l_quantity) as t_avg_quantity from lineitem where l_partkey IN (select p_partkey from q17_part) group by l_partkey ), q17_price as ( select l_quantity, l_partkey, l_extendedprice from lineitem where l_partkey IN (select p_partkey from q17_part) ) select cast(sum(l_extendedprice) / 7.0 as decimal(32,2)) as avg_yearly from q17_avg, q17_price where t_partkey = l_partkey and l_quantity < t_avg_quantity;
Copy
Copy
<script type="js">
// Q17
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
SUM(l_extendedprice) / 7.0 AS avg_yearly
FROM
lineitem,
part
WHERE
p_partkey = l_partkey
AND p_brand = 'Brand#23'
AND p_container = 'MED BOX'
AND l_quantity < (
SELECT
2e-1 * AVG(l_quantity)
FROM
lineitem
WHERE
l_partkey = p_partkey
)
`
,
`
SELECT SUM(avg_yearly) AS avg_yearly
FROM \${temp}
`
);
</script>
17.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- BRAND = 'Brand#MN' where MN is a two character string representing two numbers randomly and independently selected within [1 .. 5];
- CONTAINER is randomly selected within the list of 2-syllable strings defined for Containers in Clause 4.2.2.13.
17.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- BRAND = Brand#23;
- CONTAINER = MED BOX.
Sample Output
| avg_yearly | 348406,054285714 |
18 Q18 - Large Volume Customer Query
The Large Volume Customer Query ranks customers based on their having placed a large quantity order. Large quantity orders are defined as those orders whose total quantity is above a certain level.
18.1 Business Question
The Large Volume Customer Query finds a list of the top 100 customers who have ever placed large quantity orders.
The query lists the customer name, customer key, the order key, date and total price and the quantity for the order.
18.2 Functional Query Definition
Copy
select FIRST 100 c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 300 ) and c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate
Copy
drop view q18_tmp_cached; drop table q18_large_volume_customer_cached; create view q18_tmp_cached as select l_orderkey, sum(l_quantity) as t_sum_quantity from lineitem where l_orderkey is not null group by l_orderkey; create table q18_large_volume_customer_cached as select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, q18_tmp_cached t, lineitem l where c_custkey = o_custkey and o_orderkey = t.l_orderkey and o_orderkey is not null and t.t_sum_quantity > 300 and o_orderkey = l.l_orderkey and l.l_orderkey is not null group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 100;
Copy
Copy
<script type="js">
// Q18
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
select
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice,
sum(l_quantity) l_quantity
from
customer,
orders,
lineitem
where
o_orderkey in (
select
l_orderkey
from
lineitem
group by
l_orderkey having
sum(l_quantity) > 300
)
and c_custkey = o_custkey
and o_orderkey = l_orderkey
group by
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice
`
,
`
SELECT FIRST 100
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice,
sum(l_quantity) l_quantity
FROM \${temp}
group by
c_name,
c_custkey,
o_orderkey,
o_orderdate,
o_totalprice
order by
o_totalprice desc,
o_orderdate
`
);
</script>
18.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- QUANTITY is randomly selected within [312..315].
18.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- QUANTITY = 300
Sample Output
| c_name | Customer#000128120 |
| c_custkey | 128120 |
| o_orderkey | 4722021 |
| o_orderdate | 07-04-1994 |
| o_totalprice | 544089,09 |
| (sum) | 323,00 |
19 Q19 - Discounted Revenue Query
The Discounted Revenue Query reports the gross discounted revenue attributed to the sale of selected parts handled in a particular manner. This query is an example of code such as might be produced programmatically by a data mining tool.
19.1 Business Question
The Discounted Revenue query finds the gross discounted revenue for all orders for three different types of parts that were shipped by air and delivered in person. Parts are selected based on the combination of specific brands, a list of containers, and a range of sizes.
19.2 Functional Query Definition
Copy
SELECT
SUM(l_extendedprice* (1 - l_discount)) AS revenue
FROM
lineitem,
part
WHERE
(
p_partkey = l_partkey
AND p_brand = 'Brand#12'
AND p_container IN ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG')
AND l_quantity >= 1 AND l_quantity <= 1 + 10
AND p_size BETWEEN 1 AND 5
AND l_shipmode IN ('AIR', 'AIR REG')
AND l_shipinstruct = 'DELIVER IN PERSON'
)
OR
(
p_partkey = l_partkey
AND p_brand = 'Brand#23'
AND p_container IN ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK')
AND l_quantity >= 10 AND l_quantity <= 10 + 10
AND p_size BETWEEN 1 AND 10
AND l_shipmode IN ('AIR', 'AIR REG')
AND l_shipinstruct = 'DELIVER IN PERSON'
)
OR
(
p_partkey = l_partkey
AND p_brand = 'Brand#34'
AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG')
AND l_quantity >= 20 AND l_quantity <= 20 + 10
AND p_size BETWEEN 1 AND 15
AND l_shipmode IN ('AIR', 'AIR REG')
AND l_shipinstruct = 'DELIVER IN PERSON'
)
Copy
select
sum(l_extendedprice* (1 - l_discount)) as revenue
from
lineitem,
part
where
(
p_partkey = l_partkey
and p_brand = 'Brand#32'
and p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG')
and l_quantity >= 7 and l_quantity <= 7 + 10
and p_size between 1 and 5
and l_shipmode in ('AIR', 'AIR REG')
and l_shipinstruct = 'DELIVER IN PERSON'
)
or
(
p_partkey = l_partkey
and p_brand = 'Brand#35'
and p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK')
and l_quantity >= 15 and l_quantity <= 15 + 10
and p_size between 1 and 10
and l_shipmode in ('AIR', 'AIR REG')
and l_shipinstruct = 'DELIVER IN PERSON'
)
or
(
p_partkey = l_partkey
and p_brand = 'Brand#24'
and p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG')
and l_quantity >= 26 and l_quantity <= 26 + 10
and p_size between 1 and 15
and l_shipmode in ('AIR', 'AIR REG')
and l_shipinstruct = 'DELIVER IN PERSON'
);
Copy
Copy
<script type="js">
// Q19
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
SUM(l_extendedprice* (1 - l_discount)) AS revenue
FROM
lineitem,
part
WHERE
(
p_partkey = l_partkey
AND p_brand = 'Brand#12'
AND p_container IN ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG')
AND l_quantity >= 1 AND l_quantity <= 1 + 10
AND p_size BETWEEN 1 AND 5
AND l_shipmode IN ('AIR', 'AIR REG')
AND l_shipinstruct = 'DELIVER IN PERSON'
)
OR
(
p_partkey = l_partkey
AND p_brand = 'Brand#23'
AND p_container IN ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK')
AND l_quantity >= 10 AND l_quantity <= 10 + 10
AND p_size BETWEEN 1 AND 10
AND l_shipmode IN ('AIR', 'AIR REG')
AND l_shipinstruct = 'DELIVER IN PERSON'
)
OR
(
p_partkey = l_partkey
AND p_brand = 'Brand#34'
AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG')
AND l_quantity >= 20 AND l_quantity <= 20 + 10
AND p_size BETWEEN 1 AND 15
AND l_shipmode IN ('AIR', 'AIR REG')
AND l_shipinstruct = 'DELIVER IN PERSON'
)
`
,
`
SELECT SUM(revenue) AS revenue
FROM \${temp}
`
);
</script>
19.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- QUANTITY1 is randomly selected within [1..10]
- QUANTITY2 is randomly selected within [10..20].
- QUANTITY3 is randomly selected within [20..30
- BRAND1, BRAND2, BRAND3 = 'Brand#MN' where each MN is a two character string representing two numbers randomly and independently selected within [1 .. 5]
19.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- QUANTITY1 = 1
- QUANTITY2 = 10
- QUANTITY3 = 20.
- BRAND1 = Brand#12.
- BRAND2 = Brand#23.
- BRAND3 = Brand#34
Sample Output
| revenue | 3083843,05780000 |
20 Q20 - Potential Part Promotion Query
The Potential Part Promotion Query identifies suppliers in a particular nation having selected parts that may be candidates for a promotional offer.
20.1 Business Question
The Potential Part Promotion query identifies suppliers who have an excess of a given part available; an excess is defined to be more than 50% of the parts like the given part that the supplier shipped in a given year for a given nation. Only parts whose names share a certain naming convention are considered.
20.2 Functional Query Definition
Copy
SELECT
s_name,
s_address
FROM
supplier,
nation
WHERE
s_suppkey IN (
SELECT
ps_suppkey
FROM
partsupp
WHERE
ps_partkey IN (
SELECT
p_partkey
FROM
part
WHERE
p_name LIKE 'forest%'
)
AND ps_availqty > (
SELECT
0.5 * SUM(l_quantity)
FROM
lineitem
WHERE
l_partkey = ps_partkey
AND l_suppkey = ps_suppkey
AND l_shipdate >= MDY(1,1,1994)
AND l_shipdate < MDY(1,1,1994) + 1 UNITS YEAR
)
)
AND s_nationkey = n_nationkey
AND n_name = 'CANADA'
ORDER BY
s_name
Copy
-- explain formatted
with tmp1 as (
select p_partkey from part where p_name like 'forest%'
),
tmp2 as (
select s_name, s_address, s_suppkey
from supplier, nation
where s_nationkey = n_nationkey
and n_name = 'CANADA'
),
tmp3 as (
select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey
from lineitem, tmp2
where l_shipdate >= '1994-01-01' and l_shipdate <= '1995-01-01'
and l_suppkey = s_suppkey
group by l_partkey, l_suppkey
),
tmp4 as (
select ps_partkey, ps_suppkey, ps_availqty
from partsupp
where ps_partkey IN (select p_partkey from tmp1)
),
tmp5 as (
select
ps_suppkey
from
tmp4, tmp3
where
ps_partkey = l_partkey
and ps_suppkey = l_suppkey
and ps_availqty > sum_quantity
)
select
s_name,
s_address
from
supplier
where
s_suppkey IN (select ps_suppkey from tmp5)
order by s_name;
Copy
Copy
<script type="js">
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT
s_name,
s_address
FROM
supplier,
nation
WHERE
s_suppkey IN (
SELECT
ps_suppkey
FROM
partsupp
WHERE
ps_partkey IN (
SELECT
p_partkey
FROM
part
WHERE
p_name LIKE 'forest%'
)
AND ps_availqty > (
SELECT
0.5 * SUM(l_quantity)
FROM
lineitem
WHERE
l_partkey = ps_partkey
AND l_suppkey = ps_suppkey
AND l_shipdate >= MDY(1,1,1994)
AND l_shipdate < MDY(1,1,1994) + 1 UNITS YEAR
)
)
AND s_nationkey = n_nationkey
AND n_name = 'CANADA'
`
,
`
SELECT
s_name,
s_address
FROM \${temp}
ORDER BY
s_name
`
);
</script>
20.3 Substitution Parameters
Values for the following substitution parameters must be generated and used to build the executable query text:
- COLOR is randomly selected within the list of values defined for the generation of P_NAME.
- DATE is the first of January of a randomly selected year within 1993..1997.
- NATION is randomly selected within the list of values defined for N_NAME in Clause 4.2.3.
20.4 Query Validation
For validation against the qualification database the query must be executed using the following values for substitution parameters and must produce the following output data:
Values for substitution parameters:
- COLOR = forest.
- DATE = 1994-01-01.
- QUANTITY3 = 20.
- NATION = CANADA.
Sample Output
| s_name | Supplier#000000020 |
| s_address | iybAE,RmTymrZVYaFZva2SH,j |
21 Q21
Copy
SELECT s_name,
Count(*) AS numwait
FROM supplier,
lineitem l1,
orders,
nation
WHERE s_suppkey = l1.l_suppkey
AND o_orderkey = l1.l_orderkey
AND o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (SELECT *
FROM lineitem l2
WHERE l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey <> l1.l_suppkey)
AND NOT EXISTS (SELECT *
FROM lineitem l3
WHERE l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey <> l1.l_suppkey
AND l3.l_receiptdate > l3.l_commitdate)
AND s_nationkey = n_nationkey
AND n_name = 'EGYPT'
GROUP BY s_name
ORDER BY numwait DESC,
s_name
LIMIT 100;
Copy
create temporary table l3 stored as orc as
select l_orderkey, count(distinct l_suppkey) as cntSupp
from lineitem
where l_receiptdate > l_commitdate and l_orderkey is not null
group by l_orderkey
having cntSupp = 1
;
with location as (
select supplier.* from supplier, nation where
s_nationkey = n_nationkey and n_name = 'EGYPT'
)
select s_name, count(*) as numwait
from
(
select li.l_suppkey, li.l_orderkey
from lineitem li join orders o on li.l_orderkey = o.o_orderkey and
o.o_orderstatus = 'F'
join
(
select l_orderkey, count(distinct l_suppkey) as cntSupp
from lineitem
group by l_orderkey
) l2 on li.l_orderkey = l2.l_orderkey and
li.l_receiptdate > li.l_commitdate and
l2.cntSupp > 1
) l1 join l3 on l1.l_orderkey = l3.l_orderkey
join location s on l1.l_suppkey = s.s_suppkey
group by
s_name
order by
numwait desc,
s_name
limit 100;
Copy
<script type="js">
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT s_name,
Count(*) AS numwait
FROM supplier,
lineitem l1,
orders,
nation
WHERE s_suppkey = l1.l_suppkey
AND o_orderkey = l1.l_orderkey
AND o_orderstatus = 'F'
AND l1.l_receiptdate > l1.l_commitdate
AND EXISTS (SELECT *
FROM lineitem l2
WHERE l2.l_orderkey = l1.l_orderkey
AND l2.l_suppkey <> l1.l_suppkey)
AND NOT EXISTS (SELECT *
FROM lineitem l3
WHERE l3.l_orderkey = l1.l_orderkey
AND l3.l_suppkey <> l1.l_suppkey
AND l3.l_receiptdate > l3.l_commitdate)
AND s_nationkey = n_nationkey
AND n_name = 'EGYPT'
GROUP BY s_name
ORDER BY numwait DESC,
s_name
`
,
`
SELECT FIRST 100
s_name,
SUM(numwait) AS numwait
FROM \${temp}
GROUP BY s_name
ORDER BY numwait DESC,
s_name
`
);
</script>
22 Q22
Copy
SELECT cntrycode,
Count(*) AS numcust,
Sum(c_acctbal) AS totacctbal
FROM (SELECT Substring(c_phone FROM 1 FOR 2) AS cntrycode,
c_acctbal
FROM customer
WHERE Substring(c_phone FROM 1 FOR 2) IN ( '20', '40', '22', '30',
'39', '42', '21' )
AND c_acctbal > (SELECT Avg(c_acctbal)
FROM customer
WHERE c_acctbal > 0.00
AND Substring(c_phone FROM 1 FOR 2) IN (
'20', '40', '22', '30',
'39', '42', '21' ))
AND NOT EXISTS (SELECT *
FROM orders
WHERE o_custkey = c_custkey)) AS custsale
GROUP BY cntrycode
ORDER BY cntrycode;
Copy
drop view q22_customer_tmp_cached; drop view q22_customer_tmp1_cached; drop view q22_orders_tmp_cached; create view if not exists q22_customer_tmp_cached as select c_acctbal, c_custkey, substr(c_phone, 1, 2) as cntrycode from customer where substr(c_phone, 1, 2) = '20' or substr(c_phone, 1, 2) = '40' or substr(c_phone, 1, 2) = '22' or substr(c_phone, 1, 2) = '30' or substr(c_phone, 1, 2) = '39' or substr(c_phone, 1, 2) = '42' or substr(c_phone, 1, 2) = '21'; create view if not exists q22_customer_tmp1_cached as select avg(c_acctbal) as avg_acctbal from q22_customer_tmp_cached where c_acctbal > 0.00; create view if not exists q22_orders_tmp_cached as select o_custkey from orders group by o_custkey; select cntrycode, count(1) as numcust, sum(c_acctbal) as totacctbal from ( select cntrycode, c_acctbal, avg_acctbal from q22_customer_tmp1_cached ct1 join ( select cntrycode, c_acctbal from q22_orders_tmp_cached ot right outer join q22_customer_tmp_cached ct on ct.c_custkey = ot.o_custkey where o_custkey is null ) ct2 ) a where c_acctbal > avg_acctbal group by cntrycode order by cntrycode;
Copy
<script type="js">
var grid = new Ax.Grid("grid8");
grid.execute("SET PDQPRIORITY 100");
return grid.execute(
`
SELECT cntrycode,
Count(*) AS numcust,
Sum(c_acctbal) AS totacctbal
FROM (SELECT Substring(c_phone FROM 1 FOR 2) AS cntrycode,
c_acctbal
FROM customer
WHERE Substring(c_phone FROM 1 FOR 2) IN ( '20', '40', '22', '30',
'39', '42', '21' )
AND c_acctbal > (SELECT Avg(c_acctbal)
FROM customer
WHERE c_acctbal > 0.00
AND Substring(c_phone FROM 1 FOR 2) IN (
'20', '40', '22', '30',
'39', '42', '21' ))
AND NOT EXISTS (SELECT *
FROM orders
WHERE o_custkey = c_custkey)) AS custsale
GROUP BY cntrycode
`
,
`
SELECT
cntrycode,
Sum(numcust) AS numcust,
Sum(totacctbal) AS totacctbal
FROM \${temp}
GROUP BY cntrycode
ORDER BY cntrycode
`
);
</script>