EMR log results

==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0152848G0BFCVCU18QK/controller.gz <==

2024-09-19T00:27:39.975Z INFO Ensure step 5 jar file command-runner.jar
2024-09-19T00:27:39.975Z INFO StepRunner: Created Runner for step 5
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m unique_active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r unique_active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/unique_active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-0152848G0BFCVCU18QK/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-0152848G0BFCVCU18QK
EMR_STEP_ID=s-0152848G0BFCVCU18QK
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-0152848G0BFCVCU18QK/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-0152848G0BFCVCU18QK/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-0152848G0BFCVCU18QK
INFO ProcessRunner started child process 31943
2024-09-19T00:27:39.976Z INFO HadoopJarStepRunner.Runner: startRun() called for s-0152848G0BFCVCU18QK Child Pid: 31943
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 578 seconds
2024-09-19T00:37:18.077Z INFO Step created jobs: job_1726705207091_0004
2024-09-19T00:37:18.077Z INFO Step succeeded with exitCode 0 and took 578 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0152848G0BFCVCU18QK/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0152848G0BFCVCU18QK/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-0152848G0BFCVCU18QK/tmp/streamjob3002774000926317474.jar tmpDir=null


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0152848G0BFCVCU18QK/syslog.gz <==

2024-09-19 00:27:44,320 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:27:44,428 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:27:44,454 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:27:44,455 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:27:44,690 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0004
2024-09-19 00:27:44,710 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:27:45,136 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:27:45,138 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:27:45,276 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:27:45,358 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:27:45,371 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:27:45,372 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:27:45,462 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0004
2024-09-19 00:27:45,462 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:27:45,591 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:27:45,591 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:27:45,639 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0004
2024-09-19 00:27:45,670 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0004/
2024-09-19 00:27:45,671 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0004
2024-09-19 00:27:50,729 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0004 running in uber mode : false
2024-09-19 00:27:50,730 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:28:07,814 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2024-09-19 00:28:13,836 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2024-09-19 00:28:19,859 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2024-09-19 00:28:25,880 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2024-09-19 00:28:31,900 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2024-09-19 00:28:32,903 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2024-09-19 00:28:37,918 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2024-09-19 00:28:43,936 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2024-09-19 00:28:49,957 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2024-09-19 00:28:55,976 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2024-09-19 00:29:01,993 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-19 00:29:08,011 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2024-09-19 00:29:14,031 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2024-09-19 00:29:15,034 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2024-09-19 00:29:20,050 INFO org.apache.hadoop.mapreduce.Job (main): map 16% reduce 0%
2024-09-19 00:29:26,069 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2024-09-19 00:29:32,087 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2024-09-19 00:29:38,106 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2024-09-19 00:29:42,130 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2024-09-19 00:29:43,134 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2024-09-19 00:29:44,138 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2024-09-19 00:29:45,141 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2024-09-19 00:29:46,144 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2024-09-19 00:29:47,150 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-19 00:29:48,154 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:30:00,210 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 1%
2024-09-19 00:30:01,213 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 3%
2024-09-19 00:30:04,219 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 4%
2024-09-19 00:30:05,222 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 4%
2024-09-19 00:30:11,236 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 4%
2024-09-19 00:30:18,253 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 4%
2024-09-19 00:30:25,270 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 4%
2024-09-19 00:30:32,286 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 4%
2024-09-19 00:30:39,302 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 4%
2024-09-19 00:30:46,318 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 4%
2024-09-19 00:30:53,334 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 4%
2024-09-19 00:30:59,351 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 4%
2024-09-19 00:31:06,367 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 4%
2024-09-19 00:31:13,383 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 4%
2024-09-19 00:31:20,398 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 4%
2024-09-19 00:31:27,413 INFO org.apache.hadoop.mapreduce.Job (main): map 44% reduce 4%
2024-09-19 00:31:32,424 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 4%
2024-09-19 00:31:34,429 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 4%
2024-09-19 00:31:35,432 INFO org.apache.hadoop.mapreduce.Job (main): map 48% reduce 4%
2024-09-19 00:31:36,435 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 4%
2024-09-19 00:31:37,438 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 5%
2024-09-19 00:31:38,440 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 5%
2024-09-19 00:31:39,443 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 6%
2024-09-19 00:31:43,452 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 7%
2024-09-19 00:31:46,458 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 7%
2024-09-19 00:31:49,465 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 8%
2024-09-19 00:31:50,467 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 9%
2024-09-19 00:31:51,469 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 10%
2024-09-19 00:31:52,473 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 11%
2024-09-19 00:31:53,475 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 12%
2024-09-19 00:31:54,477 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 12%
2024-09-19 00:32:03,497 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 12%
2024-09-19 00:32:13,520 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 12%
2024-09-19 00:32:22,538 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 12%
2024-09-19 00:32:31,558 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 12%
2024-09-19 00:32:40,575 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 12%
2024-09-19 00:32:50,595 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 12%
2024-09-19 00:33:00,615 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 12%
2024-09-19 00:33:09,632 INFO org.apache.hadoop.mapreduce.Job (main): map 62% reduce 12%
2024-09-19 00:33:18,650 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 12%
2024-09-19 00:33:23,660 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 12%
2024-09-19 00:33:24,662 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 12%
2024-09-19 00:33:25,664 INFO org.apache.hadoop.mapreduce.Job (main): map 66% reduce 13%
2024-09-19 00:33:26,666 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 13%
2024-09-19 00:33:27,668 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 13%
2024-09-19 00:33:28,671 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 14%
2024-09-19 00:33:31,677 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 15%
2024-09-19 00:33:33,681 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 15%
2024-09-19 00:33:35,685 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 16%
2024-09-19 00:33:41,696 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 16%
2024-09-19 00:33:50,713 INFO org.apache.hadoop.mapreduce.Job (main): map 71% reduce 16%
2024-09-19 00:33:58,728 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduce 16%
2024-09-19 00:34:08,747 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 16%
2024-09-19 00:34:18,766 INFO org.apache.hadoop.mapreduce.Job (main): map 74% red

==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0387261USN6O8QK6W9T/controller.gz <==

2024-09-19T00:21:36.546Z INFO Ensure step 2 jar file command-runner.jar
2024-09-19T00:21:36.546Z INFO StepRunner: Created Runner for step 2
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m deleted_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r deleted_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/deleted_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-0387261USN6O8QK6W9T/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-0387261USN6O8QK6W9T
EMR_STEP_ID=s-0387261USN6O8QK6W9T
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-0387261USN6O8QK6W9T/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-0387261USN6O8QK6W9T/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-0387261USN6O8QK6W9T
INFO ProcessRunner started child process 21764
2024-09-19T00:21:36.547Z INFO HadoopJarStepRunner.Runner: startRun() called for s-0387261USN6O8QK6W9T Child Pid: 21764
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 138 seconds
2024-09-19T00:23:54.779Z INFO Step created jobs: job_1726705207091_0001
2024-09-19T00:23:54.779Z INFO Step succeeded with exitCode 0 and took 138 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0387261USN6O8QK6W9T/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0387261USN6O8QK6W9T/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-0387261USN6O8QK6W9T/tmp/streamjob1357147515800484148.jar tmpDir=null


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0387261USN6O8QK6W9T/syslog.gz <==

2024-09-19 00:21:42,417 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:21:42,543 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:21:42,579 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:21:42,579 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:21:42,859 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0001
2024-09-19 00:21:42,888 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:21:44,730 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:21:44,732 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:21:44,880 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:21:45,670 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:21:45,687 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:21:45,688 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:21:46,062 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0001
2024-09-19 00:21:46,062 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:21:46,219 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:21:46,220 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:21:46,844 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0001
2024-09-19 00:21:46,892 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0001/
2024-09-19 00:21:46,893 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0001
2024-09-19 00:21:52,956 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0001 running in uber mode : false
2024-09-19 00:21:52,957 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:22:13,046 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2024-09-19 00:22:14,050 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2024-09-19 00:22:19,080 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2024-09-19 00:22:20,084 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2024-09-19 00:22:21,089 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2024-09-19 00:22:22,100 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-19 00:22:23,104 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:22:35,144 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2024-09-19 00:22:36,147 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 0%
2024-09-19 00:22:37,150 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 0%
2024-09-19 00:22:38,153 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 2%
2024-09-19 00:22:39,156 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 3%
2024-09-19 00:22:41,161 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 3%
2024-09-19 00:22:42,167 INFO org.apache.hadoop.mapreduce.Job (main): map 46% reduce 3%
2024-09-19 00:22:43,175 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 3%
2024-09-19 00:22:44,180 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 4%
2024-09-19 00:22:45,184 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 5%
2024-09-19 00:22:46,197 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 5%
2024-09-19 00:22:50,207 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 6%
2024-09-19 00:22:56,223 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 7%
2024-09-19 00:22:57,225 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 8%
2024-09-19 00:22:58,228 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 9%
2024-09-19 00:22:59,230 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 10%
2024-09-19 00:23:00,232 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 11%
2024-09-19 00:23:01,235 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 12%
2024-09-19 00:23:03,240 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 12%
2024-09-19 00:23:04,243 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 12%
2024-09-19 00:23:05,246 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 12%
2024-09-19 00:23:06,250 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 13%
2024-09-19 00:23:07,252 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 13%
2024-09-19 00:23:08,255 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 14%
2024-09-19 00:23:09,258 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 15%
2024-09-19 00:23:10,261 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 16%
2024-09-19 00:23:18,281 INFO org.apache.hadoop.mapreduce.Job (main): map 71% reduce 16%
2024-09-19 00:23:20,285 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 16%
2024-09-19 00:23:21,287 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 16%
2024-09-19 00:23:23,292 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%
2024-09-19 00:23:25,297 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 16%
2024-09-19 00:23:26,300 INFO org.apache.hadoop.mapreduce.Job (main): map 83% reduce 17%
2024-09-19 00:23:27,302 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-19 00:23:28,305 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 17%
2024-09-19 00:23:29,307 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 18%
2024-09-19 00:23:31,312 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 19%
2024-09-19 00:23:32,314 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 20%
2024-09-19 00:23:38,328 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 20%
2024-09-19 00:23:39,330 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 20%
2024-09-19 00:23:40,333 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 20%
2024-09-19 00:23:41,336 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 20%
2024-09-19 00:23:43,340 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 21%
2024-09-19 00:23:44,343 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 22%
2024-09-19 00:23:45,345 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 25%
2024-09-19 00:23:46,347 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 26%
2024-09-19 00:23:48,351 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 26%
2024-09-19 00:23:49,356 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 28%
2024-09-19 00:23:50,359 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 65%
2024-09-19 00:23:51,362 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-19 00:23:53,371 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0001 completed successfully
2024-09-19 00:23:53,444 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=508013991
FILE: Number of bytes written=1235574357
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578057751
S3: Number of bytes written=19480981
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=270
Launched reduce tasks=126
Data-local map tasks=270
Total time spent by all maps in occupied slots (ms)=715650688
Total time spent by all reduces in occupied slots (ms)=459966848
Total time spent by all map tasks (ms)=5591021
Total time spent by all reduce tasks (ms)=7186982
Total vcore-milliseconds taken by all map tasks=5591021
Total vcore-milliseconds taken by all reduce tasks=7186982
Total megabyte-milliseconds taken by all map tasks=22900822016
Total megabyte-milliseconds taken by all reduce tasks=14718939136
Map-Reduce Framework
Map input records=6112743
Map output records=3532675
Map output bytes=1586174074
Map output materialized bytes=610336241
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=54437
Reduce shuffle bytes=610336241
Reduce input records=3532675
Reduce output records=54437
Spilled Records=7065350
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=79531
CPU time spent (ms)=5088910
Physical memory (bytes) snapshot=303342305280
Vir

==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05070263163YN5L3C450/controller.gz <==

2024-09-19T00:37:23.080Z INFO Ensure step 6 jar file command-runner.jar
2024-09-19T00:37:23.080Z INFO StepRunner: Created Runner for step 6
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m active_duration -reducer php -d memory_limit=-1 mapreduce.phar -r active_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/active_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-05070263163YN5L3C450/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-05070263163YN5L3C450
EMR_STEP_ID=s-05070263163YN5L3C450
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-05070263163YN5L3C450/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-05070263163YN5L3C450/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-05070263163YN5L3C450
INFO ProcessRunner started child process 25523
2024-09-19T00:37:23.081Z INFO HadoopJarStepRunner.Runner: startRun() called for s-05070263163YN5L3C450 Child Pid: 25523
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 732 seconds
2024-09-19T00:49:35.242Z INFO Step created jobs: job_1726705207091_0005
2024-09-19T00:49:35.242Z INFO Step succeeded with exitCode 0 and took 732 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05070263163YN5L3C450/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05070263163YN5L3C450/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-05070263163YN5L3C450/tmp/streamjob5364588510317731456.jar tmpDir=null


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05070263163YN5L3C450/syslog.gz <==

2024-09-19 00:37:26,891 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:37:26,999 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:37:27,034 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:37:27,034 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:37:27,259 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0005
2024-09-19 00:37:27,279 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:37:27,775 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:37:27,777 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:37:27,915 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:37:27,996 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:37:28,008 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:37:28,009 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:37:28,104 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0005
2024-09-19 00:37:28,105 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:37:28,245 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:37:28,245 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:37:28,298 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0005
2024-09-19 00:37:28,343 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0005/
2024-09-19 00:37:28,344 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0005
2024-09-19 00:37:33,396 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0005 running in uber mode : false
2024-09-19 00:37:33,397 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:37:50,492 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2024-09-19 00:37:56,517 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2024-09-19 00:38:02,537 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2024-09-19 00:38:08,559 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2024-09-19 00:38:14,580 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2024-09-19 00:38:20,599 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2024-09-19 00:38:26,619 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2024-09-19 00:38:32,636 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2024-09-19 00:38:38,656 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2024-09-19 00:38:44,673 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2024-09-19 00:38:50,691 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-19 00:38:56,711 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2024-09-19 00:39:02,728 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2024-09-19 00:39:08,745 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2024-09-19 00:39:14,763 INFO org.apache.hadoop.mapreduce.Job (main): map 16% reduce 0%
2024-09-19 00:39:20,781 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2024-09-19 00:39:26,799 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2024-09-19 00:39:32,817 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2024-09-19 00:39:38,845 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2024-09-19 00:39:39,850 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2024-09-19 00:39:40,854 INFO org.apache.hadoop.mapreduce.Job (main): map 23% reduce 0%
2024-09-19 00:39:41,858 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2024-09-19 00:39:42,861 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2024-09-19 00:39:43,869 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2024-09-19 00:39:44,872 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-19 00:39:47,881 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:39:56,904 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 0%
2024-09-19 00:39:57,907 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 1%
2024-09-19 00:39:58,910 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 3%
2024-09-19 00:40:00,916 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 4%
2024-09-19 00:40:02,921 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 4%
2024-09-19 00:40:11,943 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 4%
2024-09-19 00:40:18,959 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 4%
2024-09-19 00:40:25,975 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 4%
2024-09-19 00:40:34,995 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 4%
2024-09-19 00:40:43,015 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 4%
2024-09-19 00:40:50,031 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 4%
2024-09-19 00:40:59,050 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 4%
2024-09-19 00:41:07,068 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 4%
2024-09-19 00:41:14,083 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 4%
2024-09-19 00:41:21,099 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 4%
2024-09-19 00:41:30,120 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 4%
2024-09-19 00:41:37,135 INFO org.apache.hadoop.mapreduce.Job (main): map 44% reduce 4%
2024-09-19 00:41:43,148 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 4%
2024-09-19 00:41:46,156 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 4%
2024-09-19 00:41:47,158 INFO org.apache.hadoop.mapreduce.Job (main): map 48% reduce 4%
2024-09-19 00:41:48,161 INFO org.apache.hadoop.mapreduce.Job (main): map 49% reduce 5%
2024-09-19 00:41:49,164 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 5%
2024-09-19 00:41:50,167 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 5%
2024-09-19 00:41:51,169 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 5%
2024-09-19 00:41:52,171 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 6%
2024-09-19 00:41:54,175 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 7%
2024-09-19 00:42:00,188 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 7%
2024-09-19 00:42:02,192 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 8%
2024-09-19 00:42:03,194 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 10%
2024-09-19 00:42:04,196 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 11%
2024-09-19 00:42:06,200 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 12%
2024-09-19 00:42:07,202 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 12%
2024-09-19 00:42:18,224 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 12%
2024-09-19 00:42:30,246 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 12%
2024-09-19 00:42:40,266 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 12%
2024-09-19 00:42:50,285 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 12%
2024-09-19 00:43:01,309 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 12%
2024-09-19 00:43:12,331 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 12%
2024-09-19 00:43:24,356 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 12%
2024-09-19 00:43:34,376 INFO org.apache.hadoop.mapreduce.Job (main): map 62% reduce 12%
2024-09-19 00:43:44,397 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 12%
2024-09-19 00:43:49,407 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 12%
2024-09-19 00:43:50,409 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 12%
2024-09-19 00:43:52,414 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 13%
2024-09-19 00:43:53,416 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 13%
2024-09-19 00:43:55,421 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 14%
2024-09-19 00:43:56,423 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 14%
2024-09-19 00:43:58,428 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 15%
2024-09-19 00:44:00,432 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 15%
2024-09-19 00:44:04,440 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 16%
2024-09-19 00:44:09,449 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 16%
2024-09-19 00:44:19,467 INFO org.apache.hadoop.mapreduce.Job (main): map 71% reduce 16%
2024-09-19 00:44:29,485 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduc

==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05135142OS6GQU0ETXNY/controller.gz <==

2024-09-19T00:49:40.245Z INFO Ensure step 7 jar file command-runner.jar
2024-09-19T00:49:40.245Z INFO StepRunner: Created Runner for step 7
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m closed_duration -reducer php -d memory_limit=-1 mapreduce.phar -r closed_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/closed_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-05135142OS6GQU0ETXNY/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-05135142OS6GQU0ETXNY
EMR_STEP_ID=s-05135142OS6GQU0ETXNY
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-05135142OS6GQU0ETXNY/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-05135142OS6GQU0ETXNY/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-05135142OS6GQU0ETXNY
INFO ProcessRunner started child process 30114
2024-09-19T00:49:40.247Z INFO HadoopJarStepRunner.Runner: startRun() called for s-05135142OS6GQU0ETXNY Child Pid: 30114
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 78 seconds
2024-09-19T00:50:58.305Z INFO Step created jobs: job_1726705207091_0006
2024-09-19T00:50:58.305Z INFO Step succeeded with exitCode 0 and took 78 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05135142OS6GQU0ETXNY/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05135142OS6GQU0ETXNY/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-05135142OS6GQU0ETXNY/tmp/streamjob3577923195899397959.jar tmpDir=null


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-05135142OS6GQU0ETXNY/syslog.gz <==

2024-09-19 00:49:44,102 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:49:44,225 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:49:44,256 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:49:44,256 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:49:44,508 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0006
2024-09-19 00:49:44,528 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:49:45,005 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:49:45,007 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:49:45,163 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:49:45,231 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:49:45,245 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:49:45,246 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:49:45,336 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0006
2024-09-19 00:49:45,336 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:49:45,494 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:49:45,494 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:49:45,549 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0006
2024-09-19 00:49:45,612 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0006/
2024-09-19 00:49:45,613 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0006
2024-09-19 00:49:50,703 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0006 running in uber mode : false
2024-09-19 00:49:50,704 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:50:03,785 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-19 00:50:04,789 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:50:14,830 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2024-09-19 00:50:15,833 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 0%
2024-09-19 00:50:16,836 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 0%
2024-09-19 00:50:20,852 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 2%
2024-09-19 00:50:26,869 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 2%
2024-09-19 00:50:27,872 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 2%
2024-09-19 00:50:28,875 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 3%
2024-09-19 00:50:29,878 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 4%
2024-09-19 00:50:30,881 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 8%
2024-09-19 00:50:31,883 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 12%
2024-09-19 00:50:32,886 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 16%
2024-09-19 00:50:33,889 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 17%
2024-09-19 00:50:35,895 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 17%
2024-09-19 00:50:36,898 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 17%
2024-09-19 00:50:37,901 INFO org.apache.hadoop.mapreduce.Job (main): map 82% reduce 17%
2024-09-19 00:50:38,903 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 18%
2024-09-19 00:50:39,906 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 18%
2024-09-19 00:50:40,909 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 18%
2024-09-19 00:50:42,915 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 19%
2024-09-19 00:50:43,918 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 20%
2024-09-19 00:50:44,921 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 20%
2024-09-19 00:50:45,924 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 20%
2024-09-19 00:50:46,928 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 20%
2024-09-19 00:50:47,931 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 26%
2024-09-19 00:50:48,933 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 53%
2024-09-19 00:50:49,936 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 80%
2024-09-19 00:50:50,939 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 95%
2024-09-19 00:50:51,941 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 96%
2024-09-19 00:50:52,944 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 97%
2024-09-19 00:50:53,948 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 98%
2024-09-19 00:50:54,951 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 99%
2024-09-19 00:50:55,953 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-19 00:50:56,959 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0006 completed successfully
2024-09-19 00:50:57,033 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=776745478
FILE: Number of bytes written=1721713967
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2577967322
S3: Number of bytes written=4631219
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Killed reduce tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=352661632
Total time spent by all reduces in occupied slots (ms)=210428736
Total time spent by all map tasks (ms)=2755169
Total time spent by all reduce tasks (ms)=3287949
Total vcore-milliseconds taken by all map tasks=2755169
Total vcore-milliseconds taken by all reduce tasks=3287949
Total megabyte-milliseconds taken by all map tasks=11285172224
Total megabyte-milliseconds taken by all reduce tasks=6733719552
Map-Reduce Framework
Map input records=6112743
Map output records=96274186
Map output bytes=6121578766
Map output materialized bytes=827746734
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=4688
Reduce shuffle bytes=827746734
Reduce input records=96274186
Reduce output records=4688
Spilled Records=192548372
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=85565
CPU time spent (ms)=2261970
Physical memory (bytes) snapshot=314649182208
Virtual memory (bytes) snapshot=1213403320320
Total committed heap usage (bytes)=289958526976
Peak Map Physical memory (bytes)=1204535296
Peak Map Virtual memory (bytes)=3045826560
Peak Reduce Physical memory (bytes)=1189134336
Peak Reduce Virtual memory (bytes)=3519500288
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2577967322
File Output Format Counters
Bytes Written=4631219
2024-09-19 00:50:57,033 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/closed_duration/


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0607339S1ENRUF5MJD6/controller.gz <==

2024-09-19T00:23:59.785Z INFO Ensure step 3 jar file command-runner.jar
2024-09-19T00:23:59.785Z INFO StepRunner: Created Runner for step 3
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m created_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r created_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/created_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-0607339S1ENRUF5MJD6/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-0607339S1ENRUF5MJD6
EMR_STEP_ID=s-0607339S1ENRUF5MJD6
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-0607339S1ENRUF5MJD6/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-0607339S1ENRUF5MJD6/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-0607339S1ENRUF5MJD6
INFO ProcessRunner started child process 6828
2024-09-19T00:23:59.786Z INFO HadoopJarStepRunner.Runner: startRun() called for s-0607339S1ENRUF5MJD6 Child Pid: 6828
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 118 seconds
2024-09-19T00:25:57.879Z INFO Step created jobs: job_1726705207091_0002
2024-09-19T00:25:57.879Z INFO Step succeeded with exitCode 0 and took 118 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0607339S1ENRUF5MJD6/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0607339S1ENRUF5MJD6/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-0607339S1ENRUF5MJD6/tmp/streamjob4418279634708905886.jar tmpDir=null


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-0607339S1ENRUF5MJD6/syslog.gz <==

2024-09-19 00:24:03,616 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:24:03,731 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:24:03,759 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:24:03,759 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:24:03,995 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0002
2024-09-19 00:24:04,014 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:24:04,375 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:24:04,377 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:24:04,511 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:24:04,600 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:24:04,617 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:24:04,618 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:24:04,699 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0002
2024-09-19 00:24:04,699 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:24:04,836 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:24:04,837 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:24:04,894 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0002
2024-09-19 00:24:04,941 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0002/
2024-09-19 00:24:04,942 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0002
2024-09-19 00:24:10,148 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0002 running in uber mode : false
2024-09-19 00:24:10,149 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:24:27,256 INFO org.apache.hadoop.mapreduce.Job (main): map 1% reduce 0%
2024-09-19 00:24:28,260 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-19 00:24:33,281 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2024-09-19 00:24:34,295 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2024-09-19 00:24:35,299 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:24:49,352 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 0%
2024-09-19 00:24:50,356 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 0%
2024-09-19 00:24:51,358 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 0%
2024-09-19 00:24:55,371 INFO org.apache.hadoop.mapreduce.Job (main): map 48% reduce 0%
2024-09-19 00:24:56,374 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 0%
2024-09-19 00:24:57,378 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 0%
2024-09-19 00:25:10,442 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 1%
2024-09-19 00:25:11,445 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 2%
2024-09-19 00:25:12,448 INFO org.apache.hadoop.mapreduce.Job (main): map 66% reduce 8%
2024-09-19 00:25:13,452 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 10%
2024-09-19 00:25:14,455 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 12%
2024-09-19 00:25:17,464 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 12%
2024-09-19 00:25:18,467 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 12%
2024-09-19 00:25:19,470 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 12%
2024-09-19 00:25:20,473 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 12%
2024-09-19 00:25:22,479 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 13%
2024-09-19 00:25:24,485 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 15%
2024-09-19 00:25:32,506 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 16%
2024-09-19 00:25:33,511 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 16%
2024-09-19 00:25:34,515 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 18%
2024-09-19 00:25:35,517 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 18%
2024-09-19 00:25:39,528 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 18%
2024-09-19 00:25:40,531 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 19%
2024-09-19 00:25:41,534 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 19%
2024-09-19 00:25:42,537 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 20%
2024-09-19 00:25:43,539 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 21%
2024-09-19 00:25:46,547 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 22%
2024-09-19 00:25:47,549 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 22%
2024-09-19 00:25:48,552 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 22%
2024-09-19 00:25:49,555 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 22%
2024-09-19 00:25:50,558 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 22%
2024-09-19 00:25:51,560 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 22%
2024-09-19 00:25:52,564 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 24%
2024-09-19 00:25:53,567 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 58%
2024-09-19 00:25:54,570 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-19 00:25:55,578 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0002 completed successfully
2024-09-19 00:25:55,676 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=539742193
FILE: Number of bytes written=1315834372
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578068559
S3: Number of bytes written=20480885
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=695755520
Total time spent by all reduces in occupied slots (ms)=324974464
Total time spent by all map tasks (ms)=5435590
Total time spent by all reduce tasks (ms)=5077726
Total vcore-milliseconds taken by all map tasks=5435590
Total vcore-milliseconds taken by all reduce tasks=5077726
Total megabyte-milliseconds taken by all map tasks=22264176640
Total megabyte-milliseconds taken by all reduce tasks=10399182848
Map-Reduce Framework
Map input records=6112743
Map output records=4228025
Map output bytes=1712596740
Map output materialized bytes=658868054
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=59760
Reduce shuffle bytes=658868054
Reduce input records=4228025
Reduce output records=59760
Spilled Records=8456050
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=81612
CPU time spent (ms)=5194880
Physical memory (bytes) snapshot=305750237184
Virtual memory (bytes) snapshot=1212608106496
Total committed heap usage (bytes)=284857204736
Peak Map Physical memory (bytes)=1444032512
Peak Map Virtual memory (bytes)=3591798784
Peak Reduce Physical memory (bytes)=742780928
Peak Reduce Virtual memory (bytes)=3432620032
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2578068559
File Output Format Counters
Bytes Written=20480885
2024-09-19 00:25:55,676 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/created_job_count/


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-06142593L2643XPDLWT1/controller.gz <==

2024-09-19T00:26:02.883Z INFO Ensure step 4 jar file command-runner.jar
2024-09-19T00:26:02.884Z INFO StepRunner: Created Runner for step 4
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-06142593L2643XPDLWT1/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-06142593L2643XPDLWT1
EMR_STEP_ID=s-06142593L2643XPDLWT1
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-06142593L2643XPDLWT1/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-06142593L2643XPDLWT1/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-06142593L2643XPDLWT1
INFO ProcessRunner started child process 20521
2024-09-19T00:26:02.885Z INFO HadoopJarStepRunner.Runner: startRun() called for s-06142593L2643XPDLWT1 Child Pid: 20521
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 92 seconds
2024-09-19T00:27:34.971Z INFO Step created jobs: job_1726705207091_0003
2024-09-19T00:27:34.971Z INFO Step succeeded with exitCode 0 and took 92 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-06142593L2643XPDLWT1/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-06142593L2643XPDLWT1/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-06142593L2643XPDLWT1/tmp/streamjob6787679777722853768.jar tmpDir=null


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-06142593L2643XPDLWT1/syslog.gz <==

2024-09-19 00:26:07,352 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:26:07,462 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:26:07,489 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:26:07,489 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:26:07,737 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0003
2024-09-19 00:26:07,755 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:26:08,222 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:26:08,224 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:26:08,352 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:26:08,425 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:26:08,442 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:26:08,443 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:26:08,525 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0003
2024-09-19 00:26:08,525 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:26:08,674 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:26:08,675 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:26:08,732 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0003
2024-09-19 00:26:08,783 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0003/
2024-09-19 00:26:08,784 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0003
2024-09-19 00:26:12,842 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0003 running in uber mode : false
2024-09-19 00:26:12,843 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:26:29,931 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2024-09-19 00:26:30,935 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:26:43,989 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2024-09-19 00:26:44,993 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2024-09-19 00:26:45,996 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 0%
2024-09-19 00:26:47,000 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 1%
2024-09-19 00:27:00,043 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 3%
2024-09-19 00:27:01,046 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 6%
2024-09-19 00:27:02,049 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 11%
2024-09-19 00:27:03,052 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 14%
2024-09-19 00:27:04,056 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 14%
2024-09-19 00:27:06,061 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 15%
2024-09-19 00:27:07,064 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 16%
2024-09-19 00:27:09,069 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 17%
2024-09-19 00:27:15,086 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 17%
2024-09-19 00:27:16,089 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 17%
2024-09-19 00:27:17,093 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-19 00:27:18,096 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 17%
2024-09-19 00:27:19,099 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 18%
2024-09-19 00:27:20,102 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 19%
2024-09-19 00:27:21,104 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 20%
2024-09-19 00:27:25,119 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 20%
2024-09-19 00:27:26,122 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 20%
2024-09-19 00:27:27,125 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 20%
2024-09-19 00:27:28,128 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 20%
2024-09-19 00:27:29,130 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 21%
2024-09-19 00:27:30,134 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 36%
2024-09-19 00:27:31,136 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 89%
2024-09-19 00:27:32,139 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 97%
2024-09-19 00:27:33,142 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 99%
2024-09-19 00:27:34,145 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-19 00:27:34,150 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0003 completed successfully
2024-09-19 00:27:34,215 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=129700578
FILE: Number of bytes written=390165697
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578094550
S3: Number of bytes written=2985205
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=486137984
Total time spent by all reduces in occupied slots (ms)=271101056
Total time spent by all map tasks (ms)=3797953
Total time spent by all reduce tasks (ms)=4235954
Total vcore-milliseconds taken by all map tasks=3797953
Total vcore-milliseconds taken by all reduce tasks=4235954
Total megabyte-milliseconds taken by all map tasks=15556415488
Total megabyte-milliseconds taken by all reduce tasks=8675233792
Map-Reduce Framework
Map input records=6112743
Map output records=61682
Map output bytes=464325959
Map output materialized bytes=143242179
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=1004
Reduce shuffle bytes=143242179
Reduce input records=61682
Reduce output records=1004
Spilled Records=123364
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=80621
CPU time spent (ms)=3203730
Physical memory (bytes) snapshot=309080395776
Virtual memory (bytes) snapshot=1211973513216
Total committed heap usage (bytes)=293059690496
Peak Map Physical memory (bytes)=1360744448
Peak Map Virtual memory (bytes)=3500429312
Peak Reduce Physical memory (bytes)=704131072
Peak Reduce Virtual memory (bytes)=3457339392
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2578094550
File Output Format Counters
Bytes Written=2985205
2024-09-19 00:27:34,216 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/active_job_count/


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-064464536ZPQ5S556AL6/controller.gz <==

2024-09-19T00:21:29.219Z INFO Ensure step 1 jar file command-runner.jar
2024-09-19T00:21:29.219Z INFO StepRunner: Created Runner for step 1
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar /bin/bash -c echo Debugging step is a no-op'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-064464536ZPQ5S556AL6/tmp
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-064464536ZPQ5S556AL6
EMR_STEP_ID=s-064464536ZPQ5S556AL6
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-064464536ZPQ5S556AL6/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-064464536ZPQ5S556AL6/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-064464536ZPQ5S556AL6
INFO ProcessRunner started child process 21528
2024-09-19T00:21:29.221Z INFO HadoopJarStepRunner.Runner: startRun() called for s-064464536ZPQ5S556AL6 Child Pid: 21528
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 2 seconds
2024-09-19T00:21:31.541Z INFO Step created jobs:
2024-09-19T00:21:31.541Z INFO Step succeeded with exitCode 0 and took 2 seconds


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-064464536ZPQ5S556AL6/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-18-0/j-1U8X7KDGRQ0R8/steps/s-064464536ZPQ5S556AL6/stdout.gz <==

Debugging step is a no-op


Loading…
Loading the web debug toolbar…
Attempt #