EMR log results

==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-00921997DJQT4WBYAKL/controller.gz <==

2024-09-17T00:25:43.100Z INFO Ensure step 4 jar file command-runner.jar
2024-09-17T00:25:43.100Z INFO StepRunner: Created Runner for step 4
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-00921997DJQT4WBYAKL/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-00921997DJQT4WBYAKL
EMR_STEP_ID=s-00921997DJQT4WBYAKL
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-00921997DJQT4WBYAKL/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-00921997DJQT4WBYAKL/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-00921997DJQT4WBYAKL
INFO ProcessRunner started child process 19758
2024-09-17T00:25:43.102Z INFO HadoopJarStepRunner.Runner: startRun() called for s-00921997DJQT4WBYAKL Child Pid: 19758
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 92 seconds
2024-09-17T00:27:15.168Z INFO Step created jobs: job_1726532387964_0003
2024-09-17T00:27:15.168Z INFO Step succeeded with exitCode 0 and took 92 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-00921997DJQT4WBYAKL/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-00921997DJQT4WBYAKL/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-00921997DJQT4WBYAKL/tmp/streamjob7456955950595118183.jar tmpDir=null


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-00921997DJQT4WBYAKL/syslog.gz <==

2024-09-17 00:25:46,561 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:25:46,673 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:25:46,702 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:25:46,703 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:25:46,920 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0003
2024-09-17 00:25:46,936 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:25:47,273 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:25:47,274 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:25:47,393 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:25:47,465 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:25:47,485 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:25:47,486 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:25:47,569 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0003
2024-09-17 00:25:47,569 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:25:47,719 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:25:47,719 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:25:47,768 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0003
2024-09-17 00:25:47,797 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0003/
2024-09-17 00:25:47,798 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0003
2024-09-17 00:25:51,844 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0003 running in uber mode : false
2024-09-17 00:25:51,845 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:26:07,941 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2024-09-17 00:26:08,945 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-17 00:26:09,952 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:26:23,001 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 0%
2024-09-17 00:26:24,005 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 0%
2024-09-17 00:26:25,008 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 0%
2024-09-17 00:26:38,051 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 2%
2024-09-17 00:26:39,055 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 5%
2024-09-17 00:26:40,058 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 9%
2024-09-17 00:26:41,062 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 13%
2024-09-17 00:26:42,066 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 13%
2024-09-17 00:26:43,069 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 13%
2024-09-17 00:26:44,072 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 14%
2024-09-17 00:26:45,074 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 15%
2024-09-17 00:26:46,077 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 16%
2024-09-17 00:26:53,097 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%
2024-09-17 00:26:54,099 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 17%
2024-09-17 00:26:55,103 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 17%
2024-09-17 00:26:56,106 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 18%
2024-09-17 00:26:57,109 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 18%
2024-09-17 00:26:58,113 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 19%
2024-09-17 00:26:59,123 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 20%
2024-09-17 00:27:02,131 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 21%
2024-09-17 00:27:03,134 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 21%
2024-09-17 00:27:04,137 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 21%
2024-09-17 00:27:05,140 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 21%
2024-09-17 00:27:06,143 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 21%
2024-09-17 00:27:08,148 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 29%
2024-09-17 00:27:09,153 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 77%
2024-09-17 00:27:10,157 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 96%
2024-09-17 00:27:11,160 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-17 00:27:13,170 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0003 completed successfully
2024-09-17 00:27:13,235 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=129628173
FILE: Number of bytes written=390021782
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578056797
S3: Number of bytes written=2984089
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=480734848
Total time spent by all reduces in occupied slots (ms)=267831360
Total time spent by all map tasks (ms)=3755741
Total time spent by all reduce tasks (ms)=4184865
Total vcore-milliseconds taken by all map tasks=3755741
Total vcore-milliseconds taken by all reduce tasks=4184865
Total megabyte-milliseconds taken by all map tasks=15383515136
Total megabyte-milliseconds taken by all reduce tasks=8570603520
Map-Reduce Framework
Map input records=6112743
Map output records=61682
Map output bytes=464092492
Map output materialized bytes=143178174
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=1004
Reduce shuffle bytes=143178174
Reduce input records=61682
Reduce output records=1004
Spilled Records=123364
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=77419
CPU time spent (ms)=3055450
Physical memory (bytes) snapshot=310040559616
Virtual memory (bytes) snapshot=1212113440768
Total committed heap usage (bytes)=293343330304
Peak Map Physical memory (bytes)=1368006656
Peak Map Virtual memory (bytes)=3481489408
Peak Reduce Physical memory (bytes)=709853184
Peak Reduce Virtual memory (bytes)=3480350720
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2578056797
File Output Format Counters
Bytes Written=2984089
2024-09-17 00:27:13,235 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/active_job_count/


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-05941293PWESHPWTN3CF/controller.gz <==

2024-09-17T00:23:40.022Z INFO Ensure step 3 jar file command-runner.jar
2024-09-17T00:23:40.023Z INFO StepRunner: Created Runner for step 3
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m created_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r created_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/created_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-05941293PWESHPWTN3CF/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-05941293PWESHPWTN3CF
EMR_STEP_ID=s-05941293PWESHPWTN3CF
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-05941293PWESHPWTN3CF/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-05941293PWESHPWTN3CF/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-05941293PWESHPWTN3CF
INFO ProcessRunner started child process 5213
2024-09-17T00:23:40.024Z INFO HadoopJarStepRunner.Runner: startRun() called for s-05941293PWESHPWTN3CF Child Pid: 5213
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 118 seconds
2024-09-17T00:25:38.096Z INFO Step created jobs: job_1726532387964_0002
2024-09-17T00:25:38.097Z INFO Step succeeded with exitCode 0 and took 118 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-05941293PWESHPWTN3CF/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-05941293PWESHPWTN3CF/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-05941293PWESHPWTN3CF/tmp/streamjob8163018783477702776.jar tmpDir=null


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-05941293PWESHPWTN3CF/syslog.gz <==

2024-09-17 00:23:43,916 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:23:44,095 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:23:44,131 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:23:44,132 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:23:44,541 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0002
2024-09-17 00:23:44,562 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:23:45,115 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:23:45,116 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:23:45,247 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:23:45,317 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:23:45,329 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:23:45,330 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:23:45,414 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0002
2024-09-17 00:23:45,414 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:23:45,553 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:23:45,553 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:23:45,605 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0002
2024-09-17 00:23:45,653 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0002/
2024-09-17 00:23:45,654 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0002
2024-09-17 00:23:49,701 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0002 running in uber mode : false
2024-09-17 00:23:49,701 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:24:06,793 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-17 00:24:12,825 INFO org.apache.hadoop.mapreduce.Job (main): map 27% reduce 0%
2024-09-17 00:24:13,828 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:24:27,875 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 0%
2024-09-17 00:24:28,878 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2024-09-17 00:24:29,881 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 1%
2024-09-17 00:24:33,894 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 1%
2024-09-17 00:24:34,898 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 1%
2024-09-17 00:24:35,902 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 2%
2024-09-17 00:24:41,925 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 3%
2024-09-17 00:24:48,944 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 4%
2024-09-17 00:24:49,947 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 5%
2024-09-17 00:24:50,950 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 10%
2024-09-17 00:24:51,953 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 11%
2024-09-17 00:24:52,956 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 12%
2024-09-17 00:24:54,962 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 12%
2024-09-17 00:24:55,965 INFO org.apache.hadoop.mapreduce.Job (main): map 66% reduce 12%
2024-09-17 00:24:56,968 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 12%
2024-09-17 00:24:57,971 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 12%
2024-09-17 00:24:58,974 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 12%
2024-09-17 00:24:59,977 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 13%
2024-09-17 00:25:01,982 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 14%
2024-09-17 00:25:02,985 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 15%
2024-09-17 00:25:11,005 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 17%
2024-09-17 00:25:12,007 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 17%
2024-09-17 00:25:13,010 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 17%
2024-09-17 00:25:17,021 INFO org.apache.hadoop.mapreduce.Job (main): map 82% reduce 17%
2024-09-17 00:25:18,024 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-17 00:25:19,027 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 18%
2024-09-17 00:25:20,030 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 18%
2024-09-17 00:25:21,032 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 19%
2024-09-17 00:25:23,038 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 20%
2024-09-17 00:25:25,043 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 21%
2024-09-17 00:25:30,055 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 21%
2024-09-17 00:25:31,058 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 21%
2024-09-17 00:25:32,060 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 21%
2024-09-17 00:25:33,063 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 22%
2024-09-17 00:25:34,066 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 25%
2024-09-17 00:25:35,069 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 60%
2024-09-17 00:25:36,071 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 99%
2024-09-17 00:25:37,074 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-17 00:25:37,078 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0002 completed successfully
2024-09-17 00:25:37,140 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=539728038
FILE: Number of bytes written=1315812712
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578086780
S3: Number of bytes written=20479136
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=688424192
Total time spent by all reduces in occupied slots (ms)=374868544
Total time spent by all map tasks (ms)=5378314
Total time spent by all reduce tasks (ms)=5857321
Total vcore-milliseconds taken by all map tasks=5378314
Total vcore-milliseconds taken by all reduce tasks=5857321
Total megabyte-milliseconds taken by all map tasks=22029574144
Total megabyte-milliseconds taken by all reduce tasks=11995793408
Map-Reduce Framework
Map input records=6112743
Map output records=4228025
Map output bytes=1712596740
Map output materialized bytes=658868054
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=59760
Reduce shuffle bytes=658868054
Reduce input records=4228025
Reduce output records=59760
Spilled Records=8456050
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=80352
CPU time spent (ms)=5140130
Physical memory (bytes) snapshot=310197321728
Virtual memory (bytes) snapshot=1213021319168
Total committed heap usage (bytes)=289142734848
Peak Map Physical memory (bytes)=1457094656
Peak Map Virtual memory (bytes)=3597348864
Peak Reduce Physical memory (bytes)=744247296
Peak Reduce Virtual memory (bytes)=3336634368
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2578086780
File Output Format Counters
Bytes Written=20479136
2024-09-17 00:25:37,140 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/created_job_count/


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-07842242O63ANN00L24/controller.gz <==

2024-09-17T00:21:20.786Z INFO Ensure step 2 jar file command-runner.jar
2024-09-17T00:21:20.787Z INFO StepRunner: Created Runner for step 2
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m deleted_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r deleted_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/deleted_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-07842242O63ANN00L24/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-07842242O63ANN00L24
EMR_STEP_ID=s-07842242O63ANN00L24
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-07842242O63ANN00L24/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-07842242O63ANN00L24/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-07842242O63ANN00L24
INFO ProcessRunner started child process 21723
2024-09-17T00:21:20.788Z INFO HadoopJarStepRunner.Runner: startRun() called for s-07842242O63ANN00L24 Child Pid: 21723
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 134 seconds
2024-09-17T00:23:35.017Z INFO Step created jobs: job_1726532387964_0001
2024-09-17T00:23:35.017Z INFO Step succeeded with exitCode 0 and took 134 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-07842242O63ANN00L24/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-07842242O63ANN00L24/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-07842242O63ANN00L24/tmp/streamjob3273149539654220290.jar tmpDir=null


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-07842242O63ANN00L24/syslog.gz <==

2024-09-17 00:21:24,565 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:21:24,662 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:21:24,687 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:21:24,688 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:21:24,913 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0001
2024-09-17 00:21:24,939 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:21:26,214 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:21:26,215 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:21:26,342 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:21:27,196 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:21:27,212 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:21:27,213 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:21:27,341 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0001
2024-09-17 00:21:27,341 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:21:27,480 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:21:27,480 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:21:28,004 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0001
2024-09-17 00:21:28,039 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0001/
2024-09-17 00:21:28,041 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0001
2024-09-17 00:21:34,112 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0001 running in uber mode : false
2024-09-17 00:21:34,113 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:21:53,220 INFO org.apache.hadoop.mapreduce.Job (main): map 1% reduce 0%
2024-09-17 00:21:54,225 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2024-09-17 00:21:59,243 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-17 00:22:00,257 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2024-09-17 00:22:01,262 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2024-09-17 00:22:02,267 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2024-09-17 00:22:03,272 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:22:15,317 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 0%
2024-09-17 00:22:16,321 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 0%
2024-09-17 00:22:17,324 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 0%
2024-09-17 00:22:18,328 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 2%
2024-09-17 00:22:19,331 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 3%
2024-09-17 00:22:21,337 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 3%
2024-09-17 00:22:22,341 INFO org.apache.hadoop.mapreduce.Job (main): map 46% reduce 3%
2024-09-17 00:22:23,345 INFO org.apache.hadoop.mapreduce.Job (main): map 49% reduce 3%
2024-09-17 00:22:24,349 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 4%
2024-09-17 00:22:25,366 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 5%
2024-09-17 00:22:26,370 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 5%
2024-09-17 00:22:30,387 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 6%
2024-09-17 00:22:37,409 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 6%
2024-09-17 00:22:38,412 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 7%
2024-09-17 00:22:39,415 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 9%
2024-09-17 00:22:40,418 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 11%
2024-09-17 00:22:41,421 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 12%
2024-09-17 00:22:42,424 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 12%
2024-09-17 00:22:43,427 INFO org.apache.hadoop.mapreduce.Job (main): map 62% reduce 12%
2024-09-17 00:22:44,431 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 12%
2024-09-17 00:22:45,434 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 13%
2024-09-17 00:22:46,437 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 13%
2024-09-17 00:22:48,444 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 14%
2024-09-17 00:22:49,447 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 15%
2024-09-17 00:22:51,453 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 16%
2024-09-17 00:22:59,476 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 16%
2024-09-17 00:23:00,479 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 16%
2024-09-17 00:23:02,484 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 16%
2024-09-17 00:23:04,490 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%
2024-09-17 00:23:05,493 INFO org.apache.hadoop.mapreduce.Job (main): map 80% reduce 16%
2024-09-17 00:23:06,496 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 16%
2024-09-17 00:23:07,499 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-17 00:23:08,502 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 17%
2024-09-17 00:23:09,505 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 18%
2024-09-17 00:23:10,508 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 18%
2024-09-17 00:23:11,511 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 19%
2024-09-17 00:23:13,517 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 20%
2024-09-17 00:23:18,530 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 20%
2024-09-17 00:23:19,534 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 20%
2024-09-17 00:23:20,537 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 20%
2024-09-17 00:23:21,540 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 20%
2024-09-17 00:23:22,543 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 21%
2024-09-17 00:23:24,548 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 22%
2024-09-17 00:23:25,551 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 24%
2024-09-17 00:23:26,554 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 24%
2024-09-17 00:23:27,557 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 24%
2024-09-17 00:23:28,560 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 25%
2024-09-17 00:23:29,563 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 28%
2024-09-17 00:23:30,567 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 70%
2024-09-17 00:23:31,570 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-17 00:23:32,578 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0001 completed successfully
2024-09-17 00:23:32,647 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=508011134
FILE: Number of bytes written=1235563995
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2577988064
S3: Number of bytes written=19480272
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=270
Launched reduce tasks=126
Data-local map tasks=270
Total time spent by all maps in occupied slots (ms)=708789248
Total time spent by all reduces in occupied slots (ms)=455592320
Total time spent by all map tasks (ms)=5537416
Total time spent by all reduce tasks (ms)=7118630
Total vcore-milliseconds taken by all map tasks=5537416
Total vcore-milliseconds taken by all reduce tasks=7118630
Total megabyte-milliseconds taken by all map tasks=22681255936
Total megabyte-milliseconds taken by all reduce tasks=14578954240
Map-Reduce Framework
Map input records=6112743
Map output records=3532675
Map output bytes=1586174074
Map output materialized bytes=610336241
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=54437
Reduce shuffle bytes=610336241
Reduce input records=3532675
Reduce output records=54437
Spilled Records=7065350
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=77568


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-09228062W8QFDT2FMM96/controller.gz <==

2024-09-17T00:27:20.171Z INFO Ensure step 5 jar file command-runner.jar
2024-09-17T00:27:20.172Z INFO StepRunner: Created Runner for step 5
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m unique_active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r unique_active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/unique_active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-09228062W8QFDT2FMM96/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-09228062W8QFDT2FMM96
EMR_STEP_ID=s-09228062W8QFDT2FMM96
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-09228062W8QFDT2FMM96/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-09228062W8QFDT2FMM96/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-09228062W8QFDT2FMM96
INFO ProcessRunner started child process 32054
2024-09-17T00:27:20.174Z INFO HadoopJarStepRunner.Runner: startRun() called for s-09228062W8QFDT2FMM96 Child Pid: 32054
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 572 seconds
2024-09-17T00:36:52.266Z INFO Step created jobs: job_1726532387964_0004
2024-09-17T00:36:52.266Z INFO Step succeeded with exitCode 0 and took 572 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-09228062W8QFDT2FMM96/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-09228062W8QFDT2FMM96/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-09228062W8QFDT2FMM96/tmp/streamjob439320400188967834.jar tmpDir=null


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-09228062W8QFDT2FMM96/syslog.gz <==

2024-09-17 00:27:23,732 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:27:23,838 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:27:23,864 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:27:23,864 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:27:24,152 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0004
2024-09-17 00:27:24,169 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:27:24,530 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:27:24,532 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:27:24,654 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:27:24,718 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:27:24,728 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:27:24,729 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:27:24,817 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0004
2024-09-17 00:27:24,817 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:27:24,960 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:27:24,960 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:27:25,009 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0004
2024-09-17 00:27:25,039 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0004/
2024-09-17 00:27:25,040 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0004
2024-09-17 00:27:30,123 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0004 running in uber mode : false
2024-09-17 00:27:30,124 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:27:46,225 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2024-09-17 00:27:52,246 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2024-09-17 00:27:58,265 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2024-09-17 00:28:04,285 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2024-09-17 00:28:10,303 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2024-09-17 00:28:16,322 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2024-09-17 00:28:22,340 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2024-09-17 00:28:28,358 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2024-09-17 00:28:34,378 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2024-09-17 00:28:40,399 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-17 00:28:46,416 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2024-09-17 00:28:52,432 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2024-09-17 00:28:58,449 INFO org.apache.hadoop.mapreduce.Job (main): map 16% reduce 0%
2024-09-17 00:29:04,465 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2024-09-17 00:29:10,482 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2024-09-17 00:29:16,499 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2024-09-17 00:29:19,518 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2024-09-17 00:29:20,521 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2024-09-17 00:29:21,524 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2024-09-17 00:29:22,527 INFO org.apache.hadoop.mapreduce.Job (main): map 27% reduce 0%
2024-09-17 00:29:23,531 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2024-09-17 00:29:24,534 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-17 00:29:25,537 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:29:36,563 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 0%
2024-09-17 00:29:37,566 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 1%
2024-09-17 00:29:38,570 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 2%
2024-09-17 00:29:39,573 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 3%
2024-09-17 00:29:40,575 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 4%
2024-09-17 00:29:41,577 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 4%
2024-09-17 00:29:48,592 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 4%
2024-09-17 00:29:55,607 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 4%
2024-09-17 00:30:01,620 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 4%
2024-09-17 00:30:09,636 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 4%
2024-09-17 00:30:16,649 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 4%
2024-09-17 00:30:23,662 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 4%
2024-09-17 00:30:29,673 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 4%
2024-09-17 00:30:36,688 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 4%
2024-09-17 00:30:43,705 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 4%
2024-09-17 00:30:49,715 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 4%
2024-09-17 00:30:56,726 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 4%
2024-09-17 00:31:04,740 INFO org.apache.hadoop.mapreduce.Job (main): map 44% reduce 4%
2024-09-17 00:31:08,749 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 4%
2024-09-17 00:31:10,753 INFO org.apache.hadoop.mapreduce.Job (main): map 46% reduce 4%
2024-09-17 00:31:11,756 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 4%
2024-09-17 00:31:12,758 INFO org.apache.hadoop.mapreduce.Job (main): map 49% reduce 4%
2024-09-17 00:31:13,760 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 5%
2024-09-17 00:31:15,765 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 6%
2024-09-17 00:31:19,773 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 7%
2024-09-17 00:31:24,783 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 7%
2024-09-17 00:31:25,785 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 8%
2024-09-17 00:31:26,787 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 9%
2024-09-17 00:31:27,789 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 10%
2024-09-17 00:31:29,793 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 11%
2024-09-17 00:31:30,794 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 12%
2024-09-17 00:31:39,810 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 12%
2024-09-17 00:31:50,833 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 12%
2024-09-17 00:31:58,846 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 12%
2024-09-17 00:32:08,863 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 12%
2024-09-17 00:32:18,878 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 12%
2024-09-17 00:32:27,892 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 12%
2024-09-17 00:32:38,909 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 12%
2024-09-17 00:32:46,921 INFO org.apache.hadoop.mapreduce.Job (main): map 62% reduce 12%
2024-09-17 00:32:56,937 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 12%
2024-09-17 00:32:59,943 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 12%
2024-09-17 00:33:00,944 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 12%
2024-09-17 00:33:01,946 INFO org.apache.hadoop.mapreduce.Job (main): map 66% reduce 13%
2024-09-17 00:33:03,950 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 14%
2024-09-17 00:33:04,951 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 14%
2024-09-17 00:33:06,954 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 15%
2024-09-17 00:33:09,959 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 16%
2024-09-17 00:33:13,966 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 16%
2024-09-17 00:33:19,976 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 16%
2024-09-17 00:33:28,990 INFO org.apache.hadoop.mapreduce.Job (main): map 71% reduce 16%
2024-09-17 00:33:39,007 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduce 16%
2024-09-17 00:33:49,024 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 16%
2024-09-17 00:33:59,043 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 16%
2024-09-17 00:34:08,055 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 16%
2024-09-17 00:34:18,069 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 16%
2024-09-17 00:34:28,084 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%

==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-099979118CX8O5WOYZJA/controller.gz <==

2024-09-17T00:21:13.488Z INFO Ensure step 1 jar file command-runner.jar
2024-09-17T00:21:13.488Z INFO StepRunner: Created Runner for step 1
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar /bin/bash -c echo Debugging step is a no-op'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-099979118CX8O5WOYZJA/tmp
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-099979118CX8O5WOYZJA
EMR_STEP_ID=s-099979118CX8O5WOYZJA
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-099979118CX8O5WOYZJA/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-099979118CX8O5WOYZJA/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-099979118CX8O5WOYZJA
INFO ProcessRunner started child process 21494
2024-09-17T00:21:13.491Z INFO HadoopJarStepRunner.Runner: startRun() called for s-099979118CX8O5WOYZJA Child Pid: 21494
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 2 seconds
2024-09-17T00:21:15.781Z INFO Step created jobs:
2024-09-17T00:21:15.782Z INFO Step succeeded with exitCode 0 and took 2 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-099979118CX8O5WOYZJA/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-099979118CX8O5WOYZJA/stdout.gz <==

Debugging step is a no-op


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-10122202EESN0U3GC6D1/controller.gz <==

2024-09-17T00:36:57.269Z INFO Ensure step 6 jar file command-runner.jar
2024-09-17T00:36:57.269Z INFO StepRunner: Created Runner for step 6
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m active_duration -reducer php -d memory_limit=-1 mapreduce.phar -r active_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/active_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-10122202EESN0U3GC6D1/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-10122202EESN0U3GC6D1
EMR_STEP_ID=s-10122202EESN0U3GC6D1
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-10122202EESN0U3GC6D1/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-10122202EESN0U3GC6D1/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-10122202EESN0U3GC6D1
INFO ProcessRunner started child process 23264
2024-09-17T00:36:57.270Z INFO HadoopJarStepRunner.Runner: startRun() called for s-10122202EESN0U3GC6D1 Child Pid: 23264
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 726 seconds
2024-09-17T00:49:03.359Z INFO Step created jobs: job_1726532387964_0005
2024-09-17T00:49:03.359Z INFO Step succeeded with exitCode 0 and took 726 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-10122202EESN0U3GC6D1/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-10122202EESN0U3GC6D1/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-10122202EESN0U3GC6D1/tmp/streamjob6772735413900937421.jar tmpDir=null


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-10122202EESN0U3GC6D1/syslog.gz <==

2024-09-17 00:37:00,867 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:37:00,972 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:37:00,999 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:37:00,999 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:37:01,282 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0005
2024-09-17 00:37:01,299 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:37:01,875 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:37:01,877 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:37:02,013 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:37:02,074 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:37:02,086 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:37:02,087 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:37:02,161 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0005
2024-09-17 00:37:02,161 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:37:02,287 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:37:02,287 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:37:02,331 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0005
2024-09-17 00:37:02,356 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0005/
2024-09-17 00:37:02,357 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0005
2024-09-17 00:37:07,417 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0005 running in uber mode : false
2024-09-17 00:37:07,417 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:37:24,498 INFO org.apache.hadoop.mapreduce.Job (main): map 2% reduce 0%
2024-09-17 00:37:30,520 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2024-09-17 00:37:36,541 INFO org.apache.hadoop.mapreduce.Job (main): map 4% reduce 0%
2024-09-17 00:37:42,563 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2024-09-17 00:37:48,583 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2024-09-17 00:37:54,606 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2024-09-17 00:38:00,625 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2024-09-17 00:38:06,644 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2024-09-17 00:38:12,665 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2024-09-17 00:38:18,683 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2024-09-17 00:38:24,702 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-17 00:38:30,720 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2024-09-17 00:38:36,738 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2024-09-17 00:38:42,758 INFO org.apache.hadoop.mapreduce.Job (main): map 15% reduce 0%
2024-09-17 00:38:48,777 INFO org.apache.hadoop.mapreduce.Job (main): map 16% reduce 0%
2024-09-17 00:38:54,795 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2024-09-17 00:39:00,813 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2024-09-17 00:39:06,830 INFO org.apache.hadoop.mapreduce.Job (main): map 19% reduce 0%
2024-09-17 00:39:11,855 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2024-09-17 00:39:12,858 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2024-09-17 00:39:13,862 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2024-09-17 00:39:14,867 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2024-09-17 00:39:15,871 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2024-09-17 00:39:16,874 INFO org.apache.hadoop.mapreduce.Job (main): map 27% reduce 0%
2024-09-17 00:39:17,878 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2024-09-17 00:39:18,881 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-17 00:39:20,887 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:39:29,909 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 1%
2024-09-17 00:39:30,911 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 2%
2024-09-17 00:39:31,914 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 3%
2024-09-17 00:39:33,919 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 4%
2024-09-17 00:39:36,926 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 4%
2024-09-17 00:39:44,946 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 4%
2024-09-17 00:39:52,965 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 4%
2024-09-17 00:39:59,982 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 4%
2024-09-17 00:40:09,002 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 4%
2024-09-17 00:40:16,020 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 4%
2024-09-17 00:40:24,043 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 4%
2024-09-17 00:40:32,061 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 4%
2024-09-17 00:40:40,078 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 4%
2024-09-17 00:40:47,093 INFO org.apache.hadoop.mapreduce.Job (main): map 41% reduce 4%
2024-09-17 00:40:55,110 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 4%
2024-09-17 00:41:03,127 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 4%
2024-09-17 00:41:11,145 INFO org.apache.hadoop.mapreduce.Job (main): map 44% reduce 4%
2024-09-17 00:41:17,158 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 4%
2024-09-17 00:41:19,164 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 4%
2024-09-17 00:41:20,167 INFO org.apache.hadoop.mapreduce.Job (main): map 48% reduce 4%
2024-09-17 00:41:21,171 INFO org.apache.hadoop.mapreduce.Job (main): map 49% reduce 5%
2024-09-17 00:41:22,174 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 5%
2024-09-17 00:41:23,178 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 5%
2024-09-17 00:41:25,182 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 6%
2024-09-17 00:41:26,184 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 6%
2024-09-17 00:41:28,189 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 7%
2024-09-17 00:41:32,198 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 7%
2024-09-17 00:41:34,202 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 8%
2024-09-17 00:41:35,204 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 9%
2024-09-17 00:41:36,207 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 10%
2024-09-17 00:41:37,209 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 11%
2024-09-17 00:41:39,213 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 12%
2024-09-17 00:41:41,217 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 12%
2024-09-17 00:41:52,239 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 12%
2024-09-17 00:42:03,260 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 12%
2024-09-17 00:42:14,281 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 12%
2024-09-17 00:42:25,302 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 12%
2024-09-17 00:42:36,323 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 12%
2024-09-17 00:42:47,344 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 12%
2024-09-17 00:42:58,365 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 12%
2024-09-17 00:43:09,386 INFO org.apache.hadoop.mapreduce.Job (main): map 62% reduce 12%
2024-09-17 00:43:19,406 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 12%
2024-09-17 00:43:22,412 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 12%
2024-09-17 00:43:24,416 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 12%
2024-09-17 00:43:25,418 INFO org.apache.hadoop.mapreduce.Job (main): map 66% reduce 13%
2024-09-17 00:43:27,423 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 13%
2024-09-17 00:43:28,425 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 14%
2024-09-17 00:43:30,429 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 14%
2024-09-17 00:43:31,431 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 15%
2024-09-17 00:43:36,442 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 15%
2024-09-17 00:43:37,444 INFO org.apache.hadoop.mapreduce.Job (main): map 69% reduce 16%
2024-09-17 00:43:45,459 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 16%
2024-

==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-102328684JEXYWJIOS4/controller.gz <==

2024-09-17T00:49:08.362Z INFO Ensure step 7 jar file command-runner.jar
2024-09-17T00:49:08.362Z INFO StepRunner: Created Runner for step 7
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m closed_duration -reducer php -d memory_limit=-1 mapreduce.phar -r closed_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/closed_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-102328684JEXYWJIOS4/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-102328684JEXYWJIOS4
EMR_STEP_ID=s-102328684JEXYWJIOS4
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-102328684JEXYWJIOS4/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-102328684JEXYWJIOS4/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-102328684JEXYWJIOS4
INFO ProcessRunner started child process 27455
2024-09-17T00:49:08.363Z INFO HadoopJarStepRunner.Runner: startRun() called for s-102328684JEXYWJIOS4 Child Pid: 27455
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 78 seconds
2024-09-17T00:50:26.466Z INFO Step created jobs: job_1726532387964_0006
2024-09-17T00:50:26.466Z INFO Step succeeded with exitCode 0 and took 78 seconds


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-102328684JEXYWJIOS4/stderr.gz <==

Command exiting with ret '0'


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-102328684JEXYWJIOS4/stdout.gz <==

packageJobJar: [] [/usr/lib/hadoop/hadoop-streaming-3.3.6-amzn-1.jar] /mnt/var/lib/hadoop/steps/s-102328684JEXYWJIOS4/tmp/streamjob3233799428046431502.jar tmpDir=null


==> logs/MR_QAMR_2024-09-16-0/j-2P70ZOD5O7KZA/steps/s-102328684JEXYWJIOS4/syslog.gz <==

2024-09-17 00:49:12,011 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:49:12,128 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:49:12,154 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:49:12,154 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:49:12,373 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0006
2024-09-17 00:49:12,388 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:49:13,120 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:49:13,121 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:49:13,256 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:49:13,312 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:49:13,327 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:49:13,328 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:49:13,400 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0006
2024-09-17 00:49:13,400 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:49:13,524 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:49:13,524 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:49:13,566 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0006
2024-09-17 00:49:13,592 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0006/
2024-09-17 00:49:13,593 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0006
2024-09-17 00:49:18,642 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0006 running in uber mode : false
2024-09-17 00:49:18,643 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:49:31,732 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2024-09-17 00:49:32,736 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:49:42,782 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 0%
2024-09-17 00:49:43,786 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 0%
2024-09-17 00:49:44,790 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 0%
2024-09-17 00:49:53,826 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 0%
2024-09-17 00:49:54,829 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 0%
2024-09-17 00:49:55,833 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 0%
2024-09-17 00:49:56,837 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 1%
2024-09-17 00:49:57,841 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduce 2%
2024-09-17 00:49:58,844 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 8%
2024-09-17 00:49:59,848 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 14%
2024-09-17 00:50:00,852 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 16%
2024-09-17 00:50:03,862 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%
2024-09-17 00:50:04,866 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 17%
2024-09-17 00:50:05,869 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 17%
2024-09-17 00:50:06,873 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-17 00:50:07,876 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 17%
2024-09-17 00:50:09,882 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 18%
2024-09-17 00:50:10,886 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 19%
2024-09-17 00:50:11,889 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 20%
2024-09-17 00:50:12,893 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 20%
2024-09-17 00:50:13,896 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 20%
2024-09-17 00:50:14,901 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 20%
2024-09-17 00:50:15,904 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 20%
2024-09-17 00:50:16,907 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 25%
2024-09-17 00:50:17,910 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 47%
2024-09-17 00:50:18,914 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 82%
2024-09-17 00:50:19,917 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 93%
2024-09-17 00:50:20,919 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 95%
2024-09-17 00:50:21,923 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 96%
2024-09-17 00:50:22,926 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 97%
2024-09-17 00:50:23,929 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-17 00:50:24,936 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0006 completed successfully
2024-09-17 00:50:25,004 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=776735253
FILE: Number of bytes written=1721696237
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2577991235
S3: Number of bytes written=4631149
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Killed reduce tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=347228032
Total time spent by all reduces in occupied slots (ms)=210002880
Total time spent by all map tasks (ms)=2712719
Total time spent by all reduce tasks (ms)=3281295
Total vcore-milliseconds taken by all map tasks=2712719
Total vcore-milliseconds taken by all reduce tasks=3281295
Total megabyte-milliseconds taken by all map tasks=11111297024
Total megabyte-milliseconds taken by all reduce tasks=6720092160
Map-Reduce Framework
Map input records=6112743
Map output records=96274186
Map output bytes=6121578766
Map output materialized bytes=827746734
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=4688
Reduce shuffle bytes=827746734
Reduce input records=96274186
Reduce output records=4688
Spilled Records=192548372
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=79405
CPU time spent (ms)=2255890
Physical memory (bytes) snapshot=318272389120
Virtual memory (bytes) snapshot=1213320523776
Total committed heap usage (bytes)=293923717120
Peak Map Physical memory (bytes)=1215266816
Peak Map Virtual memory (bytes)=3051143168
Peak Reduce Physical memory (bytes)=1224445952
Peak Reduce Virtual memory (bytes)=3522666496
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2577991235
File Output Format Counters
Bytes Written=4631149
2024-09-17 00:50:25,004 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/closed_duration/


Loading…
Loading the web debug toolbar…
Attempt #