2024-09-17T00:25:43.100Z INFO Ensure step 4 jar file command-runner.jar
2024-09-17T00:25:43.100Z INFO StepRunner: Created Runner for step 4
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-00921997DJQT4WBYAKL/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-00921997DJQT4WBYAKL
EMR_STEP_ID=s-00921997DJQT4WBYAKL
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-00921997DJQT4WBYAKL/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-00921997DJQT4WBYAKL/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-00921997DJQT4WBYAKL
INFO ProcessRunner started child process 19758
2024-09-17T00:25:43.102Z INFO HadoopJarStepRunner.Runner: startRun() called for s-00921997DJQT4WBYAKL Child Pid: 19758
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 92 seconds
2024-09-17T00:27:15.168Z INFO Step created jobs: job_1726532387964_0003
2024-09-17T00:27:15.168Z INFO Step succeeded with exitCode 0 and took 92 seconds
2024-09-17 00:25:46,561 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:25:46,673 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:25:46,702 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:25:46,703 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:25:46,920 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0003
2024-09-17 00:25:46,936 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:25:47,273 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:25:47,274 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:25:47,393 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:25:47,465 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:25:47,485 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:25:47,486 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:25:47,569 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0003
2024-09-17 00:25:47,569 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:25:47,719 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:25:47,719 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:25:47,768 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0003
2024-09-17 00:25:47,797 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0003/
2024-09-17 00:25:47,798 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0003
2024-09-17 00:25:51,844 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0003 running in uber mode : false
2024-09-17 00:25:51,845 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:26:07,941 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2024-09-17 00:26:08,945 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2024-09-17 00:26:09,952 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:26:23,001 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 0%
2024-09-17 00:26:24,005 INFO org.apache.hadoop.mapreduce.Job (main): map 52% reduce 0%
2024-09-17 00:26:25,008 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 0%
2024-09-17 00:26:38,051 INFO org.apache.hadoop.mapreduce.Job (main): map 60% reduce 2%
2024-09-17 00:26:39,055 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 5%
2024-09-17 00:26:40,058 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 9%
2024-09-17 00:26:41,062 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 13%
2024-09-17 00:26:42,066 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 13%
2024-09-17 00:26:43,069 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 13%
2024-09-17 00:26:44,072 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 14%
2024-09-17 00:26:45,074 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 15%
2024-09-17 00:26:46,077 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 16%
2024-09-17 00:26:53,097 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%
2024-09-17 00:26:54,099 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 17%
2024-09-17 00:26:55,103 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 17%
2024-09-17 00:26:56,106 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 18%
2024-09-17 00:26:57,109 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 18%
2024-09-17 00:26:58,113 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 19%
2024-09-17 00:26:59,123 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 20%
2024-09-17 00:27:02,131 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 21%
2024-09-17 00:27:03,134 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 21%
2024-09-17 00:27:04,137 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 21%
2024-09-17 00:27:05,140 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 21%
2024-09-17 00:27:06,143 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 21%
2024-09-17 00:27:08,148 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 29%
2024-09-17 00:27:09,153 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 77%
2024-09-17 00:27:10,157 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 96%
2024-09-17 00:27:11,160 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-17 00:27:13,170 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0003 completed successfully
2024-09-17 00:27:13,235 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=129628173
FILE: Number of bytes written=390021782
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578056797
S3: Number of bytes written=2984089
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=480734848
Total time spent by all reduces in occupied slots (ms)=267831360
Total time spent by all map tasks (ms)=3755741
Total time spent by all reduce tasks (ms)=4184865
Total vcore-milliseconds taken by all map tasks=3755741
Total vcore-milliseconds taken by all reduce tasks=4184865
Total megabyte-milliseconds taken by all map tasks=15383515136
Total megabyte-milliseconds taken by all reduce tasks=8570603520
Map-Reduce Framework
Map input records=6112743
Map output records=61682
Map output bytes=464092492
Map output materialized bytes=143178174
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=1004
Reduce shuffle bytes=143178174
Reduce input records=61682
Reduce output records=1004
Spilled Records=123364
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=77419
CPU time spent (ms)=3055450
Physical memory (bytes) snapshot=310040559616
Virtual memory (bytes) snapshot=1212113440768
Total committed heap usage (bytes)=293343330304
Peak Map Physical memory (bytes)=1368006656
Peak Map Virtual memory (bytes)=3481489408
Peak Reduce Physical memory (bytes)=709853184
Peak Reduce Virtual memory (bytes)=3480350720
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2578056797
File Output Format Counters
Bytes Written=2984089
2024-09-17 00:27:13,235 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/active_job_count/
2024-09-17T00:23:40.022Z INFO Ensure step 3 jar file command-runner.jar
2024-09-17T00:23:40.023Z INFO StepRunner: Created Runner for step 3
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m created_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r created_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/created_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-05941293PWESHPWTN3CF/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-05941293PWESHPWTN3CF
EMR_STEP_ID=s-05941293PWESHPWTN3CF
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-05941293PWESHPWTN3CF/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-05941293PWESHPWTN3CF/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-05941293PWESHPWTN3CF
INFO ProcessRunner started child process 5213
2024-09-17T00:23:40.024Z INFO HadoopJarStepRunner.Runner: startRun() called for s-05941293PWESHPWTN3CF Child Pid: 5213
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 118 seconds
2024-09-17T00:25:38.096Z INFO Step created jobs: job_1726532387964_0002
2024-09-17T00:25:38.097Z INFO Step succeeded with exitCode 0 and took 118 seconds
2024-09-17T00:21:20.786Z INFO Ensure step 2 jar file command-runner.jar
2024-09-17T00:21:20.787Z INFO StepRunner: Created Runner for step 2
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m deleted_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r deleted_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/deleted_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-07842242O63ANN00L24/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-07842242O63ANN00L24
EMR_STEP_ID=s-07842242O63ANN00L24
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-07842242O63ANN00L24/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-07842242O63ANN00L24/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-07842242O63ANN00L24
INFO ProcessRunner started child process 21723
2024-09-17T00:21:20.788Z INFO HadoopJarStepRunner.Runner: startRun() called for s-07842242O63ANN00L24 Child Pid: 21723
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 134 seconds
2024-09-17T00:23:35.017Z INFO Step created jobs: job_1726532387964_0001
2024-09-17T00:23:35.017Z INFO Step succeeded with exitCode 0 and took 134 seconds
2024-09-17T00:27:20.171Z INFO Ensure step 5 jar file command-runner.jar
2024-09-17T00:27:20.172Z INFO StepRunner: Created Runner for step 5
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m unique_active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r unique_active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/unique_active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-09228062W8QFDT2FMM96/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-09228062W8QFDT2FMM96
EMR_STEP_ID=s-09228062W8QFDT2FMM96
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-09228062W8QFDT2FMM96/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-09228062W8QFDT2FMM96/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-09228062W8QFDT2FMM96
INFO ProcessRunner started child process 32054
2024-09-17T00:27:20.174Z INFO HadoopJarStepRunner.Runner: startRun() called for s-09228062W8QFDT2FMM96 Child Pid: 32054
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 572 seconds
2024-09-17T00:36:52.266Z INFO Step created jobs: job_1726532387964_0004
2024-09-17T00:36:52.266Z INFO Step succeeded with exitCode 0 and took 572 seconds
2024-09-17T00:21:13.488Z INFO Ensure step 1 jar file command-runner.jar
2024-09-17T00:21:13.488Z INFO StepRunner: Created Runner for step 1
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar /bin/bash -c echo Debugging step is a no-op'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-099979118CX8O5WOYZJA/tmp
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-099979118CX8O5WOYZJA
EMR_STEP_ID=s-099979118CX8O5WOYZJA
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-099979118CX8O5WOYZJA/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-099979118CX8O5WOYZJA/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-099979118CX8O5WOYZJA
INFO ProcessRunner started child process 21494
2024-09-17T00:21:13.491Z INFO HadoopJarStepRunner.Runner: startRun() called for s-099979118CX8O5WOYZJA Child Pid: 21494
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 2 seconds
2024-09-17T00:21:15.781Z INFO Step created jobs:
2024-09-17T00:21:15.782Z INFO Step succeeded with exitCode 0 and took 2 seconds
2024-09-17T00:36:57.269Z INFO Ensure step 6 jar file command-runner.jar
2024-09-17T00:36:57.269Z INFO StepRunner: Created Runner for step 6
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m active_duration -reducer php -d memory_limit=-1 mapreduce.phar -r active_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/active_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-10122202EESN0U3GC6D1/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-10122202EESN0U3GC6D1
EMR_STEP_ID=s-10122202EESN0U3GC6D1
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-10122202EESN0U3GC6D1/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-10122202EESN0U3GC6D1/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-10122202EESN0U3GC6D1
INFO ProcessRunner started child process 23264
2024-09-17T00:36:57.270Z INFO HadoopJarStepRunner.Runner: startRun() called for s-10122202EESN0U3GC6D1 Child Pid: 23264
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 726 seconds
2024-09-17T00:49:03.359Z INFO Step created jobs: job_1726532387964_0005
2024-09-17T00:49:03.359Z INFO Step succeeded with exitCode 0 and took 726 seconds
2024-09-17T00:49:08.362Z INFO Ensure step 7 jar file command-runner.jar
2024-09-17T00:49:08.362Z INFO StepRunner: Created Runner for step 7
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-16 -m closed_duration -reducer php -d memory_limit=-1 mapreduce.phar -r closed_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-16/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/closed_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-102328684JEXYWJIOS4/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-47
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-102328684JEXYWJIOS4
EMR_STEP_ID=s-102328684JEXYWJIOS4
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-102328684JEXYWJIOS4/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-102328684JEXYWJIOS4/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-102328684JEXYWJIOS4
INFO ProcessRunner started child process 27455
2024-09-17T00:49:08.363Z INFO HadoopJarStepRunner.Runner: startRun() called for s-102328684JEXYWJIOS4 Child Pid: 27455
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 78 seconds
2024-09-17T00:50:26.466Z INFO Step created jobs: job_1726532387964_0006
2024-09-17T00:50:26.466Z INFO Step succeeded with exitCode 0 and took 78 seconds
2024-09-17 00:49:12,011 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:49:12,128 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:49:12,154 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-47.ec2.internal/10.0.0.47:8032
2024-09-17 00:49:12,154 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-47.ec2.internal/10.0.0.47:10200
2024-09-17 00:49:12,373 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726532387964_0006
2024-09-17 00:49:12,388 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-16-0/phars/mapreduce.phar' for reading
2024-09-17 00:49:13,120 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-17 00:49:13,121 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-17 00:49:13,256 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-17 00:49:13,312 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-17 00:49:13,327 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-17 00:49:13,328 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-17 00:49:13,400 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726532387964_0006
2024-09-17 00:49:13,400 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-17 00:49:13,524 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-17 00:49:13,524 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-17 00:49:13,566 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726532387964_0006
2024-09-17 00:49:13,592 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-47.ec2.internal:20888/proxy/application_1726532387964_0006/
2024-09-17 00:49:13,593 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726532387964_0006
2024-09-17 00:49:18,642 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0006 running in uber mode : false
2024-09-17 00:49:18,643 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-17 00:49:31,732 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2024-09-17 00:49:32,736 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-17 00:49:42,782 INFO org.apache.hadoop.mapreduce.Job (main): map 34% reduce 0%
2024-09-17 00:49:43,786 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 0%
2024-09-17 00:49:44,790 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 0%
2024-09-17 00:49:53,826 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 0%
2024-09-17 00:49:54,829 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 0%
2024-09-17 00:49:55,833 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 0%
2024-09-17 00:49:56,837 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 1%
2024-09-17 00:49:57,841 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduce 2%
2024-09-17 00:49:58,844 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 8%
2024-09-17 00:49:59,848 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 14%
2024-09-17 00:50:00,852 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 16%
2024-09-17 00:50:03,862 INFO org.apache.hadoop.mapreduce.Job (main): map 77% reduce 16%
2024-09-17 00:50:04,866 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 17%
2024-09-17 00:50:05,869 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 17%
2024-09-17 00:50:06,873 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-17 00:50:07,876 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 17%
2024-09-17 00:50:09,882 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 18%
2024-09-17 00:50:10,886 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 19%
2024-09-17 00:50:11,889 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 20%
2024-09-17 00:50:12,893 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 20%
2024-09-17 00:50:13,896 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 20%
2024-09-17 00:50:14,901 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 20%
2024-09-17 00:50:15,904 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 20%
2024-09-17 00:50:16,907 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 25%
2024-09-17 00:50:17,910 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 47%
2024-09-17 00:50:18,914 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 82%
2024-09-17 00:50:19,917 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 93%
2024-09-17 00:50:20,919 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 95%
2024-09-17 00:50:21,923 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 96%
2024-09-17 00:50:22,926 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 97%
2024-09-17 00:50:23,929 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-17 00:50:24,936 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726532387964_0006 completed successfully
2024-09-17 00:50:25,004 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=776735253
FILE: Number of bytes written=1721696237
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2577991235
S3: Number of bytes written=4631149
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Killed reduce tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=347228032
Total time spent by all reduces in occupied slots (ms)=210002880
Total time spent by all map tasks (ms)=2712719
Total time spent by all reduce tasks (ms)=3281295
Total vcore-milliseconds taken by all map tasks=2712719
Total vcore-milliseconds taken by all reduce tasks=3281295
Total megabyte-milliseconds taken by all map tasks=11111297024
Total megabyte-milliseconds taken by all reduce tasks=6720092160
Map-Reduce Framework
Map input records=6112743
Map output records=96274186
Map output bytes=6121578766
Map output materialized bytes=827746734
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=4688
Reduce shuffle bytes=827746734
Reduce input records=96274186
Reduce output records=4688
Spilled Records=192548372
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=79405
CPU time spent (ms)=2255890
Physical memory (bytes) snapshot=318272389120
Virtual memory (bytes) snapshot=1213320523776
Total committed heap usage (bytes)=293923717120
Peak Map Physical memory (bytes)=1215266816
Peak Map Virtual memory (bytes)=3051143168
Peak Reduce Physical memory (bytes)=1224445952
Peak Reduce Virtual memory (bytes)=3522666496
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2577991235
File Output Format Counters
Bytes Written=4631149
2024-09-17 00:50:25,004 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-16/closed_duration/