2024-09-19T00:27:39.975Z INFO Ensure step 5 jar file command-runner.jar
2024-09-19T00:27:39.975Z INFO StepRunner: Created Runner for step 5
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m unique_active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r unique_active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/unique_active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-0152848G0BFCVCU18QK/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-0152848G0BFCVCU18QK
EMR_STEP_ID=s-0152848G0BFCVCU18QK
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-0152848G0BFCVCU18QK/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-0152848G0BFCVCU18QK/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-0152848G0BFCVCU18QK
INFO ProcessRunner started child process 31943
2024-09-19T00:27:39.976Z INFO HadoopJarStepRunner.Runner: startRun() called for s-0152848G0BFCVCU18QK Child Pid: 31943
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 578 seconds
2024-09-19T00:37:18.077Z INFO Step created jobs: job_1726705207091_0004
2024-09-19T00:37:18.077Z INFO Step succeeded with exitCode 0 and took 578 seconds
2024-09-19T00:21:36.546Z INFO Ensure step 2 jar file command-runner.jar
2024-09-19T00:21:36.546Z INFO StepRunner: Created Runner for step 2
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m deleted_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r deleted_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/deleted_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-0387261USN6O8QK6W9T/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-0387261USN6O8QK6W9T
EMR_STEP_ID=s-0387261USN6O8QK6W9T
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-0387261USN6O8QK6W9T/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-0387261USN6O8QK6W9T/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-0387261USN6O8QK6W9T
INFO ProcessRunner started child process 21764
2024-09-19T00:21:36.547Z INFO HadoopJarStepRunner.Runner: startRun() called for s-0387261USN6O8QK6W9T Child Pid: 21764
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 138 seconds
2024-09-19T00:23:54.779Z INFO Step created jobs: job_1726705207091_0001
2024-09-19T00:23:54.779Z INFO Step succeeded with exitCode 0 and took 138 seconds
2024-09-19T00:37:23.080Z INFO Ensure step 6 jar file command-runner.jar
2024-09-19T00:37:23.080Z INFO StepRunner: Created Runner for step 6
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m active_duration -reducer php -d memory_limit=-1 mapreduce.phar -r active_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/active_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-05070263163YN5L3C450/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-05070263163YN5L3C450
EMR_STEP_ID=s-05070263163YN5L3C450
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-05070263163YN5L3C450/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-05070263163YN5L3C450/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-05070263163YN5L3C450
INFO ProcessRunner started child process 25523
2024-09-19T00:37:23.081Z INFO HadoopJarStepRunner.Runner: startRun() called for s-05070263163YN5L3C450 Child Pid: 25523
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 732 seconds
2024-09-19T00:49:35.242Z INFO Step created jobs: job_1726705207091_0005
2024-09-19T00:49:35.242Z INFO Step succeeded with exitCode 0 and took 732 seconds
2024-09-19T00:49:40.245Z INFO Ensure step 7 jar file command-runner.jar
2024-09-19T00:49:40.245Z INFO StepRunner: Created Runner for step 7
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m closed_duration -reducer php -d memory_limit=-1 mapreduce.phar -r closed_duration -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/closed_duration/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-05135142OS6GQU0ETXNY/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-05135142OS6GQU0ETXNY
EMR_STEP_ID=s-05135142OS6GQU0ETXNY
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-05135142OS6GQU0ETXNY/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-05135142OS6GQU0ETXNY/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-05135142OS6GQU0ETXNY
INFO ProcessRunner started child process 30114
2024-09-19T00:49:40.247Z INFO HadoopJarStepRunner.Runner: startRun() called for s-05135142OS6GQU0ETXNY Child Pid: 30114
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 78 seconds
2024-09-19T00:50:58.305Z INFO Step created jobs: job_1726705207091_0006
2024-09-19T00:50:58.305Z INFO Step succeeded with exitCode 0 and took 78 seconds
2024-09-19 00:49:44,102 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:49:44,225 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:49:44,256 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:49:44,256 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:49:44,508 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0006
2024-09-19 00:49:44,528 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:49:45,005 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:49:45,007 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:49:45,163 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:49:45,231 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:49:45,245 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:49:45,246 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:49:45,336 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0006
2024-09-19 00:49:45,336 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:49:45,494 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:49:45,494 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:49:45,549 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0006
2024-09-19 00:49:45,612 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0006/
2024-09-19 00:49:45,613 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0006
2024-09-19 00:49:50,703 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0006 running in uber mode : false
2024-09-19 00:49:50,704 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:50:03,785 INFO org.apache.hadoop.mapreduce.Job (main): map 12% reduce 0%
2024-09-19 00:50:04,789 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:50:14,830 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2024-09-19 00:50:15,833 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 0%
2024-09-19 00:50:16,836 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 0%
2024-09-19 00:50:20,852 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 2%
2024-09-19 00:50:26,869 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 2%
2024-09-19 00:50:27,872 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 2%
2024-09-19 00:50:28,875 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 3%
2024-09-19 00:50:29,878 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 4%
2024-09-19 00:50:30,881 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 8%
2024-09-19 00:50:31,883 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 12%
2024-09-19 00:50:32,886 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 16%
2024-09-19 00:50:33,889 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 17%
2024-09-19 00:50:35,895 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 17%
2024-09-19 00:50:36,898 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 17%
2024-09-19 00:50:37,901 INFO org.apache.hadoop.mapreduce.Job (main): map 82% reduce 17%
2024-09-19 00:50:38,903 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 18%
2024-09-19 00:50:39,906 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 18%
2024-09-19 00:50:40,909 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 18%
2024-09-19 00:50:42,915 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 19%
2024-09-19 00:50:43,918 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 20%
2024-09-19 00:50:44,921 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 20%
2024-09-19 00:50:45,924 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 20%
2024-09-19 00:50:46,928 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 20%
2024-09-19 00:50:47,931 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 26%
2024-09-19 00:50:48,933 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 53%
2024-09-19 00:50:49,936 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 80%
2024-09-19 00:50:50,939 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 95%
2024-09-19 00:50:51,941 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 96%
2024-09-19 00:50:52,944 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 97%
2024-09-19 00:50:53,948 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 98%
2024-09-19 00:50:54,951 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 99%
2024-09-19 00:50:55,953 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-19 00:50:56,959 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0006 completed successfully
2024-09-19 00:50:57,033 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=776745478
FILE: Number of bytes written=1721713967
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2577967322
S3: Number of bytes written=4631219
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Killed reduce tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=352661632
Total time spent by all reduces in occupied slots (ms)=210428736
Total time spent by all map tasks (ms)=2755169
Total time spent by all reduce tasks (ms)=3287949
Total vcore-milliseconds taken by all map tasks=2755169
Total vcore-milliseconds taken by all reduce tasks=3287949
Total megabyte-milliseconds taken by all map tasks=11285172224
Total megabyte-milliseconds taken by all reduce tasks=6733719552
Map-Reduce Framework
Map input records=6112743
Map output records=96274186
Map output bytes=6121578766
Map output materialized bytes=827746734
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=4688
Reduce shuffle bytes=827746734
Reduce input records=96274186
Reduce output records=4688
Spilled Records=192548372
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=85565
CPU time spent (ms)=2261970
Physical memory (bytes) snapshot=314649182208
Virtual memory (bytes) snapshot=1213403320320
Total committed heap usage (bytes)=289958526976
Peak Map Physical memory (bytes)=1204535296
Peak Map Virtual memory (bytes)=3045826560
Peak Reduce Physical memory (bytes)=1189134336
Peak Reduce Virtual memory (bytes)=3519500288
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2577967322
File Output Format Counters
Bytes Written=4631219
2024-09-19 00:50:57,033 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/closed_duration/
2024-09-19T00:23:59.785Z INFO Ensure step 3 jar file command-runner.jar
2024-09-19T00:23:59.785Z INFO StepRunner: Created Runner for step 3
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m created_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r created_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/created_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-0607339S1ENRUF5MJD6/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-0607339S1ENRUF5MJD6
EMR_STEP_ID=s-0607339S1ENRUF5MJD6
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-0607339S1ENRUF5MJD6/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-0607339S1ENRUF5MJD6/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-0607339S1ENRUF5MJD6
INFO ProcessRunner started child process 6828
2024-09-19T00:23:59.786Z INFO HadoopJarStepRunner.Runner: startRun() called for s-0607339S1ENRUF5MJD6 Child Pid: 6828
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 118 seconds
2024-09-19T00:25:57.879Z INFO Step created jobs: job_1726705207091_0002
2024-09-19T00:25:57.879Z INFO Step succeeded with exitCode 0 and took 118 seconds
2024-09-19T00:26:02.883Z INFO Ensure step 4 jar file command-runner.jar
2024-09-19T00:26:02.884Z INFO StepRunner: Created Runner for step 4
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar hadoop-streaming -files s3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar -D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec -D stream.map.output.field.separator=, -D stream.reduce.input.field.separator=, -D stream.reduce.output.field.separator=, -D mapreduce.output.textoutputformat.separator=, -mapper php -d memory_limit=3072M mapreduce.phar --process_day 2024-09-18 -m active_job_count -reducer php -d memory_limit=-1 mapreduce.phar -r active_job_count -input s3://lu-market-reports-emr-qa/input/mysql/2024-09-18/ -output s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/active_job_count/'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-06142593L2643XPDLWT1/tmp -D=
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-06142593L2643XPDLWT1
EMR_STEP_ID=s-06142593L2643XPDLWT1
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-06142593L2643XPDLWT1/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-06142593L2643XPDLWT1/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-06142593L2643XPDLWT1
INFO ProcessRunner started child process 20521
2024-09-19T00:26:02.885Z INFO HadoopJarStepRunner.Runner: startRun() called for s-06142593L2643XPDLWT1 Child Pid: 20521
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 92 seconds
2024-09-19T00:27:34.971Z INFO Step created jobs: job_1726705207091_0003
2024-09-19T00:27:34.971Z INFO Step succeeded with exitCode 0 and took 92 seconds
2024-09-19 00:26:07,352 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:26:07,462 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:26:07,489 INFO org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider (main): Connecting to ResourceManager at ip-10-0-0-206.ec2.internal/10.0.0.206:8032
2024-09-19 00:26:07,489 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-10-0-0-206.ec2.internal/10.0.0.206:10200
2024-09-19 00:26:07,737 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1726705207091_0003
2024-09-19 00:26:07,755 INFO com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem (main): Opening 's3://lu-market-reports-emr-qa/cluster_config/mysql/2024-09-18-0/phars/mapreduce.phar' for reading
2024-09-19 00:26:08,222 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2024-09-19 00:26:08,224 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2024-09-19 00:26:08,352 INFO org.apache.hadoop.mapred.FileInputFormat (main): Total input files to process : 25
2024-09-19 00:26:08,425 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:269
2024-09-19 00:26:08,442 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2024-09-19 00:26:08,443 INFO org.apache.hadoop.conf.Configuration.deprecation (main): mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2024-09-19 00:26:08,525 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1726705207091_0003
2024-09-19 00:26:08,525 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2024-09-19 00:26:08,674 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2024-09-19 00:26:08,675 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2024-09-19 00:26:08,732 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1726705207091_0003
2024-09-19 00:26:08,783 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-10-0-0-206.ec2.internal:20888/proxy/application_1726705207091_0003/
2024-09-19 00:26:08,784 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1726705207091_0003
2024-09-19 00:26:12,842 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0003 running in uber mode : false
2024-09-19 00:26:12,843 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2024-09-19 00:26:29,931 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2024-09-19 00:26:30,935 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2024-09-19 00:26:43,989 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2024-09-19 00:26:44,993 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2024-09-19 00:26:45,996 INFO org.apache.hadoop.mapreduce.Job (main): map 57% reduce 0%
2024-09-19 00:26:47,000 INFO org.apache.hadoop.mapreduce.Job (main): map 59% reduce 1%
2024-09-19 00:27:00,043 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 3%
2024-09-19 00:27:01,046 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 6%
2024-09-19 00:27:02,049 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 11%
2024-09-19 00:27:03,052 INFO org.apache.hadoop.mapreduce.Job (main): map 73% reduce 14%
2024-09-19 00:27:04,056 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 14%
2024-09-19 00:27:06,061 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 15%
2024-09-19 00:27:07,064 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 16%
2024-09-19 00:27:09,069 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 17%
2024-09-19 00:27:15,086 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 17%
2024-09-19 00:27:16,089 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 17%
2024-09-19 00:27:17,093 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 17%
2024-09-19 00:27:18,096 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 17%
2024-09-19 00:27:19,099 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 18%
2024-09-19 00:27:20,102 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 19%
2024-09-19 00:27:21,104 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 20%
2024-09-19 00:27:25,119 INFO org.apache.hadoop.mapreduce.Job (main): map 91% reduce 20%
2024-09-19 00:27:26,122 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 20%
2024-09-19 00:27:27,125 INFO org.apache.hadoop.mapreduce.Job (main): map 98% reduce 20%
2024-09-19 00:27:28,128 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 20%
2024-09-19 00:27:29,130 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 21%
2024-09-19 00:27:30,134 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 36%
2024-09-19 00:27:31,136 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 89%
2024-09-19 00:27:32,139 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 97%
2024-09-19 00:27:33,142 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 99%
2024-09-19 00:27:34,145 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2024-09-19 00:27:34,150 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1726705207091_0003 completed successfully
2024-09-19 00:27:34,215 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 60
File System Counters
FILE: Number of bytes read=129700578
FILE: Number of bytes written=390165697
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=35777
HDFS: Number of bytes written=0
HDFS: Number of read operations=269
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3: Number of bytes read=2578094550
S3: Number of bytes written=2985205
S3: Number of read operations=0
S3: Number of large read operations=0
S3: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=269
Launched reduce tasks=126
Data-local map tasks=269
Total time spent by all maps in occupied slots (ms)=486137984
Total time spent by all reduces in occupied slots (ms)=271101056
Total time spent by all map tasks (ms)=3797953
Total time spent by all reduce tasks (ms)=4235954
Total vcore-milliseconds taken by all map tasks=3797953
Total vcore-milliseconds taken by all reduce tasks=4235954
Total megabyte-milliseconds taken by all map tasks=15556415488
Total megabyte-milliseconds taken by all reduce tasks=8675233792
Map-Reduce Framework
Map input records=6112743
Map output records=61682
Map output bytes=464325959
Map output materialized bytes=143242179
Input split bytes=35777
Combine input records=0
Combine output records=0
Reduce input groups=1004
Reduce shuffle bytes=143242179
Reduce input records=61682
Reduce output records=1004
Spilled Records=123364
Shuffled Maps =33894
Failed Shuffles=0
Merged Map outputs=33894
GC time elapsed (ms)=80621
CPU time spent (ms)=3203730
Physical memory (bytes) snapshot=309080395776
Virtual memory (bytes) snapshot=1211973513216
Total committed heap usage (bytes)=293059690496
Peak Map Physical memory (bytes)=1360744448
Peak Map Virtual memory (bytes)=3500429312
Peak Reduce Physical memory (bytes)=704131072
Peak Reduce Virtual memory (bytes)=3457339392
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2578094550
File Output Format Counters
Bytes Written=2985205
2024-09-19 00:27:34,216 INFO org.apache.hadoop.streaming.StreamJob (main): Output directory: s3://lu-market-reports-emr-qa/output/mysql/2024-09-18/active_job_count/
2024-09-19T00:21:29.219Z INFO Ensure step 1 jar file command-runner.jar
2024-09-19T00:21:29.219Z INFO StepRunner: Created Runner for step 1
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar /bin/bash -c echo Debugging step is a no-op'
INFO Environment:
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/aws/puppet/bin/
SECURITY_PROPERTIES=/emr/instance-controller/lib/security.properties
HISTCONTROL=ignoredups
HISTSIZE=1000
HADOOP_ROOT_LOGGER=INFO,DRFA
JAVA_HOME=/etc/alternatives/jre
AWS_DEFAULT_REGION=us-east-1
LANG=en_US.UTF-8
MAIL=/var/spool/mail/hadoop
LOGNAME=hadoop
PWD=/
HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-064464536ZPQ5S556AL6/tmp
_=/etc/alternatives/jre/bin/java
LESSOPEN=||/usr/bin/lesspipe.sh %s
SHELL=/bin/bash
QTINC=/usr/lib64/qt-3.3/include
USER=hadoop
HADOOP_LOGFILE=syslog
HOSTNAME=ip-10-0-0-206
QTDIR=/usr/lib64/qt-3.3
HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-064464536ZPQ5S556AL6
EMR_STEP_ID=s-064464536ZPQ5S556AL6
QTLIB=/usr/lib64/qt-3.3/lib
HOME=/home/hadoop
SHLVL=1
HADOOP_IDENT_STRING=hadoop
INFO redirectOutput to /mnt/var/log/hadoop/steps/s-064464536ZPQ5S556AL6/stdout
INFO redirectError to /mnt/var/log/hadoop/steps/s-064464536ZPQ5S556AL6/stderr
INFO Working dir /mnt/var/lib/hadoop/steps/s-064464536ZPQ5S556AL6
INFO ProcessRunner started child process 21528
2024-09-19T00:21:29.221Z INFO HadoopJarStepRunner.Runner: startRun() called for s-064464536ZPQ5S556AL6 Child Pid: 21528
INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
INFO total process run time: 2 seconds
2024-09-19T00:21:31.541Z INFO Step created jobs:
2024-09-19T00:21:31.541Z INFO Step succeeded with exitCode 0 and took 2 seconds