kettle报错,但是重新执行一遍又可以通过

0
每小时定时调度用户行为数据,是用azkaban调度的,经常半夜报错,但是早上重新执行命令进行调度后又可以成功通过不报错。很疑惑到底是什么情况。
 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - Starting job kettle_user_bhvr_ips at 1470596847997 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - azkaban.webserver.url property was not set 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - job JVM args: -Dazkaban.flowid=user_behavior_hourly -Dazkaban.execid=34638 -Dazkaban.jobid=kettle_user_bhvr_ips 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - Building command job executor. 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - 1 commands to execute. 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - effective user is: azkaban 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - Command: /appl/scripts/azkaban/kettle.sh --start 0000 --end 2359 --dir 55df1fec3861332f9228527b7d873f97 --param1 hadoop_path=/user/hadoop/tts_bi/behavior/_tmp_ --name 52.user_behavior 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - Environment variables: {JOB_OUTPUT_PROP_FILE=/appl/azkaban/azkaban-exec-server-3.0.0/executions/34638/kettle_user_bhvr_ips_output_8793921229245843691_tmp, JOB_PROP_FILE=/appl/azkaban/azkaban-exec-server-3.0.0/executions/34638/kettle_user_bhvr_ips_props_2398807630037483438_tmp, KRB5CCNAME=/tmp/krb5cc__bigdata_platform__user_behavior_hourly__kettle_user_bhvr_ips__34638__azkaban, JOB_NAME=kettle_user_bhvr_ips} 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - Working directory: /appl/azkaban/azkaban-exec-server-3.0.0/executions/34638 
08-08-2016 03:07:28 CST kettle_user_bhvr_ips INFO - Remaining arguments: 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 03:07:29,839 INFO [KarafInstance] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - ******************************************************************************* 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - *** Karaf Instance Number: 1 at /appl/kettle/data-integration/./system/kara *** 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - *** f/caches/default/data-1 *** 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - *** Karaf Port:8802 *** 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - *** 
OSGI Service Port:9051 *** 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - ******************************************************************************* 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 
03:07:29,839 INFO [KarafBoot] Checking to see if org.pentaho.clean.karaf.cache is enabled 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Aug 08, 2016 3:07:30 AM org.apache.karaf.main.Main$KarafLockCallback lockAquired 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - INFO: Lock acquired. Setting startlevel to 100 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:07:30 - Kitchen - Start of run. 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:07:30 - RepositoriesMeta - Reading repositories XML file: /appl/hadoop/.kettle/repositories.xml 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:07:31 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Aug 08, 2016 3:07:32 AM org.pentaho.caching.impl.PentahoCacheManagerFactory$RegistrationHandler$1 onSuccess 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - INFO: New Caching Service registered 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - ERROR: Bundle pentaho-big-data-api-runtimeTest [73] Error starting mvn:pentaho/pentaho-big-data-api-runtimeTest/6.1.0.1-196 (org.osgi.framework.BundleException: Unable to acquire global lock for resolve.) 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - org.osgi.framework.BundleException: Unable to acquire global lock for resolve. 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3980) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.apache.felix.framework.Felix.startBundle(Felix.java:2037) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304) 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at java.lang.Thread.run(Thread.java:745) 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - SLF4J: Class path contains multiple SLF4J bindings. 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - SLF4J: Found binding in [jar:file:/appl/kettle/data-integration/launcher/../lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - SLF4J: Found binding in [jar:file:/appl/kettle/data-integration/plugins/pentaho-big-data-plugin/lib/slf4j-log4j12-1.7.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 03:09:32,546 ERROR [KarafLifecycleListener] Error in Blueprint Watcher 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - org.pentaho.osgi.api.IKarafBlueprintWatcher$BlueprintWatcherException: Unknown error in KarafBlueprintWatcher 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.pentaho.osgi.impl.KarafBlueprintWatcherImpl.waitForBlueprint(KarafBlueprintWatcherImpl.java:103) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.pentaho.di.osgi.KarafLifecycleListener$2.run(KarafLifecycleListener.java:161) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at java.lang.Thread.run(Thread.java:745) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Caused by: org.pentaho.osgi.api.IKarafBlueprintWatcher$BlueprintWatcherException: Timed out waiting for blueprints to load: pentaho-big-data-api-runtimeTest,pentaho-big-data-kettle-plugins-common-named-cluster-bridge,pentaho-big-data-kettle-plugins-hdfs,pentaho-big-data-kettle-plugins-hbase,pentaho-big-data-kettle-plugins-mapreduce,pentaho-big-data-kettle-plugins-pig,pentaho-big-data-kettle-plugins-oozie,pentaho-big-data-kettle-plugins-sqoop,pentaho-big-data-impl-clusterTests,pentaho-big-data-impl-shim-shimTests 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - at org.pentaho.osgi.impl.KarafBlueprintWatcherImpl.waitForBlueprint(KarafBlueprintWatcherImpl.java:88) 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - ... 2 more 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:37 - 52.user_behavior - 开始执行任务 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:37 - 52.user_behavior - 开始项[12.user_bhvr_tr] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:37 - 12.user_bhvr_tr - Loading transformation from repository [12.user_bhvr_tr] in directory [/01.ODS_JOBS/TRANSF] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - 12.user_bhvr_tr - 为了转换解除补丁开始 [12.user_bhvr_tr] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - 表输出.0 - Connected to database [tts_ods@94] (commit=1000) 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - 52.user_behavior - 开始项[成功] 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - 52.user_behavior - 完成作业项[成功] (结果=[true]) 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - 52.user_behavior - 完成作业项[12.user_bhvr_tr] (结果=[true]) 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - 52.user_behavior - 任务执行完毕 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - Kitchen - Finished! 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - Kitchen - Start=2016/08/08 03:07:30.503, Stop=2016/08/08 03:09:38.485 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 2016/08/08 03:09:38 - Kitchen - Processing ended after 2 minutes and 7 seconds (127 seconds total). 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - 03:09:32,546 ERROR [KarafLifecycleListener] Error in Blueprint Watcher 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Job finished with errors. Exiting 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Process completed unsuccessfully in 130 seconds. 
08-08-2016 03:09:38 CST kettle_user_bhvr_ips ERROR - Job run failed! java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:165) at azkaban.execapp.JobRunner.runJob(JobRunner.java:590) at azkaban.execapp.JobRunner.run(JobRunner.java:443) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:131) at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:159) ... 7 more 08-08-2016 03:09:38 CST kettle_user_bhvr_ips ERROR - azkaban.jobExecutor.utils.process.ProcessFailureException cause: azkaban.jobExecutor.utils.process.ProcessFailureException 08-08-2016 03:09:38 CST kettle_user_bhvr_ips INFO - Finishing job kettle_user_bhvr_ips attempt: 0 at 1470596978938 with status FAILED
 
已邀请:
0

子建 - Kettle讲师、顾问、数据仓库架构 2016-08-08 回答

有可能是跑批时间点,比较集中,分配的内存不够,内存溢出异常
0

tony_yys 2016-09-13 回答

这个跟前边这些说的都没关系,估计你的数据源是MSSql的。或者你在中间有步骤是自定义常量之类的
[High]  High
Resolution:Fixed
Affects Version/s:6.0.1 GA, 6.1.0 GA
Fix Version/s:7.0.0 GA
Component/s:Core (Engine), Logging
Labels:

tech_groomed

Environment:
6.0.1.0 Build #383

Story Points:
0
Notice:
When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
Steps to Reproduce:

Open Spoon.
Open the attached job.
Right-click in Transformation job entry.
Select Open Referenced Object and click Transformation.
Check the console (spoon or command line) for any error.

OR

Start Spoon with debug log level

Description

Opening the attacted transformation from attached job, throw to Spoon console the following warnings:2015/11/27 17:19:37 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp 2015/11/27 17:19:37 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp 2015/11/27 17:19:37 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftp 2015/11/27 17:19:37 - cfgbuilder - Warning: The configuration parameter [org] is not supported by the default configuration builder for scheme: sftpAnd if we switch tabs, i.e., click in job then transformation, the warning message displays everytime we open the transformation.

要回复问题请先登录注册