https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=979 Summary: Runtime errors in SLURM provider are not halting run and getting sent to user Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: major Priority: P2 Component: Providers AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected], [email protected], [email protected] This might be the standard way all providers are running, but errors that the provider enounters in running jobs are only visible in the log and not brought to the user's attention on say stderr. For example, if all SLURM jobs are failing because the user has set an invalid reservation to use, then Swift just keeps indicating lack of progress, but no jobs are submitted and not error is made visible to the user. The log is below. Minimally, errors of this type should make it to the progress log, where the user could see them and adress the problem. --- 2013-04-10 03:36:42,200+0000 INFO Block Block task status changed: Submitting 2013-04-10 03:36:42,246+0000 DEBUG AbstractExecutor Writing Slurm script to /home/wilde/.globus/scripts/Slurm1915108282528394750.sub\ mit 2013-04-10 03:36:42,278+0000 DEBUG AbstractExecutor Wrote Slurm script to /home/wilde/.globus/scripts/Slurm1915108282528394750.submi\ t 2013-04-10 03:36:42,278+0000 DEBUG AbstractExecutor Command line: sbatch /home/wilde/.globus/scripts/Slurm1915108282528394750.submit 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Waiting for output from sbatch 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Output from sbatch is: "" 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Waiting for output from sbatch 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Output from sbatch is: "sbatch: error: Batch job submission failed: Requested re\ servation is invalid" 2013-04-10 03:36:42,325+0000 INFO BlockTaskSubmitter Error submitting block task org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Cannot submit job: Could not submit job (sbatch reported an exi\ t code of 1). sbatch: error: Batch job submission failed: Requested reservation is invalid at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler\ .java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:61) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:70) Caused by: org.globus.cog.abstraction.impl.scheduler.common.ProcessException: Could not submit job (sbatch reported an exit code of \ 1). sbatch: error: Batch job submission failed: Requested reservation is invalid at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:108) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler\ .java:53) ... 3 more 2013-04-10 03:36:52,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:02,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:11,711+0000 INFO RuntimeStats$ProgressTicker Submitted:10 2013-04-10 03:37:12,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:22,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:32,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:35,894+0000 INFO ServiceManager Cleaning up... 2013-04-10 03:37:35,894+0000 INFO ServiceManager Shutting down service at https://10.50.180.71:50002 2013-04-10 03:37:35,895+0000 INFO RequestHandler Handler(tag: 12, SHUTDOWNSERVICE) unregistering (send) 2013-04-10 03:37:35,996+0000 INFO RequestHandler Handler(tag: 3, UNREGISTER) unregistering (send) 2013-04-10 03:37:36,019+0000 INFO BlockQueueProcessor Shutting down blocks 2013-04-10 03:37:36,019+0000 INFO Block Shutting down block Block 0410-3603420-000000 (48x1740.000s) 2013-04-10 03:37:36,020+0000 INFO Block Block Block 0410-3603420-000000 (48x1740.000s) not running. Cancelling job. 2013-04-10 03:37:36,020+0000 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0410-3603420-000000 2013-04-10 03:37:36,020+0000 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send) 2013-04-10 03:37:36,020+0000 WARN Block Failed to shut down block: Block 0410-3603420-000000 (48x1740.000s) org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Can only cancel an active task at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:196) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler\ .java:85) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.