how to ssh to a running FireCloud job node?

bhaasbhaas Broad InstituteMember, Broadie

Is it possible to ssh into a node that's running a workflow?

I've got ~500 jobs running and most are taking a lot longer than I expected. I'd like to check on a few of them to see where they're at in the workflow. It seems the stderr and stdout from the firecloud gui can only be viewed once the job has finished.

Issue · Github
by Geraldine_VdAuwera

Issue Number
2375
State
closed
Last Updated
Assignee
Array
Milestone
Array
Closed By
vdauwera

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Good question -- I believe it's possible through the Google console but I can't seem to find the right place to look. Will consult the team and get back to you.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    And the word is: it's not currently possible to ssh into VMs controlled through the Google Pipelines API that FireCloud uses under the hood, in part for security reasons. I was thinking of the case where you create your own VMs more directly. Sorry to disappoint...

  • bhaasbhaas Broad InstituteMember, Broadie

    Thanks, Geraldine. It does appear that most of my jobs have finished from the sample-set run, but the firecloud monitoring stats aren't updating in the gui, so I'm not sure how many of the 550 total jobs are actually done vs. still running. Most that I click on have final results stored in the google bucket. Are there other ways I can check into the status of my jobs?

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    If you click on the call operations ID, the detailed status of the job (retrieved directly from the Google Pipelines API) will be displayed.

    -Chet

  • bhaasbhaas Broad InstituteMember, Broadie

    Thanks, Chet. Where is the call operations ID? The submission ID for the job brings me to the bucket, but I'm not finding a call operations ID.

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    When you expand a call by clicking on "show" next to the call name, you'll see something like this:

    Calls:
    HaplotypeCallerGvcfScatterWf.MergeVCFs Hide
    Call #1:
    Operation:operations/EI79yL3PKxjO0M6Y5K7o-dQBIJGfndXTDioPcHJvZHVjdGlvblF1ZXVl
    Status:Done
    Started:June 30, 2017, 2:31 AM (1 month ago)
    Ended:June 30, 2017, 2:33 AM (1 month ago)
    Inputs:Show
    Outputs:Show
    stdout:MergeVCFs-stdout.log
    stderr:MergeVCFs-stderr.log
    JES log:MergeVCFs.log

    The operations ID is on the line that says "Operation:operations/EI79yL3PKxjO0M6Y5K7o-dQBIJGfndXTDioPcHJvZHVjdGlvblF1ZXVl"

    Clicking it will open a dialog that allows you to read or download the latest metadata report.

    Incidentally, we realize that "what to click on to get what result/info" is fairly unintuitive in the current state of the UI, and we'd like to improve that. If you have any suggestions of what you'd prefer to see / how you'd like the information to be organized, please let us know.

Sign In or Register to comment.