RFE: Return memory usage stats

dheimandheiman Member, Broadie ✭✭

It would be amazing if we could get back how much memory our tools use in FireCloud.

Because a lot of what is run on FC is too big or cumbersome to run locally, pre-FC memory profiling can be hit or miss, and rolling our own custom solutions into our Docker images is less than ideal.

Being able to quickly see memory usage, especially in large-scale runs, could vastly assist in being able to come up with prediction formulae that we could incorporate into the memory requests in our WDLs, and potentially save a lot of money lost due to requesting too much memory/core.

Such a thing should be relatively straightforward to implement, as cromwell has a monitoring_script workflow option with just such usages in mind.

Tagged:

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I will pass this on as a feature request to the development team. However, our field operations team might already have a script or other solution to help with memory analysis, as I'd imagine this is a common request. I will ask them and get back to you.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin
    edited December 2017

    Hey @dheiman,

    I think this is a great feature request for FireCloud to have an option of exposing memory/disk usage stats per job.

    Until then, it seems like you've already explored building some monitoring usage into your docker directly. A similar solution I've seen users adapt is to add a monitoring script file as a task input, run it in the background as a part of the command and collect the monitoring log as the task's output.

    For example:

    task t {
        String s = "world"
        File monitoringScript = <path-to-monitoring_script>
    
        command {
            chmod u+x ${monitoringScript}
            ${monitoringScript} > monitoring.log &
    
            echo hello ${s}!
        }
    
        output {
            File monitoringLog = "monitoring.log"
        }
    }
    

    I'm posting an example monitoring script for reference here.

Sign In or Register to comment.