Re: [gentoo-dev] EJOBS variable for EAPI 5?

["Gregory M. Turner" <gmt@malth.us>]

On 9/12/2012 5:58 AM, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 12/09/12 05:55 AM, Gregory M. Turner wrote:
>>
>> Note that, effectively, we have this already, and it's called
>> "portage". But one could certainly make a case for modularizing it
>> better, since, in truth, we are talking about a very common, very
>> abstract problem here which portage shares with any number of
>> batch-build systems.
>>
>> Such an engine could very well do exactly the right thing if it
>> were faced with a constraint that a certain part of a certain build
>> needed to proceed without parallelism due to limitations coming
>> from the build.
>>
>> Also, there are very large parts of most builds -- configure comes
>> to mind -- that don't parallelize even if, perhaps, they should.
>> In such cases, a really smart global parallelism arbiter could
>> easily respond by spawning more jobs from other builds.
>>
>
> So essentially what you're saying here is that it might be worthwhile
> to look into parallelism as a whole and possibly come up with a
> solution that combines 'emerge --jobs' and build-system parallelism
> together to maximum benefit?

Yeah, couldn't have said it better myself ... apparently :)

> Advanced HPC systems (sys-cluster/torque along with an appropriate
> scheduler, for instance) can do such things with their jobs when the
> jobs are properly built; I could see portage being able to handle this
> as well given most of what is necessary is already known (ebuild
> phases, build system type (via eclass), etc).   However, given the
> limitations already put on parallelism in terms of emerge order, etc,
> I could see this solution needing to be -very- complex and integration
> needing to occur on multiple levels.  We'd also need to consider
> distcc (and other cluster-shared compilation methods if there are
> any??)..  It would be an interesting project, though.

ACK all of the above.

Tempting to think more deeply about this but probably the last thing I 
need to do right now is to talk myself into another speculative project.

I've hurt my wrist a bit -- probably an RSI -- so should help deter me :S

Only a few major sources of parallelism exist in portage: --jobs / 
--load-average in emerge opts, multiprocessing eclass & equiv. ebuild 
helper, distcc, and make...  Infrastructure is already in place for all 
of those, so perhaps a good holistic solution exists that isn't /too/ 
complicated.

...OK another f!#!%$^ brainstorm incoming :)

For "JOBS" syntax... what really seems missing in portage are:

   o a clean way to say "dont parallelize this particular make
     invocation" in ebuilds

   o a clean way to globally say "try to use this parallelization
     strategy when emerging."

So what about something like:

   o EMERGE_JOBS and EMERGE_LOAD_AVERAGE make.conf vars equiv. to
     --jobs and --load-average emerge options

   o EBUILD_JOBS and EBUILD_LOAD_AVERAGE make.conf vars

   o If the latter are not specified, they are copied respectively from
     the former (debatable for *_JOBS, since now we get 16 processes when
     we asked for four).

   o MAKEOPTS is auto-extended to reflect EBUILD_JOBS/EBUILD_LOAD_AVERAGE
     if & only if -j|--jobs|-l|--load-average options aren't provided in
     make.conf/profile/envvar MAKEOPTS

   o however, if MAKEOPTS "override" EBUILD_JOBS or EBUILD_LOAD_AVERAGE,
     issue a conspicuous yellow-stars warning

   o extend "emake" to accept a "--non-parallel" option which will
     strip all -j|--jobs|-l|--load-average options from MAKEOPTS;
     perhaps support an equivalent EBUILD_NON_PARALLEL envvar as well,
     with support for override in profile.bashrc. Don't warn about this
     overriding EBUILD_JOBS -- treat as SOP.

   o debatable: respect EBUILD_NON_PARALLEL in multiprocessing, etc?
     or, perhaps, something like:

     EMAKE_NON_PARALLEL=${EMAKE_NON_PARALLEL:-${EBUILD_NON_PARALLEL:-no}}

     could be used to distinguish between "don't use any parallelism"
     and "don't use GNU's make parallelism in emake".  Also maybe a
     better name exists that doesn't use double-negatives.

?

Seems to me something vaguely like the above would provide

   o backward compatibility for ebuilds and make.conf

   o not so vastly different than what we have

   o a decent way to specify what "we really want" globally;
     insofar as portage doesn't do the best job effecting the requested
     parallelization strategy, more ambitious tactics could be
     implemented later, hopefully without huge interface revisions.

-gmt

P.S.:

(Kind-of-crazy additional idea: put ceil(sqrt(EMERGE_JOBS)) into 
EBUILD_JOBS when only the former is specified, and then let 
effective_emerge_jobs equal floor(EMERGE_JOBS/EBUILD_JOBS).... but maybe 
too much automagic for this to be a good idea.)