Construyendo cientos de componentes C ++ usando múltiples hilos -- performance campo con beginner campo con multithreading campo con perl camp codereview Relacionados El problema

Building hundreds of C++ components using multiple threads


4
vote

problema

Español

Soy bastante nuevo en Perl ( y MultiShReading ), así que me preguntaba si alguien estaría dispuesto a echar un vistazo a mi guión y darme consejos sobre el aumento del rendimiento.

Motivación

El script está diseñado para construir varios cientos de componentes C ++, utilizando múltiples hilos (uno por componente)

Diseño

En un nivel alto, el script debe seguir este flujo:

  • Obtener parámetros
  • predeterminado el número de recopilación y enlace de subprocesos para usar
  • leer en una lista de componentes
    • El formato aquí sería algo así, si pasa en 'servidor', tenemos un archivo que asigna ese nombre de componente a ~ 40 directorios que necesitan construido
  • Lanza 16 hilos en estado 'escuchar'
  • que hasta el trabajo a realizar
    • dependiente de las acciones pasadas en

Código

comentado a lo largo para aclarar qué Espero estar sucediendo

  [22 4] [4 22] [21 5] [5 21] [20 6] [6 20] [19 7] [7 19] [18 8] [8 18] [4 4 18] [4 18 4] [18 4 4] [17 9] [9 17] [17 4 5] [4 17 5] [5 4 17] [4 5 17] [5 17 4] [17 5 4] [16 10] [10 16] [16 4 6] [4 16 6] [6 4 16] [4 6 16] [6 16 4] [16 6 4] [5 5 16] [5 16 5] [16 5 5] [15 11] [11 15] [15 4 7] [4 15 7] [7 4 15] [4 7 15] [7 15 4] [15 7 4] [15 5 6] [5 15 6] [6 5 15] [5 6 15] [6 15 5] [15 6 5] [14 12] [12 14] [14 4 8] [4 14 8] [8 4 14] [4 8 14] [8 14 4] [14 8 4] [14 5 7] [5 14 7] [7 5 14] [5 7 14] [7 14 5] [14 7 5] [6 6 14] [6 14 6] [14 6 6] [13 13] [13 4 9] [4 13 9] [9 4 13] [4 9 13] [9 13 4] [13 9 4] [13 5 8] [5 13 8] [8 5 13] [5 8 13] [8 13 5] [13 8 5] [13 6 7] [6 13 7] [7 6 13] [6 7 13] [7 13 6] [13 7 6] [12 4 10] [4 12 10] [10 4 12] [4 10 12] [10 12 4] [12 10 4] [12 5 9] [5 12 9] [9 5 12] [5 9 12] [9 12 5] [12 9 5] [12 6 8] [6 12 8] [8 6 12] [6 8 12] [8 12 6] [12 8 6] [7 7 12] [7 12 7] [12 7 7] [11 4 11] [4 11 11] [11 11 4] [11 5 10] [5 11 10] [10 5 11] [5 10 11] [10 11 5] [11 10 5] [11 6 9] [6 11 9] [9 6 11] [6 9 11] [9 11 6] [11 9 6] [11 7 8] [7 11 8] [8 7 11] [7 8 11] [8 11 7] [11 8 7] 2  

Supuestos

Como dije, soy bastante nuevo tanto para PERL como para MultiThreading. El supuesto principal que he hecho aquí de lo que realmente no estoy seguro de que se trata del número de hilos para usar. Me he estrechado en base a la cantidad de núcleos disponibles en la máquina, ¿es la forma correcta de hacerlo? Asumí que cualquier cosa y GT; #cores simplemente causaría algunas palas y competencias de recursos; ¿Es eso preciso?

Original en ingles

I'm pretty new to Perl (and multithreading), so I was wondering if someone would be willing to take a look at my script and give me any tips on increasing performance.

Motivation

The script is designed to build several hundred C++ components, using multiple threads (one per component)

Design

At a high level, the script should follow this flow:

  • Get parameters
  • Default the number of compilation and linking threads to use
  • Read in a list of components
    • The format here would be something like, if you pass in 'SERVER', we have a file that maps that component name to ~40 directories that need built
  • Launch 16 threads in 'listening' state
  • Que up the work to be performed
    • Dependent on the actions passed in

Code

Commented throughout to clarify what I expect to be happening

use warnings; use strict;  use POSIX qw(ceil floor strftime); use Perl::OSType ':all';  use Getopt::Long; use Pod::Usage; use List::Util qw(min max); use Log::Message::Simple qw(msg error debug); use File;  use threads; use threads::shared; use Thread::Semaphore;  use Thread::Queue 3.01 qw( );  my $q = Thread::Queue->new();  my @procs; my @comps; my @compsCopy; my @retCodes; share(@retCodes);  my @failedComponents; share(@failedComponents);  my $maxThreadsSem = Thread::Semaphore->new(0);  my $compStr = "";  my $MAKE_INVOCATION_PATH; my $MAKE;  ## Parse options and print usage if there is a syntax error, ## or if usage was explicitly requested. GetOptions('help|?' => \my $help, man => \my $man, 'verbose' => \my $verbose, 'comps:s{,}' => \my @compList, 'COMPLIST:s' => \my $legacyCOMPLIST, 'action:s{1,3}' => \my @actionList,            'clthreads:i' => \my $clthreads, 'linkthreads:i' => \my $linkthreads) or pod2usage(2); pod2usage(1) if $help; pod2usage(-verbose => 2) if $man;  #set silent flag for make $ENV{SL} = '@' unless $verbose;  @compList = qw(SERVER TOOLS) if !@compList; @actionList = qw(clean depend build link) if !@actionList;  #default thread counts ### --> The assumption here is that compile consumes less resources than link, ###     so I default to the number of cores for compile, and slightly lower for linking my $numCores; if(is_os_type('Windows')) {   $numCores = `grep -c '^processor' \/proc\/cpuinfo`; } elsif(is_os_type('Unix')) {   my $numCoresText = `lsconf | grep Processors`;   $numCoresText =~ m/(\d+)/;   $numCores = $1; }  $clthreads = $numCores if !$clthreads; $linkthreads = ceil($numCores * .75) if !$linkthreads;  SetMake(); my $startTime = (strftime "%m-%d-%Y %H:%M:%S", gmtime);  #the work BuildInit(@ARGV);  print "Started At: \t$startTime\n"; print "Ended At: \t" . (strftime "%m-%d-%Y %H:%M:%S", gmtime) ."\n";  my $maxReturnCode = max @retCodes; print "Highest Error Code: $maxReturnCode\n"; print "Components with failures: @failedComponents\n" if @failedComponents;   exit($maxReturnCode);  #---------------------------------------------------------- sub SetMake {   if ( $ENV{MAKECMD} ) {      $MAKE=$ENV{MAKECMD}   } else {      # -k option tells make to continue its attempt to build a component after failures.      # make no longer quits compiling an entire directory after 1 compilation error.      $MAKE="make -k -C";   }    $MAKE_INVOCATION_PATH = $ENV{PATH_CYGDRIVE} || $ENV{PATH}; }  sub BuildInit {     my $actionStr = "";    my $compStr = "";                                       ### --> Gets the list of directories for each 'component'     my @component_dirs;      foreach my $comp (@compList){       @component_dirs = (@component_dirs, GetDirs($comp));      }      ### --> Limit the number of threads that can work at one time     my $maxThreadsSem = Thread::Semaphore->new($clthreads);          print "Action List: @actionList\n";     print "Component List: @compList\n";     print "Component Dir List: @component_dirs\n";     print "Compile Threads: $clthreads\n";     print "Link Threads: $linkthreads\n";      #---------------------------------------     #----   Setup Worker Threads  ----------      for (1..16) {       async {          while (defined(my $job = $q->dequeue())) {             worker($job);          }       };     }      #-----------------------------------     #----   Enqueue The Work  ----------      for my $action (@actionList){       my $sem = Thread::Semaphore->new(0);  ## used to lock execution between 'actions'        #manage our resources while linking       if ($action eq 'link'){         $maxThreadsSem = Thread::Semaphore->new($linkthreads);       }       elsif ($action eq 'build'){         $maxThreadsSem = Thread::Semaphore->new($clthreads);       }        if($action eq 'depend'){         $ENV{PATH} = "$ENV{BUILD_PATH}/makedepend/bin;" . $ENV{PATH};         system("rm dependlist") == 0 or die "Failed on attempt to remove dependlist";         (system("$MAKE $MAKE_INVOCATION_PATH/makedepend")) == 0 or die "Could not build 'makedepend' - killing process";       }        ### --> perform 'action' on each 'component directory'       $q->enqueue([$_, $action, $sem, $maxThreadsSem]) for @component_dirs;        ### --> we don't want to move on to the next action before this action finishes       ### --> e.g., we don't want to start linking prior to compiling finishing       $sem->down(scalar @component_dirs);       print "\n------>> Waiting for prior actions to finish up... <<------\n";     }      # Nothing more to do - notify the Queue that we're not adding anything else     $q->end();     for my $thread (threads->list())     { $thread->join(); }      return 0; }  sub worker {     my ($job) = @_;     my ($component, $action, $sem, $maxThreadsSem) = @$job;      #used to throttle the build / link process according the to specified threads     $maxThreadsSem->down() if $action eq 'link' or $action eq 'build';     Build($component, $action);     $sem->up();     $maxThreadsSem->up() if $action eq 'link' or $action eq 'build'; }  sub Build {      my ($comp, $action) = @_;     my $cmd = "$MAKE $MAKE_INVOCATION_PATH/$comp ";     my $retCode = -1;      #lack of reliable switch support in perl     if($action eq "depend")     {$cmd .= "$action >nul 2>&1"} #suppress output     elsif($action eq "clean")     {$cmd .= $action}     elsif($action eq "build")     {$cmd .= 'l1'}     elsif($action eq "link")     {$cmd .= ''} #add nothing; default is to link     else {die "Action: $action is unknown to me."}      print "\n\t\t*** Performing Action: \'$cmd\' on >>> $comp <<< ***" if ($verbose || $action eq 'depend');      if ($action eq "link"){       # hack around potential race conditions -- will only be an issue during linking       my $tries = 4;       until ($retCode == 0 or $tries == 0){         last if ($retCode = system($cmd)) == 2; #compile error; stop trying         $tries--;       };     }     else{       $retCode = system($cmd);     }      return $retCode if $action eq "depend"; #stop here for dependlist building      lock(@retCodes);  #lock to keep thread-safe     push(@retCodes, ($retCode>>8));          #testing     if($retCode != 0){       if($verbose){         print "\n\t\t*** ERROR IN $comp: $@ !! ***\n";         print "\t\t*** Action: $cmd -->> Error Level: " . ($retCode >>8) . "\n";       }       lock(@failedComponents);  #lock to keep thread-safe       push(@failedComponents, $comp);     }      return $retCode; }   # searches the $strDataFile for the given $strComponentName, then  # fills the @dir_array with the appropriate directories for that component sub GetDirs {    # ... <SNIP> ...    # mostly irrelevant     return @dir_array; }   #----------------------------------------------------------------- #----------------  Documentation / Usage / Help ------------------ =head1 NAME    sample - Using GetOpt::Long and Pod::Usage  =head1 SYNOPSIS    buildmom.pl [options]    options:      -help            Brief help message      -man             Full documentation      -comp s          The list of components to perform actions on (from your comps.dat) | space delimited      -COMPLIST        Legacy support for old build script      -action          A list of actions to perform on each component; executed in the order given  | space delimited                       Available actions include clean, depend, build, link      -clthreads       Number of threads used during compiling      -linkthreads     Number of threads used during linking      -verbose         Enables addition build output - may increase build time   =head1 DESCRIPTION  B<buildmom.pl> is used to perform various actions on your Momentum C++ project.   Parameters worth note:   -comps | [optional: defaulted to ALL] The list of components from your comps.dat file on which to have the            specified actions performed. Example usage:  buildmom.pl -comps FOUNDATION COMMON TRANSPORT SERVER    -COMPLIST | [optional: legacy support] This is the same as the `-comps` parameter, with the exception that it allows                 legacy support for older parameter passing (e.g., -COMPLIST=FOUNDATION:COMMON:TRANSPORT:SERVER)    -action | [optional: defaulted to `clean depend build link`] The set of actions, in order, which you want performed              on the comps list. Available actions include:                   clean: Executes `make clean` for each component specified(excluding ffsfld)                 depend: Builds the make depend list for each component specified                 build: Builds each component specified (formerly 'the first pass')                 link: Links each component specified (formerly 'the second pass')    -clthreads | [optional: defaulted to 100%] Specifies the number of threads to use during compilation   -linkthreads | [optional: defaulted to 75%] Specifies the number of threads to use during linking       { For certain actions, specifically clean and depend, <buildmom.pl> will assume 100% thread usage every time.         These actions are lightweight, and the performance loss from these actions is minimal. If you specify multiple actions         e.g., buildmom.pl -actions clean depend build link, the script will consume 100% CPU for a couple of minutes while performing         the `clean` and `depend` steps, and then throttle back to the specified number of threads.           Another example: buildmom.pl -actions clean depend build link -clthreads 2 -linkthreads 2                     On a 4 core non-HT system, you would see your CPU usage jump to 100% for a few minutes while                     we perform the `clean` and `depend` steps. Once compilation begins, your CPU usage will throttle                      back down to ~50% due to the number of threads specified. } =cut 

Assumptions

Like I said, I'm pretty new to both Perl and multithreading. The primary assumption I've made here that I'm really not sure about is the number of threads to use. I've throttled it based on the number of cores available on the machine - is that the correct way to go about it? I assumed that anything >#cores would just cause some thrashing and competition of resources; is that accurate?

           
         
         

Lista de respuestas


Relacionados problema

4  Dormir con la pantalla cuenta abajo  ( Sleep with count down display ) 
Tengo una función de sueño que muestra el tiempo restante en la SEC, se apreciarían algunos comentarios this9 Este código y actualizaciones están dispon...

14  Subrutina para llamar a otras subrutinas  ( Subroutine to call other subroutines ) 
Tengo una aplicación PERL que le permite al usuario elegir entre dos formatos diferentes para la salida de datos (y probablemente habrá más en un futuro próxi...

7  Diagrama de Voronoi esférico, enfoque de división binario [cerrado]  ( Spherical voronoi diagram binary splitting approach ) 
cerrado. Esta pregunta es off-topic . Actualmente no está aceptando respuestas. ¿Quieres ...

3  Determinar los promedios de columnas  ( Determine column averages ) 
Estoy tratando de escribir código de TERSE PERL para calcular el promedio de cada columna en un archivo. El archivo puede tener & gt; = 1 columnas. use str...

6  Generador de ruido de gradiente Perl  ( Perl gradient noise generator ) 
Estoy trabajando en un motor de gráficos de arriba hacia abajo 2-D y necesito una forma de simular las nubes. Mi idea era que pudiera crear una capa compuesta...

3  ¿Simplificar la expresión regular? (Conversión de Fracciones Unicode a Tex)  ( Simplify regular expression converting unicode fractions to tex ) 
Fondo Estoy convirtiendo el texto Unicode a tex para tipingetting. En la entrada, estoy permitiendo fracciones simples como ½ y ⅔ usando caracteres únic...

6  Herramientas de línea de comandos para formatear tablas  ( Command line tools to format tables ) 
Soy un gran fanático de un revestimiento usando sed awk 9988777665544332 y otras herramientas. Pero hay cosas difíciles de hacer en una sola línea, como...

2  Lectura de un archivo binario de longitud de registro dinámico  ( Reading a dynamic record length binary file ) 
Estoy tratando de leer un archivo binario de longitud de registro dinámico, que tiene datos como este: field1field2field3field4vector1vector2 Aquí, Fie...

6  Hosth de descarga en columnas  ( Dump hash in columns ) 
Intenté responder a la pregunta "¿Cómo imprimir un hash en Perl, de modo que se imprimen 3 pares de valor clave en cada línea? " como este . Como obtuve u...

3  Biblioteca de Wrapper Perl (escrito usando Moose) para una API de descanso  ( Perl wrapper library written using moose for a rest api ) 
Escribí una biblioteca de envoltura para una API de descanso en Perl usando la Biblioteca Moose. Me gustaría reunir algunos comentarios sobre ella (principalm...




© 2022 respuesta.top Reservados todos los derechos. Centro de preguntas y respuestas reservados todos los derechos