Remake
 All Classes Files Functions Variables Typedefs Enumerations Enumerator Macros Groups Pages
remake.cpp
Go to the documentation of this file.
1 /**
2 @mainpage Remake, a build system that bridges the gap between make and redo.
3 
4 As with <b>make</b>, <b>remake</b> uses a centralized rule file, which is
5 named <b>Remakefile</b>. It contains rules with a <em>make</em>-like
6 syntax:
7 
8 @verbatim
9 target1 target2 ... : prerequisite1 prerequisite2 ...
10  shell script
11  that builds
12  the targets
13 @endverbatim
14 
15 A target is known to be up-to-date if all its prerequisites are. If it
16 has no known prerequisites yet the file already exits, it is assumed to
17 be up-to-date. Obsolete targets are rebuilt thanks to the shell script
18 provided by the rule.
19 
20 As with <b>redo</b>, <b>remake</b> supports dynamic dependencies in
21 addition to these static dependencies. Whenever a script executes
22 <tt>remake prerequisite4 prerequisite5 ...</tt>, these prerequisites are
23 rebuilt if they are obsolete. (So <b>remake</b> acts like
24 <b>redo-ifchange</b>.) Moreover, all the dependencies are stored in file
25 <b>.remake</b> so that they are remembered in subsequent runs. Note that
26 dynamic dependencies from previous runs are only used to decide whether a
27 target is obsolete; they are not automatically rebuilt when they are
28 obsolete yet a target depends on them. They will only be rebuilt once the
29 dynamic call to <b>remake</b> is executed.
30 
31 In other words, the following two rules have almost the same behavior.
32 
33 @verbatim
34 target1 target2 ... : prerequisite1 prerequisite2 ...
35  shell script
36 
37 target1 target2 ... :
38  remake prerequisite1 prerequisite2 ...
39  shell script
40 @endverbatim
41 
42 (There is a difference if the targets already exist, have never been
43 built before, and the prerequisites are either younger or obsolete, since
44 the targets will not be rebuilt in the second case.)
45 
46 The above usage of dynamic dependencies is hardly useful. Their strength
47 lies in the fact that they can be computed on the fly:
48 
49 @verbatim
50 %.o : %.c
51  gcc -MMD -MF $@.d -o $@ -c $<
52  remake -r < $@.d
53  rm $@.d
54 
55 %.cmo : %.ml
56  ocamldep $< | remake -r $@
57  ocamlc -c $<
58 
59 after.xml: before.xml rules.xsl
60  xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
61  remake `sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps`
62  rm deps
63 @endverbatim
64 
65 Note that the first rule fails if any of the header files included by
66 a C source file has to be automatically generated. In that case, one
67 should perform a first call to <b>remake</b> them before calling the
68 compiler. (Dependencies from several calls to <b>remake</b> are
69 cumulative, so they will all be remembered the next time.)
70 
71 \section sec-usage Usage
72 
73 Usage: <tt>remake <i>options</i> <i>targets</i></tt>
74 
75 Options:
76 
77 - <tt>-d</tt>: Echo script commands.
78 - <tt>-f FILE</tt>: Read <tt>FILE</tt> as <b>Remakefile</b>.
79 - <tt>-j[N]</tt>, <tt>--jobs=[N]</tt>: Allow <tt>N</tt> jobs at once;
80  infinite jobs with no argument.
81 - <tt>-k</tt>, <tt>--keep-going</tt>: Keep going when some targets cannot be made.
82 - <tt>-r</tt>: Look up targets from the dependencies on standard input.
83 - <tt>-s</tt>, <tt>--silent</tt>, <tt>--quiet</tt>: Do not echo targets.
84 
85 \section sec-syntax Syntax
86 
87 Lines starting with a space character or a tabulation are assumed to be rule
88 scripts. They are only allowed after a rule header.
89 
90 Lines starting with <tt>#</tt> are considered to be comments and are ignored.
91 They do interrupt rule scripts though.
92 
93 Any other line is either a rule header or a variable definition. If such a
94 line ends with a backslash, the following line break is ignored and the line
95 extends to the next one.
96 
97 Rule headers are a nonempty list of names, followed by a colon, followed by
98 another list of names, possibly empty. Variable definitions are a single
99 name followed by equal followed by a list of names, possibly empty. Basically,
100 the syntax of a rule is as follows:
101 
102 @verbatim
103 targets : prerequisites
104  shell script
105 @endverbatim
106 
107 List of names are space-separated sequences of names. If a name contains a
108 space character, it should be put into double quotes. Names can not be any
109 of the following special characters <tt>:$(),="</tt>. Again, quotation
110 should be used. Quotation marks can be escaped by a backslash inside
111 quoted names.
112 
113 \subsection sec-variables Variables
114 
115 Variables can be used to factor lists of targets or prerequisites. They are
116 expanded as they are encountered during <b>Remakefile</b> parsing.
117 
118 @verbatim
119 VAR2 = a
120 VAR1 = c d
121 VAR2 += $(VAR1) b
122 $(VAR2) e :
123 @endverbatim
124 
125 Variable assignments can appear instead of prerequisites inside non-generic
126 rules with no script. They are then expanded inside the corresponding
127 generic rule.
128 
129 @verbatim
130 foo.o: CFLAGS += -DBAR
131 
132 %.o : %.c
133  gcc $(CFLAGS) -MMD -MF $@.d -o $@ -c $<
134  remake -r < $@.d
135  rm $@.d
136 @endverbatim
137 
138 Note: contrarily to <b>make</b>, variable names have to be enclosed in
139 parentheses. For instance, <tt>$y</tt> is not a shorthand for <tt>\$(y)</tt> and
140 is left unexpanded.
141 
142 \subsection sec-autovars Automatic variables
143 
144 The following special symbols can appear inside scripts:
145 
146 - <tt>$&lt;</tt> expands to the first static prerequisite of the rule.
147 - <tt>$^</tt> expands to all the static prerequisites of the rule, including
148  duplicates if any.
149 - <tt>$\@</tt> expands to the first target of the rule.
150 - <tt>$*</tt> expands to the string that matched <tt>%</tt> in a generic rule.
151 - <tt>$$</tt> expands to a single dollar symbol.
152 
153 Note: contrarily to <b>make</b>, there are no corresponding variables. For
154 instance, <tt>$^</tt> is not a shorthand for <tt>$(^)</tt>. Another
155 difference is that <tt>$\@</tt> is always the first target, not the one that
156 triggered the rule.
157 
158 \subsection sec-functions Built-in functions
159 
160 <b>remake</b> also supports a few built-in functions inspired from <b>make</b>.
161 
162 - <tt>$(addprefix <i>prefix</i>, <i>list</i>)</tt> returns the list obtained
163  by prepending its first argument to each element of its second argument.
164 - <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
165  by appending its first argument to each element of its second argument.
166 
167 \section sec-semantics Semantics
168 
169 \subsection src-obsolete When are targets obsolete?
170 
171 A target is obsolete:
172 
173 - if there is no file corresponding to the target, or to one of its siblings
174  in a multi-target rule,
175 - if any of its dynamic prerequisites from a previous run or any of its static
176  prerequisites is obsolete,
177 - if the latest file corresponding to its siblings or itself is older than any
178  of its dynamic prerequisites or static prerequisites.
179 
180 In all the other cases, it is assumed to be up-to-date (and so are all its
181 siblings). Note that the last rule above says "latest" and not "earliest". While
182 it might cause some obsolete targets to go unnoticed in corner cases, it allows
183 for the following kind of rules:
184 
185 @verbatim
186 config.h stamp-config_h: config.h.in config.status
187  ./config.status config.h
188  touch stamp-config_h
189 @endverbatim
190 
191 A <tt>config.status</tt> file generally does not update header files (here
192 <tt>config.h</tt>) if they would not change. As a consequence, if not for the
193 <tt>stamp-config_h</tt> file above, a header would always be considered obsolete
194 once one of its prerequisites is modified. Note that touching <tt>config.h</tt>
195 rather than <tt>stamp-config_h</tt> would defeat the point of not updating it
196 in the first place, since the program files would need to be rebuilt.
197 
198 Once all the static prerequisites of a target have been rebuilt, <b>remake</b>
199 checks whether the target still needs to be built. If it was obsolete only
200 because its prerequisites needed to be rebuilt and none of them changed, the
201 target is assumed to be up-to-date.
202 
203 \subsection sec-rules How are targets (re)built?
204 
205 There are two kinds of rules. If any of the targets or prerequisites contains
206 a <tt>%</tt> character, the rule is said to be <em>generic</em>. All the
207 targets of the rule shall then contain a single <tt>%</tt> character. All the
208 other rules are said to be <em>specific</em>.
209 
210 A rule is said to <em>match</em> a given target:
211 
212 - if it is specific and the target appears inside its target list,
213 - if it is generic and there is a way to replace the <tt>%</tt> character
214  from one of its targets so that it matches the given target.
215 
216 When <b>remake</b> tries to build a given target, it looks for a specific rule
217 that matches it. If there is one and its script is nonempty, it uses it to
218 rebuild the target.
219 
220 Otherwise, it looks for a generic rule that match the target. If there are
221 several matching rules, it chooses the one with the shortest pattern (and if
222 there are several ones, the earliest one). <b>remake</b> then looks for
223 specific rules that match each target of the generic rule. All the
224 prerequisites of these specific rules are added to those of the generic rule.
225 The script of the generic rule is used to build the target.
226 
227 Example:
228 
229 @verbatim
230 t%1 t2%: p1 p%2
231  commands building t%1 and t2%
232 
233 t2z: p4
234  commands building t2z
235 
236 ty1: p3
237 
238 # t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
239 # t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
240 # t2z is built by the second rule and its prerequisite is p4
241 @endverbatim
242 
243 The set of rules from <b>Remakefile</b> is ill-formed:
244 
245 - if any specific rule matching a target of the generic rule has a nonempty script,
246 - if any target of the generic rule is matched by a generic rule with a shorter pattern.
247 
248 \section sec-compilation Compilation
249 
250 - On Linux, MacOSX, and BSD: <tt>g++ -o remake remake.cpp</tt>
251 - On Windows: <tt>g++ -o remake.exe remake.cpp -lws2_32</tt>
252 
253 Installing <b>remake</b> is needed only if <b>Remakefile</b> does not
254 specify the path to the executable for its recursive calls. Thanks to its
255 single source file, <b>remake</b> can be shipped inside other packages and
256 built at configuration time.
257 
258 \section sec-differences Differences with other build systems
259 
260 Differences with <b>make</b>:
261 
262 - Dynamic dependencies are supported.
263 - For rules with multiple targets, the shell script is executed only once
264  and is assumed to build all the targets. There is no need for
265  convoluted rules that are robust enough for parallel builds. For generic
266  rules, this is similar to the behavior of pattern rules from <b>gmake</b>.
267 - As with <b>redo</b>, only one shell is run when executing a script,
268  rather than one per script line. Note that the shells are run with
269  option <tt>-e</tt>, thus causing them to exit as soon as an error is
270  encountered.
271 - The prerequisites of generic rules (known as implicit rules in make lingo)
272  are not used to decide between several of them. <b>remake</b> does not
273  select one for which it could satisfy the dependencies.
274 - Variables and built-in functions are expanded as they are encountered
275  during <b>Remakefile</b> parsing.
276 
277 Differences with <b>redo</b>:
278 
279 - As with <b>make</b>, it is possible to write the following kind of rules
280  in <b>remake</b>.
281 @verbatim
282 Remakefile: Remakefile.in ./config.status
283  ./config.status Remakefile
284 @endverbatim
285 - If a target is already built the first time <b>remake</b> runs, it still
286  uses the static prerequisites of rules mentioning it to check whether it
287  needs to be rebuilt. It does not assume it to be up-to-date. As with
288  <b>redo</b> though, if its obsolete status would be due to a dynamic
289  prerequisite, it will go unnoticed; it should be removed beforehand.
290 - Multiple targets are supported.
291 - <b>remake</b> has almost no features: no checksum-based dependencies, no
292  compatibility with job servers, etc.
293 
294 \section sec-limitations Limitations
295 
296 - When the user calls <b>remake</b>, the current working directory should be
297  the one containing <b>.remake</b>. Rules are understood relatively to this
298  directory. If a rule script calls <b>remake</b>, the current working
299  directory should be the same as the one from the original <b>remake</b>.
300 - Some cases of ill-formed rules are not caught by <b>remake</b> and can
301  thus lead to unpredictable behaviors.
302 
303 \section sec-links Links
304 
305 @see http://cr.yp.to/redo.html for the philosophy of <b>redo</b> and
306 https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.
307 
308 \section sec-licensing Licensing
309 
310 @author Guillaume Melquiond
311 @version 0.8
312 @date 2012-2013
313 @copyright
314 This program is free software: you can redistribute it and/or modify
315 it under the terms of the GNU General Public License as published by
316 the Free Software Foundation, either version 3 of the License, or
317 (at your option) any later version.
318 \n
319 This program is distributed in the hope that it will be useful,
320 but WITHOUT ANY WARRANTY; without even the implied warranty of
321 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
322 GNU General Public License for more details.
323 
324 \section sec-internals Internals
325 
326 The parent <b>remake</b> process acts as a server. The other ones have a
327 REMAKE_SOCKET environment variable that tells them how to contact the
328 server. They send the content of the REMAKE_JOB_ID environment variable,
329 so that the server can associate the child targets to the jobs that
330 spawned them. They then wait for completion and exit with the status
331 returned by the server. This is handled by #client_mode.
332 
333 The server calls #load_dependencies and #save_dependencies to serialize
334 dynamic dependencies from <b>.remake</b>. It loads <b>Remakefile</b> with
335 #load_rules. It then runs #server_mode, which calls #server_loop.
336 
337 When building a target, the following sequence of events happens:
338 
339 - #start calls #find_rule (and #find_generic_rule) to get the rule.
340 - It then creates a pseudo-client if the rule has static dependencies, or
341  calls #run_script otherwise. In both cases, a new job is created and its
342  targets are put into #job_targets.
343 - #run_script creates a shell process and stores it in #job_pids. It
344  increases #running_jobs.
345 - The child process possibly calls <b>remake</b> with a list of targets.
346 - #accept_client receives a build request from a child process and adds
347  it to #clients. It also records the new dependencies of the job into
348  #dependencies. It increases #waiting_jobs.
349 - #handle_clients uses #get_status to look up the obsoleteness of the
350  targets.
351 - Once the targets of a request have been built or one of them has failed,
352  #handle_clients calls #complete_request and removes the request from
353  #clients.
354 - If the build targets come from a pseudo-client, #complete_request calls
355  #run_script. Otherwise it sends the reply to the corresponding child
356  process and decreases #waiting_jobs.
357 - When a child process ends, #server_loop calls #finalize_job, which
358  removes the process from #job_pids, decreases #running_jobs, and calls
359  #complete_job.
360 - #complete_job removes the job from #job_targets and calls #update_status
361  to change the status of the targets. It also removes the target files in
362  case of failure.
363 */
364 
365 #ifdef _WIN32
366 #define WIN32_LEAN_AND_MEAN
367 #define WINDOWS
368 #endif
369 
370 #include <fstream>
371 #include <iostream>
372 #include <list>
373 #include <map>
374 #include <set>
375 #include <sstream>
376 #include <string>
377 #include <vector>
378 #include <cassert>
379 #include <cstdlib>
380 #include <ctime>
381 #include <errno.h>
382 #include <fcntl.h>
383 #include <signal.h>
384 #include <unistd.h>
385 #include <sys/stat.h>
386 #include <sys/types.h>
387 
388 #ifdef __APPLE__
389 #define MACOSX
390 #endif
391 
392 #ifdef __linux__
393 #define LINUX
394 #endif
395 
396 #ifdef WINDOWS
397 #include <windows.h>
398 #include <winbase.h>
399 #include <winsock2.h>
400 #define pid_t HANDLE
401 typedef SOCKET socket_t;
402 #else
403 #include <sys/socket.h>
404 #include <sys/un.h>
405 #include <sys/wait.h>
406 typedef int socket_t;
407 enum { INVALID_SOCKET = -1 };
408 #endif
409 
410 #if defined(WINDOWS) || defined(MACOSX)
411 enum { MSG_NOSIGNAL = 0 };
412 #endif
413 
414 typedef std::list<std::string> string_list;
415 
416 typedef std::set<std::string> string_set;
417 
418 /**
419  * Reference-counted shared object.
420  * @note The default constructor delays the creation of the object until it
421  * is first dereferenced.
422  */
423 template<class T>
424 struct ref_ptr
425 {
426  struct content
427  {
428  size_t cnt;
429  T val;
430  content(): cnt(1) {}
431  content(T const &t): cnt(1), val(t) {}
432  };
433  mutable content *ptr;
434  ref_ptr(): ptr(NULL) {}
435  ref_ptr(T const &t): ptr(new content(t)) {}
436  ref_ptr(ref_ptr const &p): ptr(p.ptr) { if (ptr) ++ptr->cnt; }
437  ~ref_ptr() { if (ptr && --ptr->cnt == 0) delete ptr; }
439  {
440  if (ptr == p.ptr) return *this;
441  if (ptr && --ptr->cnt == 0) delete ptr;
442  ptr = p.ptr;
443  if (ptr) ++ptr->cnt;
444  return *this;
445  }
446  T &operator*() const
447  {
448  if (!ptr) ptr = new content;
449  return ptr->val;
450  }
451  T *operator->() const { return &**this; }
452 };
453 
455 {
458 };
459 
460 typedef std::map<std::string, ref_ptr<dependency_t> > dependency_map;
461 
462 typedef std::map<std::string, string_list> variable_map;
463 
464 /**
465  * Build status of a target.
466  */
468 {
469  Uptodate, ///< Target is up-to-date.
470  Todo, ///< Target is missing or obsolete.
471  Recheck, ///< Target has an obsolete dependency.
472  Running, ///< Target is being rebuilt.
473  Remade, ///< Target was successfully rebuilt.
474  Failed ///< Build failed for target.
475 };
476 
477 /**
478  * Build status of a target.
479  */
480 struct status_t
481 {
482  status_e status; ///< Actual status.
483  time_t last; ///< Last-modified date.
484 };
485 
486 typedef std::map<std::string, status_t> status_map;
487 
488 /**
489  * Delayed assignment to a variable.
490  */
491 struct assign_t
492 {
493  std::string name;
494  bool append;
496 };
497 
498 typedef std::list<assign_t> assign_list;
499 
500 /**
501  * A rule loaded from Remakefile.
502  */
503 struct rule_t
504 {
505  string_list targets; ///< Files produced by this rule.
506  string_list deps; ///< Files used for an implicit call to remake at the start of the script.
507  assign_list vars; ///< Values of variables.
508  std::string script; ///< Shell script for building the targets.
509  std::string stem; ///< String used to instantiate a generic rule.
510 };
511 
512 typedef std::list<rule_t> rule_list;
513 
514 typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
515 
516 typedef std::map<int, string_list> job_targets_map;
517 
518 typedef std::map<pid_t, int> pid_job_map;
519 
520 /**
521  * Client waiting for a request to complete.
522  *
523  * There are two kinds of clients:
524  * - real clients, which are instances of remake created by built scripts,
525  * - pseudo clients, which are created by the server to build specific targets.
526  *
527  * Among pseudo clients, there are two categories:
528  * - original clients, which are created for the targets passed on the
529  * command line by the user or for the initial regeneration of the rule file,
530  * - dependency clients, which are created to handle rules that have
531  * explicit dependencies and thus to emulate a call to remake.
532  */
533 struct client_t
534 {
535  socket_t socket; ///< Socket used to reply to the client (invalid for pseudo clients).
536  int job_id; ///< Job for which the built script called remake and spawned the client (negative for original clients).
537  bool failed; ///< Whether some targets failed in mode -k.
538  string_list pending; ///< Targets not yet started.
539  string_set running; ///< Targets being built.
540  rule_t *delayed; ///< Rule that implicitly created a dependency client, and which script has to be started on request completion.
541  client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(NULL) {}
542 };
543 
544 typedef std::list<client_t> client_list;
545 
546 /**
547  * Map from variable names to their content.
548  */
550 
551 /**
552  * Map from targets to their known dependencies.
553  */
555 
556 /**
557  * Map from targets to their build status.
558  */
560 
561 /**
562  * Set of generic rules loaded from Remakefile.
563  */
565 
566 /**
567  * Map from targets to specific rules loaded from Remakefile.
568  */
570 
571 /**
572  * Map from jobs to targets being built.
573  */
575 
576 /**
577  * Map from jobs to shell pids.
578  */
580 
581 /**
582  * List of clients waiting for a request to complete.
583  * New clients are put to front, so that the build process is depth-first.
584  */
586 
587 /**
588  * Maximum number of parallel jobs (non-positive if unbounded).
589  * Can be modified by the -j option.
590  */
591 static int max_active_jobs = 1;
592 
593 /**
594  * Whether to keep building targets in case of failure.
595  * Can be modified by the -k option.
596  */
597 static bool keep_going = false;
598 
599 /**
600  * Number of jobs currently running:
601  * - it increases when a process is created in #run_script,
602  * - it decreases when a completion message is received in #finalize_job.
603  *
604  * @note There might be some jobs running while #clients is empty.
605  * Indeed, if a client requested two targets to be rebuilt, if they
606  * are running concurrently, if one of them fails, the client will
607  * get a failure notice and might terminate before the other target
608  * finishes.
609  */
610 static int running_jobs = 0;
611 
612 /**
613  * Number of jobs currently waiting for a build request to finish:
614  * - it increases when a build request is received in #accept_client
615  * (since the client is presumably waiting for the reply),
616  * - it decreases when a reply is sent in #complete_request.
617  */
618 static int waiting_jobs = 0;
619 
620 /**
621  * Global counter used to produce increasing job numbers.
622  * @see job_targets
623  */
624 static int job_counter = 0;
625 
626 /**
627  * Socket on which the server listens for client request.
628  */
630 
631 /**
632  * Whether the request of an original client failed.
633  */
634 static bool build_failure;
635 
636 #ifndef WINDOWS
637 /**
638  * Name of the server socket in the file system.
639  */
640 static char *socket_name;
641 #endif
642 
643 /**
644  * Name of the first target of the first specific rule, used for default run.
645  */
646 static std::string first_target;
647 
648 /**
649  * Whether a short message should be displayed for each target.
650  */
651 static bool show_targets = true;
652 
653 /**
654  * Whether script commands are echoed.
655  */
656 static bool echo_scripts = false;
657 
658 static time_t now = time(NULL);
659 
660 static std::string working_dir;
661 
662 #ifndef WINDOWS
663 static volatile sig_atomic_t got_SIGCHLD = 0;
664 
665 static void sigchld_handler(int)
666 {
667  got_SIGCHLD = 1;
668 }
669 
670 static void sigint_handler(int)
671 {
672  // Child processes will receive the signal too, so just prevent
673  // new jobs from starting and wait for the running jobs to fail.
674  keep_going = false;
675 }
676 #endif
677 
678 struct log
679 {
680  bool active, open;
681  int depth;
682  log(): active(false), open(false), depth(0)
683  {
684  }
685  std::ostream &operator()()
686  {
687  if (open) std::cerr << std::endl;
688  assert(depth >= 0);
689  std::cerr << std::string(depth * 2, ' ');
690  open = false;
691  return std::cerr;
692  }
693  std::ostream &operator()(bool o)
694  {
695  if (o && open) std::cerr << std::endl;
696  if (!o) --depth;
697  assert(depth >= 0);
698  if (o || !open) std::cerr << std::string(depth * 2, ' ');
699  if (o) ++depth;
700  open = o;
701  return std::cerr;
702  }
703 };
704 
706 
708 {
711  {
712  }
714  {
715  if (debug.active && still_open) debug(false) << "done\n";
716  }
717 };
718 
719 #define DEBUG if (debug.active) debug()
720 #define DEBUG_open log_auto_close auto_close; if (debug.active) debug(true)
721 #define DEBUG_close if ((auto_close.still_open = false), debug.active) debug(false)
722 
723 /**
724  * Strong typedef for strings that need escaping.
725  * @note The string is stored as a reference, so the constructed object is
726  * meant to be immediately consumed.
727  */
729 {
730  std::string const &input;
731  escape_string(std::string const &s): input(s) {}
732 };
733 
734 /**
735  * Write the string in @a se to @a out if it does not contain any special
736  * characters, a quoted and escaped string otherwise.
737  */
738 static std::ostream &operator<<(std::ostream &out, escape_string const &se)
739 {
740  std::string const &s = se.input;
741  char const *quoted_char = ",: '";
742  char const *escaped_char = "\"\\$!";
743  bool need_quotes = false;
744  char *buf = NULL;
745  size_t len = s.length(), last = 0, j = 0;
746  for (size_t i = 0; i < len; ++i)
747  {
748  if (strchr(escaped_char, s[i]))
749  {
750  need_quotes = true;
751  if (!buf) buf = new char[len * 2];
752  memcpy(&buf[j], &s[last], i - last);
753  j += i - last;
754  buf[j++] = '\\';
755  buf[j++] = s[i];
756  last = i + 1;
757  }
758  if (!need_quotes && strchr(quoted_char, s[i]))
759  need_quotes = true;
760  }
761  if (!need_quotes) return out << s;
762  out << '"';
763  if (!buf) return out << s << '"';
764  out.write(buf, j);
765  out.write(&s[last], len - last);
766  delete[] buf;
767  return out << '"';
768 }
769 
770 /**
771  * @defgroup paths Path helpers
772  *
773  * @{
774  */
775 
776 /**
777  * Initialize #working_dir.
778  */
780 {
781  char buf[1024];
782  char *res = getcwd(buf, sizeof(buf));
783  if (!res)
784  {
785  perror("Failed to get working directory");
786  exit(EXIT_FAILURE);
787  }
788  working_dir = buf;
789 }
790 
791 /**
792  * Normalize an absolute path with respect to the working directory.
793  * Paths outside the working subtree are left unchanged.
794  */
795 static std::string normalize_abs(std::string const &s)
796 {
797  size_t l = working_dir.length();
798  if (s.compare(0, l, working_dir)) return s;
799  size_t ll = s.length();
800  if (ll == l) return ".";
801  if (s[l] != '/')
802  {
803  size_t pos = s.rfind('/', l);
804  assert(pos != std::string::npos);
805  return s.substr(pos + 1);
806  }
807  if (ll == l + 1) return ".";
808  return s.substr(l + 1);
809 }
810 
811 /**
812  * Normalize a target name.
813  */
814 static std::string normalize(std::string const &s)
815 {
816 #ifdef WINDOWS
817  char const *delim = "/\\";
818 #else
819  char delim = '/';
820 #endif
821  size_t prev = 0, len = s.length();
822  size_t pos = s.find_first_of(delim);
823  if (pos == std::string::npos) return s;
824  bool absolute = pos == 0;
825  string_list l;
826  for (;;)
827  {
828  if (pos != prev)
829  {
830  std::string n = s.substr(prev, pos - prev);
831  if (n == "..")
832  {
833  if (!l.empty()) l.pop_back();
834  else if (!absolute)
835  return normalize(working_dir + '/' + s);
836  }
837  else if (n != ".")
838  l.push_back(n);
839  }
840  ++pos;
841  if (pos >= len) break;
842  prev = pos;
843  pos = s.find_first_of(delim, prev);
844  if (pos == std::string::npos) pos = len;
845  }
846  string_list::const_iterator i = l.begin(), i_end = l.end();
847  if (i == i_end) return absolute ? "/" : ".";
848  std::string n;
849  if (absolute) n.push_back('/');
850  n.append(*i);
851  for (++i; i != i_end; ++i)
852  {
853  n.push_back('/');
854  n.append(*i);
855  }
856  if (absolute) return normalize_abs(n);
857  return n;
858 }
859 
860 /**
861  * Normalize the content of a list of targets.
862  */
864 {
865  for (string_list::iterator i = l.begin(),
866  i_end = l.end(); i != i_end; ++i)
867  {
868  *i = normalize(*i);
869  }
870 }
871 
872 /** @} */
873 
874 /**
875  * @defgroup lexer Lexer
876  *
877  * @{
878  */
879 
880 /**
881  * Skip spaces.
882  */
883 static void skip_spaces(std::istream &in)
884 {
885  char c;
886  while (strchr(" \t", (c = in.get()))) {}
887  if (in.good()) in.putback(c);
888 }
889 
890 /**
891  * Skip empty lines.
892  */
893 static void skip_empty(std::istream &in)
894 {
895  char c;
896  while (strchr("\r\n", (c = in.get()))) {}
897  if (in.good()) in.putback(c);
898 }
899 
900 /**
901  * Skip end of line. If @a multi is true, skip the following empty lines too.
902  * @return true if there was a line to end.
903  */
904 static bool skip_eol(std::istream &in, bool multi = false)
905 {
906  char c = in.get();
907  if (c == '\r') c = in.get();
908  if (c != '\n' && in.good()) in.putback(c);
909  if (c != '\n' && !in.eof()) return false;
910  if (multi) skip_empty(in);
911  return true;
912 }
913 
914 enum
915 {
917  Word = 1 << 1,
918  Colon = 1 << 2,
919  Equal = 1 << 3,
920  Dollarpar = 1 << 4,
921  Rightpar = 1 << 5,
922  Comma = 1 << 6,
923  Plusequal = 1 << 7,
924 };
925 
926 /**
927  * Skip spaces and peek at the next token.
928  * If it is one of @a mask, skip it (if it is not Word) and return it.
929  * @note For composite tokens allowed by @a mask, input characters might
930  * have been eaten even for an Unexpected result.
931  */
932 static int expect_token(std::istream &in, int mask)
933 {
934  while (true)
935  {
936  skip_spaces(in);
937  char c = in.peek();
938  if (!in.good()) return Unexpected;
939  int tok;
940  switch (c)
941  {
942  case '\r':
943  case '\n': return Unexpected;
944  case ':': tok = Colon; break;
945  case ',': tok = Comma; break;
946  case '=': tok = Equal; break;
947  case ')': tok = Rightpar; break;
948  case '$':
949  if (!(mask & Dollarpar)) return Unexpected;
950  in.ignore(1);
951  tok = Dollarpar;
952  if (in.peek() != '(') return Unexpected;
953  break;
954  case '+':
955  if (!(mask & Plusequal)) return Unexpected;
956  in.ignore(1);
957  tok = Plusequal;
958  if (in.peek() != '=') return Unexpected;
959  break;
960  case '\\':
961  in.ignore(1);
962  if (skip_eol(in)) continue;
963  in.putback('\\');
964  return mask & Word ? Word : Unexpected;
965  default:
966  return mask & Word ? Word : Unexpected;
967  }
968  if (!(tok & mask)) return Unexpected;
969  in.ignore(1);
970  return tok;
971  }
972 }
973 
974 /**
975  * Read a (possibly quoted) word.
976  */
977 static std::string read_word(std::istream &in)
978 {
979  int c = in.get();
980  std::string res;
981  if (!in.good()) return res;
982  char const *separators = " \t\r\n:$(),=+\"";
983  bool quoted = c == '"';
984  if (!quoted)
985  {
986  if (strchr(separators, c))
987  {
988  in.putback(c);
989  return res;
990  }
991  res += c;
992  }
993  while (true)
994  {
995  c = in.get();
996  if (!in.good()) return res;
997  if (quoted)
998  {
999  if (c == '\\')
1000  res += in.get();
1001  else if (c == '"')
1002  return res;
1003  else
1004  res += c;
1005  }
1006  else
1007  {
1008  if (strchr(separators, c))
1009  {
1010  in.putback(c);
1011  return res;
1012  }
1013  res += c;
1014  }
1015  }
1016 }
1017 
1018 /** @} */
1019 
1020 /**
1021  * @defgroup stream Token streams
1022  *
1023  * @{
1024  */
1025 
1026 /**
1027  * Possible results from word producers.
1028  */
1030 {
1034 };
1035 
1036 /**
1037  * Interface for word producers.
1038  */
1040 {
1041  virtual ~generator() {}
1042  virtual input_status next(std::string &) = 0;
1043 };
1044 
1045 /**
1046  * Generator for the words of a variable.
1047  */
1049 {
1050  std::string name;
1051  string_list::const_iterator cur1, end1;
1052  assign_list::const_iterator cur2, end2;
1053  variable_generator(std::string const &, assign_list const *);
1054  input_status next(std::string &);
1055 };
1056 
1058  assign_list const *local_variables): name(n)
1059 {
1060  bool append = true;
1061  if (local_variables)
1062  {
1063  // Set cur2 to the last variable overwriter, if any.
1064  cur2 = local_variables->begin();
1065  end2 = local_variables->end();
1066  for (assign_list::const_iterator i = cur2; i != end2; ++i)
1067  {
1068  if (i->name == name && !i->append)
1069  {
1070  append = false;
1071  cur2 = i;
1072  }
1073  }
1074  }
1075  else
1076  {
1077  static assign_list dummy;
1078  cur2 = dummy.begin();
1079  end2 = dummy.end();
1080  }
1081  static string_list dummy;
1082  cur1 = dummy.begin();
1083  end1 = dummy.end();
1084  if (append)
1085  {
1086  variable_map::const_iterator i = variables.find(name);
1087  if (i == variables.end()) return;
1088  cur1 = i->second.begin();
1089  end1 = i->second.end();
1090  }
1091 }
1092 
1094 {
1095  restart:
1096  if (cur1 != end1)
1097  {
1098  res = *cur1;
1099  ++cur1;
1100  return Success;
1101  }
1102  while (cur2 != end2)
1103  {
1104  if (cur2->name == name)
1105  {
1106  cur1 = cur2->value.begin();
1107  end1 = cur2->value.end();
1108  ++cur2;
1109  goto restart;
1110  }
1111  ++cur2;
1112  }
1113  return Eof;
1114 }
1115 
1116 /**
1117  * Generator for the words of an input stream.
1118  */
1120 {
1121  std::istream &in;
1125  input_generator(std::istream &i, assign_list const *lv, bool e = false)
1126  : in(i), nested(NULL), local_variables(lv), earliest_exit(e), done(false) {}
1127  input_status next(std::string &);
1128  ~input_generator() { assert(!nested); }
1129 };
1130 
1131 static generator *get_function(input_generator const &, std::string const &);
1132 
1134 {
1135  if (nested)
1136  {
1137  restart:
1138  input_status s = nested->next(res);
1139  if (s == Success) return Success;
1140  delete nested;
1141  nested = NULL;
1142  if (s == SyntaxError) return SyntaxError;
1143  }
1144  if (done) return Eof;
1145  if (earliest_exit) done = true;
1146  switch (expect_token(in, Word | Dollarpar))
1147  {
1148  case Word:
1149  res = read_word(in);
1150  return Success;
1151  case Dollarpar:
1152  {
1153  std::string name = read_word(in);
1154  if (name.empty()) return SyntaxError;
1155  if (expect_token(in, Rightpar))
1157  else
1158  {
1159  nested = get_function(*this, name);
1160  if (!nested) return SyntaxError;
1161  }
1162  goto restart;
1163  }
1164  default:
1165  return Eof;
1166  }
1167 }
1168 
1169 /**
1170  * Read a list of words from an input generator.
1171  * @return false if a syntax error was encountered.
1172  */
1174 {
1175  while (true)
1176  {
1177  res.push_back(std::string());
1178  input_status s = in.next(res.back());
1179  if (s == Success) continue;
1180  res.pop_back();
1181  return s == Eof;
1182  }
1183 }
1184 
1185 static bool read_words(std::istream &in, string_list &res)
1186 {
1187  input_generator gen(in, NULL);
1188  return read_words(gen, res);
1189 }
1190 
1191 /**
1192  * Generator for the result of function addprefix.
1193  */
1195 {
1198  string_list::const_iterator prei;
1199  size_t prej, prel;
1200  std::string suf;
1201  addprefix_generator(input_generator const &, bool &);
1202  input_status next(std::string &);
1203 };
1204 
1206  : gen(top.in, top.local_variables)
1207 {
1208  if (!read_words(gen, pre)) return;
1209  if (!expect_token(gen.in, Comma)) return;
1210  prej = 0;
1211  prel = pre.size();
1212  ok = true;
1213 }
1214 
1216 {
1217  if (prej)
1218  {
1219  produce:
1220  if (prej == prel)
1221  {
1222  res = *prei + suf;
1223  prej = 0;
1224  }
1225  else
1226  {
1227  res = *prei++;
1228  ++prej;
1229  }
1230  return Success;
1231  }
1232  switch (gen.next(res))
1233  {
1234  case Success:
1235  if (!prel) return Success;
1236  prei = pre.begin();
1237  prej = 1;
1238  suf = res;
1239  goto produce;
1240  case Eof:
1241  return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1242  default:
1243  return SyntaxError;
1244  }
1245 }
1246 
1247 /**
1248  * Generator for the result of function addsuffix.
1249  */
1251 {
1254  string_list::const_iterator sufi;
1255  size_t sufj, sufl;
1256  std::string pre;
1257  addsuffix_generator(input_generator const &, bool &);
1258  input_status next(std::string &);
1259 };
1260 
1262  : gen(top.in, top.local_variables)
1263 {
1264  if (!read_words(gen, suf)) return;
1265  if (!expect_token(gen.in, Comma)) return;
1266  sufj = 0;
1267  sufl = suf.size();
1268  ok = true;
1269 }
1270 
1272 {
1273  if (sufj)
1274  {
1275  if (sufj != sufl)
1276  {
1277  res = *sufi++;
1278  ++sufj;
1279  return Success;
1280  }
1281  sufj = 0;
1282  }
1283  switch (gen.next(res))
1284  {
1285  case Success:
1286  if (!sufl) return Success;
1287  sufi = suf.begin();
1288  sufj = 1;
1289  res += *sufi++;
1290  return Success;
1291  case Eof:
1292  return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1293  default:
1294  return SyntaxError;
1295  }
1296 }
1297 
1298 /**
1299  * Return a generator for function @a name.
1300  */
1301 generator *get_function(input_generator const &in, std::string const &name)
1302 {
1303  skip_spaces(in.in);
1304  generator *g = NULL;
1305  bool ok = false;
1306  if (name == "addprefix") g = new addprefix_generator(in, ok);
1307  else if (name == "addsuffix") g = new addsuffix_generator(in, ok);
1308  if (!g || ok) return g;
1309  delete g;
1310  return NULL;
1311 }
1312 
1313 /** @} */
1314 
1315 /**
1316  * @defgroup database Dependency database
1317  *
1318  * @{
1319  */
1320 
1321 /**
1322  * Load dependencies from @a in.
1323  */
1324 static void load_dependencies(std::istream &in)
1325 {
1326  if (false)
1327  {
1328  error:
1329  std::cerr << "Failed to load database" << std::endl;
1330  exit(EXIT_FAILURE);
1331  }
1332 
1333  while (!in.eof())
1334  {
1335  string_list targets;
1336  if (!read_words(in, targets)) goto error;
1337  if (in.eof()) return;
1338  if (targets.empty()) goto error;
1339  DEBUG << "reading dependencies of target " << targets.front() << std::endl;
1340  if (in.get() != ':') goto error;
1342  dep->targets = targets;
1343  string_list deps;
1344  if (!read_words(in, deps)) goto error;
1345  dep->deps.insert(deps.begin(), deps.end());
1346  for (string_list::const_iterator i = targets.begin(),
1347  i_end = targets.end(); i != i_end; ++i)
1348  {
1349  dependencies[*i] = dep;
1350  }
1351  skip_empty(in);
1352  }
1353 }
1354 
1355 /**
1356  * Load known dependencies from file <tt>.remake</tt>.
1357  */
1358 static void load_dependencies()
1359 {
1360  DEBUG_open << "Loading database... ";
1361  std::ifstream in(".remake");
1362  if (!in.good())
1363  {
1364  DEBUG_close << "not found\n";
1365  return;
1366  }
1367  load_dependencies(in);
1368 }
1369 
1370 
1371 /**
1372  * Save all the dependencies in file <tt>.remake</tt>.
1373  */
1374 static void save_dependencies()
1375 {
1376  DEBUG_open << "Saving database... ";
1377  std::ofstream db(".remake");
1378  while (!dependencies.empty())
1379  {
1380  ref_ptr<dependency_t> dep = dependencies.begin()->second;
1381  for (string_list::const_iterator i = dep->targets.begin(),
1382  i_end = dep->targets.end(); i != i_end; ++i)
1383  {
1384  db << escape_string(*i) << ' ';
1385  dependencies.erase(*i);
1386  }
1387  db << ':';
1388  for (string_set::const_iterator i = dep->deps.begin(),
1389  i_end = dep->deps.end(); i != i_end; ++i)
1390  {
1391  db << ' ' << escape_string(*i);
1392  }
1393  db << std::endl;
1394  }
1395 }
1396 
1397 /** @} */
1398 
1399 /**
1400  * @defgroup parser Rule parser
1401  *
1402  * @{
1403  */
1404 
1405 /**
1406  * Register a specific rule with an empty script:
1407  *
1408  * - Check that none of the targets already has an associated rule with a
1409  * nonempty script.
1410  * - Create a new rule with a single target for each target, if needed.
1411  * - Add the prerequisites of @a rule to all these associated rules.
1412  */
1413 static void register_transparent_rule(rule_t const &rule, string_list const &targets)
1414 {
1415  assert(rule.script.empty());
1416  for (string_list::const_iterator i = targets.begin(),
1417  i_end = targets.end(); i != i_end; ++i)
1418  {
1419  std::pair<rule_map::iterator, bool> j =
1420  specific_rules.insert(std::make_pair(*i, ref_ptr<rule_t>()));
1421  ref_ptr<rule_t> &r = j.first->second;
1422  if (j.second)
1423  {
1424  r = ref_ptr<rule_t>(rule);
1425  r->targets = string_list(1, *i);
1426  continue;
1427  }
1428  if (!r->script.empty())
1429  {
1430  std::cerr << "Failed to load rules: " << *i
1431  << " cannot be the target of several rules" << std::endl;
1432  exit(EXIT_FAILURE);
1433  }
1434  assert(r->targets.size() == 1 && r->targets.front() == *i);
1435  r->deps.insert(r->deps.end(), rule.deps.begin(), rule.deps.end());
1436  r->vars.insert(r->vars.end(), rule.vars.begin(), rule.vars.end());
1437  }
1438 
1439  for (string_list::const_iterator i = targets.begin(),
1440  i_end = targets.end(); i != i_end; ++i)
1441  {
1443  if (dep->targets.empty()) dep->targets.push_back(*i);
1444  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1445  }
1446 }
1447 
1448 /**
1449  * Register a specific rule with a nonempty script:
1450  *
1451  * - Check that none of the targets already has an associated rule.
1452  * - Create a single shared rule and associate it to all the targets.
1453  * - Merge the prerequisites of all the targets into a single set and
1454  * add the prerequisites of the rule to it. (The preexisting
1455  * prerequisites, if any, come from a previous run.)
1456  */
1457 static void register_scripted_rule(rule_t const &rule)
1458 {
1459  ref_ptr<rule_t> r(rule);
1460  for (string_list::const_iterator i = rule.targets.begin(),
1461  i_end = rule.targets.end(); i != i_end; ++i)
1462  {
1463  std::pair<rule_map::iterator, bool> j =
1464  specific_rules.insert(std::make_pair(*i, r));
1465  if (j.second) continue;
1466  std::cerr << "Failed to load rules: " << *i
1467  << " cannot be the target of several rules" << std::endl;
1468  exit(EXIT_FAILURE);
1469  }
1470 
1472  dep->targets = rule.targets;
1473  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1474  for (string_list::const_iterator i = rule.targets.begin(),
1475  i_end = rule.targets.end(); i != i_end; ++i)
1476  {
1478  dep->deps.insert(d->deps.begin(), d->deps.end());
1479  d = dep;
1480  }
1481 }
1482 
1483 /**
1484  * Read a rule starting with target @a first, if nonempty.
1485  * Store into #generic_rules or #specific_rules depending on its genericity.
1486  */
1487 static void load_rule(std::istream &in, std::string const &first)
1488 {
1489  DEBUG_open << "Reading rule for target " << first << "... ";
1490  if (false)
1491  {
1492  error:
1493  DEBUG_close << "failed\n";
1494  std::cerr << "Failed to load rules: syntax error" << std::endl;
1495  exit(EXIT_FAILURE);
1496  }
1497  rule_t rule;
1498 
1499  // Read targets and check genericity.
1500  string_list targets;
1501  if (!read_words(in, targets)) goto error;
1502  if (!first.empty()) targets.push_front(first);
1503  else if (targets.empty()) goto error;
1504  else DEBUG << "actual target: " << targets.front() << std::endl;
1505  bool generic = false;
1506  normalize_list(targets);
1507  for (string_list::const_iterator i = targets.begin(),
1508  i_end = targets.end(); i != i_end; ++i)
1509  {
1510  if (i->empty()) goto error;
1511  if ((i->find('%') != std::string::npos) != generic)
1512  {
1513  if (i == targets.begin()) generic = true;
1514  else goto error;
1515  }
1516  }
1517  std::swap(rule.targets, targets);
1518  skip_spaces(in);
1519  if (in.get() != ':') goto error;
1520 
1521  bool assignment = false;
1522 
1523  // Read dependencies.
1524  if (expect_token(in, Word))
1525  {
1526  std::string d = read_word(in);
1527  if (int tok = expect_token(in, Equal | Plusequal))
1528  {
1529  rule.vars.push_back(assign_t());
1530  string_list v;
1531  if (!read_words(in, v)) goto error;
1532  assign_t &a = rule.vars.back();
1533  a.name = d;
1534  a.append = tok == Plusequal;
1535  a.value.swap(v);
1536  assignment = true;
1537  }
1538  else
1539  {
1540  string_list v;
1541  if (!read_words(in, v)) goto error;
1542  v.push_front(d);
1543  normalize_list(v);
1544  rule.deps.swap(v);
1545  }
1546  }
1547  else
1548  {
1549  string_list v;
1550  if (!read_words(in, v)) goto error;
1551  normalize_list(v);
1552  rule.deps.swap(v);
1553  }
1554  skip_spaces(in);
1555  if (!skip_eol(in, true)) goto error;
1556 
1557  // Read script.
1558  std::ostringstream buf;
1559  while (true)
1560  {
1561  char c = in.get();
1562  if (!in.good()) break;
1563  if (c == '\t' || c == ' ')
1564  {
1565  in.get(*buf.rdbuf());
1566  if (in.fail() && !in.eof()) in.clear();
1567  }
1568  else if (c == '\r' || c == '\n')
1569  buf << c;
1570  else
1571  {
1572  in.putback(c);
1573  break;
1574  }
1575  }
1576  rule.script = buf.str();
1577 
1578  // Add generic rules to the correct set.
1579  if (generic)
1580  {
1581  if (assignment) goto error;
1582  generic_rules.push_back(rule);
1583  return;
1584  }
1585 
1586  if (!rule.script.empty())
1587  {
1588  if (assignment) goto error;
1589  register_scripted_rule(rule);
1590  }
1591  else
1592  {
1593  // Swap away the targets to avoid costly copies when registering.
1594  string_list targets;
1595  std::swap(rule.targets, targets);
1596  register_transparent_rule(rule, targets);
1597  std::swap(rule.targets, targets);
1598  }
1599 
1600  // If there is no default target yet, mark it as such.
1601  if (first_target.empty())
1602  first_target = rule.targets.front();
1603 }
1604 
1605 /**
1606  * Load rules from @a remakefile.
1607  * If some rules have dependencies and non-generic targets, add these
1608  * dependencies to the targets.
1609  */
1610 static void load_rules(std::string const &remakefile)
1611 {
1612  DEBUG_open << "Loading rules... ";
1613  if (false)
1614  {
1615  error:
1616  std::cerr << "Failed to load rules: syntax error" << std::endl;
1617  exit(EXIT_FAILURE);
1618  }
1619  std::ifstream in(remakefile.c_str());
1620  if (!in.good())
1621  {
1622  std::cerr << "Failed to load rules: no Remakefile found" << std::endl;
1623  exit(EXIT_FAILURE);
1624  }
1625  skip_empty(in);
1626 
1627  // Read rules
1628  while (in.good())
1629  {
1630  char c = in.peek();
1631  if (c == '#')
1632  {
1633  while (in.get() != '\n') {}
1634  skip_empty(in);
1635  continue;
1636  }
1637  if (c == ' ' || c == '\t') goto error;
1638  if (expect_token(in, Word))
1639  {
1640  std::string name = read_word(in);
1641  if (name.empty()) goto error;
1642  if (int tok = expect_token(in, Equal | Plusequal))
1643  {
1644  DEBUG << "Assignment to variable " << name << std::endl;
1645  string_list value;
1646  if (!read_words(in, value)) goto error;
1647  string_list &dest = variables[name];
1648  if (tok == Equal) dest.swap(value);
1649  else dest.splice(dest.end(), value);
1650  if (!skip_eol(in, true)) goto error;
1651  }
1652  else load_rule(in, name);
1653  }
1654  else load_rule(in, std::string());
1655  }
1656 }
1657 
1658 /** @} */
1659 
1660 /**
1661  * @defgroup rules Rule resolution
1662  *
1663  * @{
1664  */
1665 
1666 /**
1667  * Substitute a pattern into a list of strings.
1668  */
1669 static void substitute_pattern(std::string const &pat, string_list const &src, string_list &dst)
1670 {
1671  for (string_list::const_iterator i = src.begin(),
1672  i_end = src.end(); i != i_end; ++i)
1673  {
1674  size_t pos = i->find('%');
1675  if (pos == std::string::npos)dst.push_back(*i);
1676  else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
1677  }
1678 }
1679 
1680 /**
1681  * Find a generic rule matching @a target:
1682  * - the one leading to shorter matches has priority,
1683  * - among equivalent rules, the earliest one has priority.
1684  */
1685 static rule_t find_generic_rule(std::string const &target)
1686 {
1687  size_t tlen = target.length(), plen = tlen + 1;
1688  rule_t rule;
1689  for (rule_list::const_iterator i = generic_rules.begin(),
1690  i_end = generic_rules.end(); i != i_end; ++i)
1691  {
1692  for (string_list::const_iterator j = i->targets.begin(),
1693  j_end = i->targets.end(); j != j_end; ++j)
1694  {
1695  size_t len = j->length();
1696  if (tlen < len) continue;
1697  if (plen <= tlen - (len - 1)) continue;
1698  size_t pos = j->find('%');
1699  if (pos == std::string::npos) continue;
1700  size_t len2 = len - (pos + 1);
1701  if (j->compare(0, pos, target, 0, pos) ||
1702  j->compare(pos + 1, len2, target, tlen - len2, len2))
1703  continue;
1704  plen = tlen - (len - 1);
1705  rule = rule_t();
1706  rule.stem = target.substr(pos, plen);
1707  rule.script = i->script;
1708  substitute_pattern(rule.stem, i->targets, rule.targets);
1709  substitute_pattern(rule.stem, i->deps, rule.deps);
1710  break;
1711  }
1712  }
1713  return rule;
1714 }
1715 
1716 /**
1717  * Find a specific rule matching @a target. Return a generic one otherwise.
1718  * If there is both a specific rule with an empty script and a generic rule, the
1719  * generic one is returned after adding the dependencies of the specific one.
1720  */
1721 static rule_t find_rule(std::string const &target)
1722 {
1723  rule_map::const_iterator i = specific_rules.find(target),
1724  i_end = specific_rules.end();
1725  // If there is a specific rule with a script, return it.
1726  if (i != i_end && !i->second->script.empty()) return *i->second;
1727  rule_t grule = find_generic_rule(target);
1728  // If there is no generic rule, return the specific rule (no script), if any.
1729  if (grule.targets.empty())
1730  {
1731  if (i != i_end) return *i->second;
1732  return grule;
1733  }
1734  // Optimize the lookup when there is only one target (already looked up).
1735  if (grule.targets.size() == 1)
1736  {
1737  if (i == i_end) return grule;
1738  grule.deps.insert(grule.deps.end(),
1739  i->second->deps.begin(), i->second->deps.end());
1740  grule.vars.insert(grule.vars.end(),
1741  i->second->vars.begin(), i->second->vars.end());
1742  return grule;
1743  }
1744  // Add the dependencies of the specific rules of every target to the
1745  // generic rule. If any of those rules has a nonempty script, error out.
1746  for (string_list::const_iterator j = grule.targets.begin(),
1747  j_end = grule.targets.end(); j != j_end; ++j)
1748  {
1749  i = specific_rules.find(*j);
1750  if (i == i_end) continue;
1751  if (!i->second->script.empty()) return rule_t();
1752  grule.deps.insert(grule.deps.end(),
1753  i->second->deps.begin(), i->second->deps.end());
1754  grule.vars.insert(grule.vars.end(),
1755  i->second->vars.begin(), i->second->vars.end());
1756  }
1757  return grule;
1758 }
1759 
1760 /** @} */
1761 
1762 /**
1763  * @defgroup status Target status
1764  *
1765  * @{
1766  */
1767 
1768 /**
1769  * Compute and memoize the status of @a target:
1770  * - if the file does not exist, the target is obsolete,
1771  * - if any dependency is obsolete or younger than the file, it is obsolete,
1772  * - otherwise it is up-to-date.
1773  *
1774  * @note For rules with multiple targets, all the targets share the same
1775  * status. (If one is obsolete, they all are.) The second rule above
1776  * is modified in that case: the latest target is chosen, not the oldest!
1777  */
1778 static status_t const &get_status(std::string const &target)
1779 {
1780  std::pair<status_map::iterator,bool> i =
1781  status.insert(std::make_pair(target, status_t()));
1782  status_t &ts = i.first->second;
1783  if (!i.second) return ts;
1784  DEBUG_open << "Checking status of " << target << "... ";
1785  dependency_map::const_iterator j = dependencies.find(target);
1786  if (j == dependencies.end())
1787  {
1788  struct stat s;
1789  if (stat(target.c_str(), &s) != 0)
1790  {
1791  DEBUG_close << "missing\n";
1792  ts.status = Todo;
1793  ts.last = 0;
1794  return ts;
1795  }
1796  DEBUG_close << "up-to-date\n";
1797  ts.status = Uptodate;
1798  ts.last = s.st_mtime;
1799  return ts;
1800  }
1801  dependency_t const &dep = *j->second;
1802  status_e st = Uptodate;
1803  time_t latest = 0;
1804  for (string_list::const_iterator k = dep.targets.begin(),
1805  k_end = dep.targets.end(); k != k_end; ++k)
1806  {
1807  struct stat s;
1808  if (stat(k->c_str(), &s) != 0)
1809  {
1810  if (st == Uptodate) DEBUG_close << *k << " missing\n";
1811  s.st_mtime = 0;
1812  st = Todo;
1813  }
1814  status[*k].last = s.st_mtime;
1815  if (s.st_mtime > latest) latest = s.st_mtime;
1816  }
1817  if (st == Todo) goto update;
1818  for (string_set::const_iterator k = dep.deps.begin(),
1819  k_end = dep.deps.end(); k != k_end; ++k)
1820  {
1821  status_t const &ts_ = get_status(*k);
1822  if (latest < ts_.last)
1823  {
1824  DEBUG_close << "older than " << *k << std::endl;
1825  st = Todo;
1826  goto update;
1827  }
1828  if (ts_.status == Uptodate) continue;
1829  if (st == Uptodate)
1830  DEBUG << "obsolete dependency " << *k << std::endl;
1831  st = Recheck;
1832  }
1833  if (st == Uptodate) DEBUG_close << "all siblings up-to-date\n";
1834  update:
1835  for (string_list::const_iterator k = dep.targets.begin(),
1836  k_end = dep.targets.end(); k != k_end; ++k)
1837  {
1838  status[*k].status = st;
1839  }
1840  return ts;
1841 }
1842 
1843 /**
1844  * Change the status of @a target to #Remade or #Uptodate depending on whether
1845  * its modification time changed.
1846  */
1847 static void update_status(std::string const &target)
1848 {
1849  DEBUG_open << "Rechecking status of " << target << "... ";
1850  status_map::iterator i = status.find(target);
1851  assert(i != status.end());
1852  status_t &ts = i->second;
1853  ts.status = Remade;
1854  if (ts.last >= now)
1855  {
1856  DEBUG_close << "possibly remade\n";
1857  return;
1858  }
1859  struct stat s;
1860  if (stat(target.c_str(), &s) != 0)
1861  {
1862  DEBUG_close << "missing\n";
1863  ts.last = 0;
1864  }
1865  else if (s.st_mtime != ts.last)
1866  {
1867  DEBUG_close << "remade\n";
1868  ts.last = s.st_mtime;
1869  }
1870  else
1871  {
1872  DEBUG_close << "unchanged\n";
1873  ts.status = Uptodate;
1874  }
1875 }
1876 
1877 /**
1878  * Check whether all the prerequisites of @a target ended being up-to-date.
1879  */
1880 static bool still_need_rebuild(std::string const &target)
1881 {
1882  DEBUG_open << "Rechecking obsoleteness of " << target << "... ";
1883  status_map::const_iterator i = status.find(target);
1884  assert(i != status.end());
1885  if (i->second.status != Recheck) return true;
1886  dependency_map::const_iterator j = dependencies.find(target);
1887  assert(j != dependencies.end());
1888  dependency_t const &dep = *j->second;
1889  for (string_set::const_iterator k = dep.deps.begin(),
1890  k_end = dep.deps.end(); k != k_end; ++k)
1891  {
1892  if (status[*k].status != Uptodate) return true;
1893  }
1894  for (string_list::const_iterator k = dep.targets.begin(),
1895  k_end = dep.targets.end(); k != k_end; ++k)
1896  {
1897  status[*k].status = Uptodate;
1898  }
1899  DEBUG_close << "no longer obsolete\n";
1900  return false;
1901 }
1902 
1903 /** @} */
1904 
1905 /**
1906  * @defgroup server Server
1907  *
1908  * @{
1909  */
1910 
1911 /**
1912  * Handle job completion.
1913  */
1914 static void complete_job(int job_id, bool success)
1915 {
1916  DEBUG_open << "Completing job " << job_id << "... ";
1917  job_targets_map::iterator i = job_targets.find(job_id);
1918  assert(i != job_targets.end());
1919  string_list const &targets = i->second;
1920  if (success)
1921  {
1922  for (string_list::const_iterator j = targets.begin(),
1923  j_end = targets.end(); j != j_end; ++j)
1924  {
1925  update_status(*j);
1926  }
1927  }
1928  else
1929  {
1930  DEBUG_close << "failed\n";
1931  std::cerr << "Failed to build";
1932  for (string_list::const_iterator j = targets.begin(),
1933  j_end = targets.end(); j != j_end; ++j)
1934  {
1935  status[*j].status = Failed;
1936  std::cerr << ' ' << *j;
1937  remove(j->c_str());
1938  }
1939  std::cerr << std::endl;
1940  }
1941  job_targets.erase(i);
1942 }
1943 
1944 /**
1945  * Return the script obtained by substituting variables.
1946  */
1947 static std::string prepare_script(rule_t const &rule)
1948 {
1949  std::string const &s = rule.script;
1950  std::istringstream in(s);
1951  std::ostringstream out;
1952  size_t len = s.size();
1953 
1954  while (!in.eof())
1955  {
1956  size_t pos = in.tellg(), p = s.find('$', pos);
1957  if (p == std::string::npos || p == len - 1) p = len;
1958  out.write(&s[pos], p - pos);
1959  if (p == len) break;
1960  ++p;
1961  switch (s[p])
1962  {
1963  case '$':
1964  out << '$';
1965  in.seekg(p + 1);
1966  break;
1967  case '<':
1968  if (!rule.deps.empty())
1969  out << rule.deps.front();
1970  in.seekg(p + 1);
1971  break;
1972  case '^':
1973  {
1974  bool first = true;
1975  for (string_list::const_iterator i = rule.deps.begin(),
1976  i_end = rule.deps.end(); i != i_end; ++i)
1977  {
1978  if (first) first = false;
1979  else out << ' ';
1980  out << *i;
1981  }
1982  in.seekg(p + 1);
1983  break;
1984  }
1985  case '@':
1986  assert(!rule.targets.empty());
1987  out << rule.targets.front();
1988  in.seekg(p + 1);
1989  break;
1990  case '*':
1991  out << rule.stem;
1992  in.seekg(p + 1);
1993  break;
1994  case '(':
1995  {
1996  in.seekg(p - 1);
1997  bool first = true;
1998  input_generator gen(in, &rule.vars, true);
1999  while (true)
2000  {
2001  std::string w;
2002  input_status s = gen.next(w);
2003  if (s == SyntaxError)
2004  {
2005  // TODO
2006  return "false";
2007  }
2008  if (s == Eof) break;
2009  if (first) first = false;
2010  else out << ' ';
2011  out << w;
2012  }
2013  break;
2014  }
2015  default:
2016  // Let dollars followed by an unrecognized character
2017  // go through. This differs from Make, which would
2018  // use a one-letter variable.
2019  out << '$';
2020  in.seekg(p);
2021  }
2022  }
2023 
2024  return out.str();
2025 }
2026 
2027 /**
2028  * Execute the script from @a rule.
2029  */
2030 static bool run_script(int job_id, rule_t const &rule)
2031 {
2032  if (show_targets)
2033  {
2034  std::cout << "Building";
2035  for (string_list::const_iterator i = rule.targets.begin(),
2036  i_end = rule.targets.end(); i != i_end; ++i)
2037  {
2038  std::cout << ' ' << *i;
2039  }
2040  std::cout << std::endl;
2041  }
2042 
2044  dep->targets = rule.targets;
2045  dep->deps.insert(rule.deps.begin(), rule.deps.end());
2046  for (string_list::const_iterator i = rule.targets.begin(),
2047  i_end = rule.targets.end(); i != i_end; ++i)
2048  {
2049  dependencies[*i] = dep;
2050  }
2051 
2052  std::string script = prepare_script(rule);
2053 
2054  std::ostringstream job_id_buf;
2055  job_id_buf << job_id;
2056  std::string job_id_ = job_id_buf.str();
2057 
2058  DEBUG_open << "Starting script for job " << job_id << "... ";
2059  if (false)
2060  {
2061  error:
2062  DEBUG_close << "failed\n";
2063  complete_job(job_id, false);
2064  return false;
2065  }
2066 
2067 #ifdef WINDOWS
2068  HANDLE pfd[2];
2069  if (false)
2070  {
2071  error2:
2072  CloseHandle(pfd[0]);
2073  CloseHandle(pfd[1]);
2074  goto error;
2075  }
2076  if (!CreatePipe(&pfd[0], &pfd[1], NULL, 0))
2077  goto error;
2078  if (!SetHandleInformation(pfd[0], HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT))
2079  goto error2;
2080  STARTUPINFO si;
2081  ZeroMemory(&si, sizeof(STARTUPINFO));
2082  si.cb = sizeof(STARTUPINFO);
2083  si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
2084  si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
2085  si.hStdInput = pfd[0];
2086  si.dwFlags |= STARTF_USESTDHANDLES;
2087  PROCESS_INFORMATION pi;
2088  ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
2089  if (!SetEnvironmentVariable("REMAKE_JOB_ID", job_id_.c_str()))
2090  goto error2;
2091  char const *argv = echo_scripts ? "SH.EXE -e -s -v" : "SH.EXE -e -s";
2092  if (!CreateProcess(NULL, (char *)argv, NULL, NULL,
2093  true, 0, NULL, NULL, &si, &pi))
2094  {
2095  goto error2;
2096  }
2097  CloseHandle(pi.hThread);
2098  DWORD len = script.length(), wlen;
2099  if (!WriteFile(pfd[1], script.c_str(), len, &wlen, NULL) || wlen < len)
2100  std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2101  CloseHandle(pfd[0]);
2102  CloseHandle(pfd[1]);
2103  ++running_jobs;
2104  job_pids[pi.hProcess] = job_id;
2105  return true;
2106 #else
2107  int pfd[2];
2108  if (false)
2109  {
2110  error2:
2111  close(pfd[0]);
2112  close(pfd[1]);
2113  goto error;
2114  }
2115  if (pipe(pfd) == -1)
2116  goto error;
2117  if (pid_t pid = fork())
2118  {
2119  if (pid == -1) goto error2;
2120  ssize_t len = script.length();
2121  if (write(pfd[1], script.c_str(), len) < len)
2122  std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2123  close(pfd[0]);
2124  close(pfd[1]);
2125  ++running_jobs;
2126  job_pids[pid] = job_id;
2127  return true;
2128  }
2129  // Child process starts here.
2130  if (setenv("REMAKE_JOB_ID", job_id_.c_str(), 1))
2131  _exit(EXIT_FAILURE);
2132  char const *argv[5] = { "sh", "-e", "-s", NULL, NULL };
2133  if (echo_scripts) argv[3] = "-v";
2134  if (pfd[0] != 0)
2135  {
2136  dup2(pfd[0], 0);
2137  close(pfd[0]);
2138  }
2139  close(pfd[1]);
2140  execv("/bin/sh", (char **)argv);
2141  _exit(EXIT_FAILURE);
2142 #endif
2143 }
2144 
2145 /**
2146  * Create a job for @a target according to the loaded rules.
2147  * Mark all the targets from the rule as running and reset their dependencies.
2148  * If the rule has dependencies, create a new client to build them just
2149  * before @a current, and change @a current so that it points to it.
2150  */
2151 static bool start(std::string const &target, client_list::iterator &current)
2152 {
2153  DEBUG_open << "Starting job " << job_counter << " for " << target << "... ";
2154  rule_t rule = find_rule(target);
2155  if (rule.targets.empty())
2156  {
2157  status[target].status = Failed;
2158  DEBUG_close << "failed\n";
2159  std::cerr << "No rule for building " << target << std::endl;
2160  return false;
2161  }
2162  for (string_list::const_iterator i = rule.targets.begin(),
2163  i_end = rule.targets.end(); i != i_end; ++i)
2164  {
2165  status[*i].status = Running;
2166  }
2167  int job_id = job_counter++;
2168  job_targets[job_id] = rule.targets;
2169  if (!rule.deps.empty())
2170  {
2171  current = clients.insert(current, client_t());
2172  current->job_id = job_id;
2173  current->pending = rule.deps;
2174  current->delayed = new rule_t(rule);
2175  return true;
2176  }
2177  return run_script(job_id, rule);
2178 }
2179 
2180 /**
2181  * Send a reply to a client then remove it.
2182  * If the client was a dependency client, start the actual script.
2183  */
2184 static void complete_request(client_t &client, bool success)
2185 {
2186  DEBUG_open << "Completing request from client of job " << client.job_id << "... ";
2187  if (client.delayed)
2188  {
2189  assert(client.socket == INVALID_SOCKET);
2190  if (success)
2191  {
2192  if (still_need_rebuild(client.delayed->targets.front()))
2193  run_script(client.job_id, *client.delayed);
2194  else complete_job(client.job_id, true);
2195  }
2196  else complete_job(client.job_id, false);
2197  delete client.delayed;
2198  }
2199  else if (client.socket != INVALID_SOCKET)
2200  {
2201  char res = success ? 1 : 0;
2202  send(client.socket, &res, 1, 0);
2203  #ifdef WINDOWS
2204  closesocket(client.socket);
2205  #else
2206  close(client.socket);
2207  #endif
2208  --waiting_jobs;
2209  }
2210 
2211  if (client.job_id < 0 && !success) build_failure = true;
2212 }
2213 
2214 /**
2215  * Return whether there are slots for starting new jobs.
2216  */
2217 static bool has_free_slots()
2218 {
2219  if (max_active_jobs <= 0) return true;
2221 }
2222 
2223 /**
2224  * Handle client requests:
2225  * - check for running targets that have finished,
2226  * - start as many pending targets as allowed,
2227  * - complete the request if there are neither running nor pending targets
2228  * left or if any of them failed.
2229  *
2230  * @return true if some child processes are still running.
2231  *
2232  * @post If there are pending requests, at least one child process is running.
2233  */
2234 static bool handle_clients()
2235 {
2236  DEBUG_open << "Handling client requests... ";
2237  restart:
2238 
2239  for (client_list::iterator i = clients.begin(), i_next = i,
2240  i_end = clients.end(); i != i_end && has_free_slots(); i = i_next)
2241  {
2242  ++i_next;
2243  DEBUG_open << "Handling client from job " << i->job_id << "... ";
2244  if (false)
2245  {
2246  failed:
2247  complete_request(*i, false);
2248  clients.erase(i);
2249  DEBUG_close << "failed\n";
2250  continue;
2251  }
2252 
2253  // Remove running targets that have finished.
2254  for (string_set::iterator j = i->running.begin(), j_next = j,
2255  j_end = i->running.end(); j != j_end; j = j_next)
2256  {
2257  ++j_next;
2258  status_map::const_iterator k = status.find(*j);
2259  assert(k != status.end());
2260  switch (k->second.status)
2261  {
2262  case Running:
2263  break;
2264  case Failed:
2265  if (!keep_going) goto failed;
2266  i->failed = true;
2267  // no break
2268  case Uptodate:
2269  case Remade:
2270  i->running.erase(j);
2271  break;
2272  case Recheck:
2273  case Todo:
2274  assert(false);
2275  }
2276  }
2277 
2278  // Start pending targets.
2279  while (!i->pending.empty())
2280  {
2281  std::string target = i->pending.front();
2282  i->pending.pop_front();
2283  switch (get_status(target).status)
2284  {
2285  case Running:
2286  i->running.insert(target);
2287  break;
2288  case Failed:
2289  pending_failed:
2290  if (!keep_going) goto failed;
2291  i->failed = true;
2292  // no break
2293  case Uptodate:
2294  case Remade:
2295  break;
2296  case Recheck:
2297  case Todo:
2298  client_list::iterator j = i;
2299  if (!start(target, i)) goto pending_failed;
2300  j->running.insert(target);
2301  if (!has_free_slots()) return true;
2302  // Job start might insert a dependency client.
2303  i_next = i;
2304  ++i_next;
2305  break;
2306  }
2307  }
2308 
2309  // Try to complete the request.
2310  // (This might start a new job if it was a dependency client.)
2311  if (i->running.empty())
2312  {
2313  if (i->failed) goto failed;
2314  complete_request(*i, true);
2315  clients.erase(i);
2316  DEBUG_close << "finished\n";
2317  }
2318  }
2319 
2320  if (running_jobs != waiting_jobs) return true;
2321  if (running_jobs == 0 && clients.empty()) return false;
2322 
2323  // There is a circular dependency.
2324  // Try to break it by completing one of the requests.
2325  assert(!clients.empty());
2326  std::cerr << "Circular dependency detected" << std::endl;
2327  client_list::iterator i = clients.begin();
2328  complete_request(*i, false);
2329  clients.erase(i);
2330  goto restart;
2331 }
2332 
2333 /**
2334  * Create a named unix socket that listens for build requests. Also set
2335  * the REMAKE_SOCKET environment variable that will be inherited by all
2336  * the job scripts.
2337  */
2338 static void create_server()
2339 {
2340  if (false)
2341  {
2342  error:
2343  perror("Failed to create server");
2344 #ifndef WINDOWS
2345  error2:
2346 #endif
2347  exit(EXIT_FAILURE);
2348  }
2349  DEBUG_open << "Creating server... ";
2350 
2351 #ifdef WINDOWS
2352  // Prepare a windows socket.
2353  struct sockaddr_in socket_addr;
2354  socket_addr.sin_family = AF_INET;
2355  socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2356  socket_addr.sin_port = 0;
2357 
2358  // Create and listen to the socket.
2359  socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2360  if (socket_fd == INVALID_SOCKET) goto error;
2361  if (!SetHandleInformation((HANDLE)socket_fd, HANDLE_FLAG_INHERIT, 0))
2362  goto error;
2363  if (bind(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2364  goto error;
2365  int len = sizeof(sockaddr_in);
2366  if (getsockname(socket_fd, (struct sockaddr *)&socket_addr, &len))
2367  goto error;
2368  std::ostringstream buf;
2369  buf << socket_addr.sin_port;
2370  if (!SetEnvironmentVariable("REMAKE_SOCKET", buf.str().c_str()))
2371  goto error;
2372  if (listen(socket_fd, 1000)) goto error;
2373 #else
2374  // Set signal handlers for SIGCHLD and SIGINT.
2375  // Block SIGCHLD (unblocked during select).
2376  sigset_t sigmask;
2377  sigemptyset(&sigmask);
2378  sigaddset(&sigmask, SIGCHLD);
2379  if (sigprocmask(SIG_BLOCK, &sigmask, NULL) == -1) goto error;
2380  struct sigaction sa;
2381  sa.sa_flags = 0;
2382  sigemptyset(&sa.sa_mask);
2383  sa.sa_handler = &sigchld_handler;
2384  if (sigaction(SIGCHLD, &sa, NULL) == -1) goto error;
2385  sa.sa_handler = &sigint_handler;
2386  if (sigaction(SIGINT, &sa, NULL) == -1) goto error;
2387 
2388  // Prepare a named unix socket in temporary directory.
2389  socket_name = tempnam(NULL, "rmk-");
2390  if (!socket_name) goto error2;
2391  struct sockaddr_un socket_addr;
2392  size_t len = strlen(socket_name);
2393  if (len >= sizeof(socket_addr.sun_path) - 1) goto error2;
2394  socket_addr.sun_family = AF_UNIX;
2395  strcpy(socket_addr.sun_path, socket_name);
2396  len += sizeof(socket_addr.sun_family);
2397  if (setenv("REMAKE_SOCKET", socket_name, 1)) goto error;
2398 
2399  // Create and listen to the socket.
2400 #ifdef LINUX
2401  socket_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
2402  if (socket_fd == INVALID_SOCKET) goto error;
2403 #else
2404  socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2405  if (socket_fd == INVALID_SOCKET) goto error;
2406  if (fcntl(socket_fd, F_SETFD, FD_CLOEXEC) < 0) goto error;
2407 #endif
2408  if (bind(socket_fd, (struct sockaddr *)&socket_addr, len))
2409  goto error;
2410  if (listen(socket_fd, 1000)) goto error;
2411 #endif
2412 }
2413 
2414 /**
2415  * Accept a connection from a client, get the job it spawned from,
2416  * get the targets, and mark them as dependencies of the job targets.
2417  */
2419 {
2420  DEBUG_open << "Handling client request... ";
2421 
2422  // Accept connection.
2423 #ifdef WINDOWS
2424  socket_t fd = accept(socket_fd, NULL, NULL);
2425  if (fd == INVALID_SOCKET) return;
2426  if (!SetHandleInformation((HANDLE)fd, HANDLE_FLAG_INHERIT, 0))
2427  {
2428  error2:
2429  std::cerr << "Unexpected failure while setting connection with client" << std::endl;
2430  closesocket(fd);
2431  return;
2432  }
2433  // WSAEventSelect puts sockets into nonblocking mode, so disable it here.
2434  u_long nbio = 0;
2435  if (ioctlsocket(fd, FIONBIO, &nbio)) goto error2;
2436 #elif defined(LINUX)
2437  int fd = accept4(socket_fd, NULL, NULL, SOCK_CLOEXEC);
2438  if (fd < 0) return;
2439 #else
2440  int fd = accept(socket_fd, NULL, NULL);
2441  if (fd < 0) return;
2442  if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) return;
2443 #endif
2444  clients.push_front(client_t());
2445  client_list::iterator proc = clients.begin();
2446 
2447  if (false)
2448  {
2449  error:
2450  DEBUG_close << "failed\n";
2451  std::cerr << "Received an ill-formed client message" << std::endl;
2452  #ifdef WINDOWS
2453  closesocket(fd);
2454  #else
2455  close(fd);
2456  #endif
2457  clients.erase(proc);
2458  return;
2459  }
2460 
2461  // Receive message. Stop when encountering two nuls in a row.
2462  std::vector<char> buf;
2463  size_t len = 0;
2464  while (len < sizeof(int) + 2 || buf[len - 1] || buf[len - 2])
2465  {
2466  buf.resize(len + 1024);
2467  ssize_t l = recv(fd, &buf[0] + len, 1024, 0);
2468  if (l <= 0) goto error;
2469  len += l;
2470  }
2471 
2472  // Parse job that spawned the client.
2473  int job_id;
2474  memcpy(&job_id, &buf[0], sizeof(int));
2475  proc->socket = fd;
2476  proc->job_id = job_id;
2477  job_targets_map::const_iterator i = job_targets.find(job_id);
2478  if (i == job_targets.end()) goto error;
2479  DEBUG << "receiving request from job " << job_id << std::endl;
2480 
2481  // Parse the targets and mark them as dependencies from the job targets.
2482  dependency_t &dep = *dependencies[job_targets[job_id].front()];
2483  char const *p = &buf[0] + sizeof(int);
2484  while (true)
2485  {
2486  len = strlen(p);
2487  if (len == 0)
2488  {
2489  ++waiting_jobs;
2490  return;
2491  }
2492  std::string target(p, p + len);
2493  DEBUG << "adding dependency " << target << " to job\n";
2494  proc->pending.push_back(target);
2495  dep.deps.insert(target);
2496  p += len + 1;
2497  }
2498 }
2499 
2500 /**
2501  * Handle child process exit status.
2502  */
2503 void finalize_job(pid_t pid, bool res)
2504 {
2505  pid_job_map::iterator i = job_pids.find(pid);
2506  assert(i != job_pids.end());
2507  int job_id = i->second;
2508  job_pids.erase(i);
2509  --running_jobs;
2510  complete_job(job_id, res);
2511 }
2512 
2513 /**
2514  * Loop until all the jobs have finished.
2515  *
2516  * @post There are no client requests left, not even virtual ones.
2517  */
2519 {
2520  while (handle_clients())
2521  {
2522  DEBUG_open << "Handling events... ";
2523  #ifdef WINDOWS
2524  size_t len = job_pids.size() + 1;
2525  HANDLE h[len];
2526  int num = 0;
2527  for (pid_job_map::const_iterator i = job_pids.begin(),
2528  i_end = job_pids.end(); i != i_end; ++i, ++num)
2529  {
2530  h[num] = i->first;
2531  }
2532  WSAEVENT aev = WSACreateEvent();
2533  h[num] = aev;
2534  WSAEventSelect(socket_fd, aev, FD_ACCEPT);
2535  DWORD w = WaitForMultipleObjects(len, h, false, INFINITE);
2536  WSAEventSelect(socket_fd, aev, 0);
2537  WSACloseEvent(aev);
2538  if (len <= w)
2539  continue;
2540  if (w == len - 1)
2541  {
2542  accept_client();
2543  continue;
2544  }
2545  pid_t pid = h[w];
2546  DWORD s = 0;
2547  bool res = GetExitCodeProcess(pid, &s) && s == 0;
2548  CloseHandle(pid);
2549  finalize_job(pid, res);
2550  #else
2551  sigset_t emptymask;
2552  sigemptyset(&emptymask);
2553  fd_set fdset;
2554  FD_ZERO(&fdset);
2555  FD_SET(socket_fd, &fdset);
2556  int ret = pselect(socket_fd + 1, &fdset, NULL, NULL, NULL, &emptymask);
2557  if (ret > 0 /* && FD_ISSET(socket_fd, &fdset)*/) accept_client();
2558  if (!got_SIGCHLD) continue;
2559  got_SIGCHLD = 0;
2560  pid_t pid;
2561  int status;
2562  while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
2563  {
2564  bool res = WIFEXITED(status) && WEXITSTATUS(status) == 0;
2565  finalize_job(pid, res);
2566  }
2567  #endif
2568  }
2569 
2570  assert(clients.empty());
2571 }
2572 
2573 /**
2574  * Load dependencies and rules, listen to client requests, and loop until
2575  * all the requests have completed.
2576  * If Remakefile is obsolete, perform a first run with it only, then reload
2577  * the rules, and perform a second with the original clients.
2578  */
2579 void server_mode(std::string const &remakefile, string_list const &targets)
2580 {
2582  load_rules(remakefile);
2583  create_server();
2584  if (get_status(remakefile).status != Uptodate)
2585  {
2586  clients.push_back(client_t());
2587  clients.back().pending.push_back(remakefile);
2588  server_loop();
2589  if (build_failure) goto early_exit;
2590  variables.clear();
2591  specific_rules.clear();
2592  generic_rules.clear();
2593  first_target.clear();
2594  load_rules(remakefile);
2595  }
2596  clients.push_back(client_t());
2597  if (!targets.empty()) clients.back().pending = targets;
2598  else if (!first_target.empty())
2599  clients.back().pending.push_back(first_target);
2600  server_loop();
2601  early_exit:
2602  close(socket_fd);
2603 #ifndef WINDOWS
2604  remove(socket_name);
2605  free(socket_name);
2606 #endif
2608  exit(build_failure ? EXIT_FAILURE : EXIT_SUCCESS);
2609 }
2610 
2611 /** @} */
2612 
2613 /**
2614  * @defgroup client Client
2615  *
2616  * @{
2617  */
2618 
2619 /**
2620  * Connect to the server @a socket_name, send a build request for @a targets,
2621  * and exit with the status returned by the server.
2622  */
2623 void client_mode(char *socket_name, string_list const &targets)
2624 {
2625  if (false)
2626  {
2627  error:
2628  perror("Failed to send targets to server");
2629  exit(EXIT_FAILURE);
2630  }
2631  if (targets.empty()) exit(EXIT_SUCCESS);
2632  DEBUG_open << "Connecting to server... ";
2633 
2634  // Connect to server.
2635 #ifdef WINDOWS
2636  struct sockaddr_in socket_addr;
2637  socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2638  if (socket_fd == INVALID_SOCKET) goto error;
2639  socket_addr.sin_family = AF_INET;
2640  socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2641  socket_addr.sin_port = atoi(socket_name);
2642  if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2643  goto error;
2644 #else
2645  struct sockaddr_un socket_addr;
2646  size_t len = strlen(socket_name);
2647  if (len >= sizeof(socket_addr.sun_path) - 1) exit(EXIT_FAILURE);
2648  socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2649  if (socket_fd == INVALID_SOCKET) goto error;
2650  socket_addr.sun_family = AF_UNIX;
2651  strcpy(socket_addr.sun_path, socket_name);
2652  if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(socket_addr.sun_family) + len))
2653  goto error;
2654 #ifdef MACOSX
2655  int set_option = 1;
2656  if (setsockopt(socket_fd, SOL_SOCKET, SO_NOSIGPIPE, &set_option, sizeof(set_option)))
2657  goto error;
2658 #endif
2659 #endif
2660 
2661  // Send current job id.
2662  char *id = getenv("REMAKE_JOB_ID");
2663  int job_id = id ? atoi(id) : -1;
2664  if (send(socket_fd, (char *)&job_id, sizeof(job_id), MSG_NOSIGNAL) != sizeof(job_id))
2665  goto error;
2666 
2667  // Send tagets.
2668  for (string_list::const_iterator i = targets.begin(),
2669  i_end = targets.end(); i != i_end; ++i)
2670  {
2671  DEBUG_open << "Sending " << *i << "... ";
2672  ssize_t len = i->length() + 1;
2673  if (send(socket_fd, i->c_str(), len, MSG_NOSIGNAL) != len)
2674  goto error;
2675  }
2676 
2677  // Send terminating nul and wait for reply.
2678  char result = 0;
2679  if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2680  if (recv(socket_fd, &result, 1, 0) != 1) exit(EXIT_FAILURE);
2681  exit(result ? EXIT_SUCCESS : EXIT_FAILURE);
2682 }
2683 
2684 /** @} */
2685 
2686 /**
2687  * @defgroup ui User interface
2688  *
2689  * @{
2690  */
2691 
2692 /**
2693  * Display usage and exit with @a exit_status.
2694  */
2695 void usage(int exit_status)
2696 {
2697  std::cerr << "Usage: remake [options] [target] ...\n"
2698  "Options\n"
2699  " -d Echo script commands.\n"
2700  " -d -d Print lots of debugging information.\n"
2701  " -f FILE Read FILE as Remakefile.\n"
2702  " -h, --help Print this message and exit.\n"
2703  " -j[N], --jobs=[N] Allow N jobs at once; infinite jobs with no arg.\n"
2704  " -k Keep going when some targets cannot be made.\n"
2705  " -r Look up targets from the dependencies on standard input.\n"
2706  " -s, --silent, --quiet Do not echo targets.\n";
2707  exit(exit_status);
2708 }
2709 
2710 /**
2711  * This program behaves in two different ways.
2712  *
2713  * - If the environment contains the REMAKE_SOCKET variable, the client
2714  * connects to this socket and sends to the server its build targets.
2715  * It exits once it receives the server reply.
2716  *
2717  * - Otherwise, it creates a server that waits for build requests. It
2718  * also creates a pseudo-client that requests the targets passed on the
2719  * command line.
2720  */
2721 int main(int argc, char *argv[])
2722 {
2723  init_working_dir();
2724 
2725  std::string remakefile = "Remakefile";
2726  string_list targets;
2727  bool indirect_targets = false;
2728 
2729  // Parse command-line arguments.
2730  for (int i = 1; i < argc; ++i)
2731  {
2732  std::string arg = argv[i];
2733  if (arg.empty()) usage(EXIT_FAILURE);
2734  if (arg == "-h" || arg == "--help") usage(EXIT_SUCCESS);
2735  if (arg == "-d")
2736  if (echo_scripts) debug.active = true;
2737  else echo_scripts = true;
2738  else if (arg == "-k" || arg =="--keep-going")
2739  keep_going = true;
2740  else if (arg == "-s" || arg == "--silent" || arg == "--quiet")
2741  show_targets = false;
2742  else if (arg == "-r")
2743  indirect_targets = true;
2744  else if (arg == "-f")
2745  {
2746  if (++i == argc) usage(EXIT_FAILURE);
2747  remakefile = argv[i];
2748  }
2749  else if (arg.compare(0, 2, "-j") == 0)
2750  max_active_jobs = atoi(arg.c_str() + 2);
2751  else if (arg.compare(0, 7, "--jobs=") == 0)
2752  max_active_jobs = atoi(arg.c_str() + 7);
2753  else
2754  {
2755  if (arg[0] == '-') usage(EXIT_FAILURE);
2756  targets.push_back(normalize(arg));
2757  DEBUG << "New target: " << arg << '\n';
2758  }
2759  }
2760 
2761  if (indirect_targets)
2762  {
2763  load_dependencies(std::cin);
2764  string_list l;
2765  targets.swap(l);
2766  if (l.empty() && !dependencies.empty())
2767  {
2768  l.push_back(dependencies.begin()->second->targets.front());
2769  }
2770  for (string_list::const_iterator i = l.begin(),
2771  i_end = l.end(); i != i_end; ++i)
2772  {
2773  dependency_map::const_iterator j = dependencies.find(*i);
2774  if (j == dependencies.end()) continue;
2775  dependency_t const &dep = *j->second;
2776  for (string_set::const_iterator k = dep.deps.begin(),
2777  k_end = dep.deps.end(); k != k_end; ++k)
2778  {
2779  targets.push_back(normalize(*k));
2780  }
2781  }
2782  dependencies.clear();
2783  }
2784 
2785 #ifdef WINDOWS
2786  WSADATA wsaData;
2787  if (WSAStartup(MAKEWORD(2,2), &wsaData))
2788  {
2789  std::cerr << "Unexpected failure while initializing Windows Socket" << std::endl;
2790  return 1;
2791  }
2792 #endif
2793 
2794  // Run as client if REMAKE_SOCKET is present in the environment.
2795  if (char *sn = getenv("REMAKE_SOCKET")) client_mode(sn, targets);
2796 
2797  // Otherwise run as server.
2798  server_mode(remakefile, targets);
2799 }
2800 
2801 /** @} */