Configure separate compile and link pools for ninja. Configures link
parallelism based on expected memory use, which can be reduced by
setting IMPALA_MINIMAL_DEBUG_INFO=true or IMPALA_SPLIT_DEBUG_INFO=true.
Adds IMPALA_MAKE_CMD to simplify using the ninja build tool for all make
operations in scripts. Install ninja on Ubuntu. Adds a '-make' option to
buildall.sh to force using 'make'.
Adds MOLD_JOBS=1 to avoid overloading the system when trying 'mold' and
linking test binaries. However 'mold' is not selected as the default
due to test failures around SASL/GSSAPI (see IMPALA-14527).
Switches bin/jenkins/all-tests.sh to use ninja and removes the guard in
bootstrap_development.sh limiting IMPALA_BUILD_THREADS as it's no longer
needed with ninja.
SKIP_BE_TEST_PATTERN in run-backend-tests is unused (only used with
TARGET_FILESYSTEM=local) so I don't attempt to make it work with ninja.
Tested with local 'IMPALA_SPLIT_DEBUG_INFO=true buildall.sh -skiptests'
with default (make) and IMPALA_MAKE_CMD=ninja.
Change-Id: I0952dc19ace5c9c42bed0d2ffb61499656c0a2db
Reviewed-on: http://gerrit.cloudera.org:8080/23572
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Reviewed-by: Pranav Lodha <pranav.lodha@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Updates IMPALA_BUILD_THREADS to bound it based on guideline of 2 GB
memory per core during builds. Computes cores and memory from cgroup
limits if applicable; memory is used as a bound on physical memory, as
sometimes cgroups will report a larger limit than available physical
memory.
Uses IMPALA_BUILD_THREADS for load-data.
Adds a default in case USER is unset during bootstrap, which can occur
in devcontainer.
Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Reviewed-on: http://gerrit.cloudera.org:8080/21200
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Riza Suminto <riza.suminto@cloudera.com>
Many scripts source bin/impala-config.sh to get necessary
environment variables. The print statements in bin/impala-config.sh
for those scripts are not interesting and make the build logs
noisier.
This changes a variety of build scripts / utility scripts to
silence the output of sourcing bin/impala-config.sh. This continues
to print the output for invocations of buildall.sh.
Testing:
- Ran a build and looked at the output
Change-Id: Ib4e39f50c7efb8c42a6d3597be0e18c4c79457c5
Reviewed-on: http://gerrit.cloudera.org:8080/20098
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Reviewed-by: Yifan Zhang <chinazhangyifan@163.com>
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Since it's not straight-forward to get Java 8 for Ubuntu 14.04, I'm
removing it from the setups supported by the bootstrap scripts.
Change-Id: I4b6d63c9713cc93a786f82e086feca1a91937e16
Reviewed-on: http://gerrit.cloudera.org:8080/9873
Reviewed-by: Thomas Tauber-Marshall <tmarshall@cloudera.com>
Tested-by: Impala Public Jenkins
This commit instructs bootstrap_development.sh, which is
used in Gerrity-Verify-Dryrun jobs, to clean before
running the build. In the common case, this is essentially
a no-op, because the workspace is new to begin with.
This handles an issue where GVDs would fail if they
happened to run on a machine where a previous build ran
against a workspace where the fe/generated-sources
directories were incompatible. I confirmed that
our clean script does clean out fe/generated-sources
nicely.
Change-Id: I238528ca3938200f0750505a62113630290e8c96
Reviewed-on: http://gerrit.cloudera.org:8080/9841
Reviewed-by: Philip Zeyliger <philip@cloudera.com>
Tested-by: Impala Public Jenkins
This commit bundles two changes.
The first extracts from bootstrap_development.sh the commands to
prepare a system to build-and-test without actually doing so. This
enables custom build commands that might not load the test data, for
instance.
The second changes bootstrap_build.sh to work with Ubuntu 16.04. It
should still work with Ubuntu 14.04, but I don't anticipate that being
part of the Jenkins pre-merge job anymore, so I removed that from the
comment at the top of the file explaining what it does.
Change-Id: I8196a2a87bce5893a349a1b290c3f3d04fd80317
Reviewed-on: http://gerrit.cloudera.org:8080/8262
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Jim Apple <jbapple-impala@apache.org>
Having recently gone through setting up Eclipse for frontend debugging,
I'm amending the process a tad to make it smoother for new developers.
* Adds the JDK sources package to the developer bootstrap
process. These are useful if you need to point Eclipse to the source
of Java libraries and are harmless to everyone else.
* Defaults to "downloadSource" for the "mvn eclipse:eclipse" plugin.
This was being set explicitly in the Wiki instructions, but
there's no reason not to set it on by default for everyone.
* Adds a "launcher" file for debugging the FE. Instead of creating
a debug configuration manually, it's possible to share these across
developers by checking them in. I checked it in via a mechanism
that "mvn eclipse:eclipse" has to create additional files, to
avoid literring the directory for non-Eclipse users.
* Changes the output directory used by Eclipse's compiler to
a different one than Maven uses. Because we start Impala
from the commandline, it's easy to start Impala and then
change a line of Java, which forces Eclipse to kick in and
replace the .class files. When this happens, the existing Impala
starts complaining about class not found, because the underlying
files have been overwritten. It's preferable to separate the
directories and rely on Maven for the compiled Java.
Change-Id: Ia723fbf706cf409a8fb6b5ff0297c2b1ff7c9590
Reviewed-on: http://gerrit.cloudera.org:8080/7803
Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Tested-by: Impala Public Jenkins
This patch adds quadratic backoff to apt-get, which can fail for
network reasons.
While I'm here, fix bug in call to diff: diff returns failure when
there is a difference, but it is used in this script only for the
output, not the return value.
Change-Id: I2829064c9f27c39f13529b3d1a1e4f7d7a0e0128
Reviewed-on: http://gerrit.cloudera.org:8080/7657
Reviewed-by: Michael Brown <mikeb@cloudera.com>
Tested-by: Impala Public Jenkins
Before this change, Impala has relied on a chef setup in
https://github.com/awleblang/impala-setup for setting up a development
environment. This has a number of downsides:
1. It makes understanding what the script is doing difficult: there
are 40k or so lines in that repo last I checked.
2. It makes porting to new distributions difficult unless the
providers of various chef "recipes" have already ported their code.
3. It makes coordinated changes between the main repo and the
impala-setup repo more awkward.
This patch adds a shell script to replace that repo. It works on
Ubuntu 14.04 and 16.04, while impala-setup repo only works on 14.04
and the now-unmaintained Ubuntu 15.04.
Change-Id: I728abfa806ecd9461dfb443278c2a464714d984c
Reviewed-on: http://gerrit.cloudera.org:8080/7587
Reviewed-by: Jim Apple <jbapple-impala@apache.org>
Tested-by: Impala Public Jenkins
From bash's manual page on redirecting with '>'
Redirection of output causes the file whose name results from the
expansion of word to be opened for writing on file descriptor n,
or the standard output (file descriptor 1) if n is not specified.
If the file does not exist it is created; if it does exist it is
truncated to zero size.
Change-Id: I0d1a56441fcb5a2a2aed043fc1ece866c5d8287a
Reviewed-on: http://gerrit.cloudera.org:8080/4967
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Impala Public Jenkins
This script bootstraps an Impala dev environment on Ubuntu 14.04. It
is not hermetic -- it changes some config files for the user and for
the OS.
It is green on Jenkins, and it runs in about 6.5 hours. The intention
is to have this script run in a CI tool for post-commit testing, with
the hope that this will make it easier for new developers to get a
working development environment. Previously, the new developer
workflow lived on wiki pages and tended to bit-rot.
Still left to do: migrating the install script into the official
Impala repo.
Change-Id: If166a8a286d7559af547da39f6cc09e723f34c7e
Reviewed-on: http://gerrit.cloudera.org:8080/4674
Reviewed-by: Jim Apple <jbapple@cloudera.com>
Tested-by: Internal Jenkins
