jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-11856: Use POST requests to set log level

Browse Source 
Set and reset loglevel handlers now require a POST. Implements
Cross-Site Request Forgery (CSRF) prevention in Impala's webserver using
the Double Submit Cookie pattern - where POST requests must include a
csrf_token field in their post with the random value from the cookie -
or a custom header.

CSRF attacks rely on the browser always sending a cookie or
'Authorization: Basic' header.
- With cookies, attacks don't have access to default form values or the
  original cookie, so we can include the cookie's random value in the
  form as a cross-check. As the cookie is cryptographically signed, they
  also can't be replaced with one that would match an attack's forms.
- When not using cookies, a custom header (X-Requested-By) is required
  as CSRFs are unable to add custom headers. This approach is also used
  by Jersey; see
  http://blog.alutam.com/2011/09/14/jersey-and-cross-site-request-forgery-csrf

In a broader implementation this would require a separate cookie so it
can be used to protect logins as well, but login is handled external to
Impala so we re-use the cookie the page already has.

Cookies are now generated for the HTPASSWD authentication method.
Authenticating via JWT still omits cookies because the JWT is already
provided via custom header (preventing CSRF) and disabling
authentication (NONE) means anyone could directly send a request so CSRF
protection is meaningless.

We also start an additional Webserver instance with authentication NONE
when metrics_webserver_port > 0, and the Webserver metric
"impala.webserver.total-cookie-auth-success" can only be registered
once. Additional changes would be necessary to make metric names unique
in Webserver (based on port); for the moment we avoid that by ensuring
all metrics counters are only instantiated for Webservers that use
authentication.

Cookie generation and authentication were updated to provide access to
the random value.

Adds flag to enable SameSite=Strict for defense in depth as mentioned in
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis.
This can be enabled if another CSRF attack method is found.

Verified that this prevents CSRF attacks by disabling SameSite=Strict
and visiting (via https://security.love/CSRF-PoC-Genorator):
```
<html>
  <form enctype="application/x-www-form-urlencoded" method="POST"
        action="http://localhost:45000/set_glog_level">
    <table>
      <tr>
        <td>glog</td>
        <td><input type="text" value="1" name="glog"></td>
      </tr>
    </table>
    <input type="submit" value="http://localhost:45000/set_glog_level">
  </form>
</html>
```

Adds tests for the webserver with basic authentication, LDAP, and SPNEGO
that authorization fails on POST unless
- using a cookie and csrf_token is correctly set in the POST body
- the X-Requested-By header is set

Change-Id: I4be8694492b8ba16737f644ac8c56d8124f19693
Reviewed-on: http://gerrit.cloudera.org:8080/19199
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Michael Smith
2022-11-03 09:23:54 -07:00
committed by Impala Public Jenkins
parent 026f6993ed
commit 86d33a0a3d
18 changed files with 802 additions and 271 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
be/src/rpc/authentication-util.cc
View File

@@ -35,6 +35,8 @@ DEFINE_bool(cookie_require_secure, true,
DEFINE_int64(max_cookie_lifetime_s, 24 * 60 * 60,
"Maximum amount of time in seconds that an authentication cookie will remain valid. "
"Setting to 0 disables use of cookies. Defaults to 1 day.");
DEFINE_bool(samesite_strict, false,
"(Advanced) If true, authentication cookies will include SameSite=Strict.");
using namespace strings;
@@ -59,13 +61,14 @@ static const int SHA256_BASE64_LEN =
CalculateBase64EscapedLen(AuthenticationHash::HashLen(), /* do_padding */ true);
// Since we only return cookies with a single name, well behaved clients should only ever
// return one cookie to us. To accomodate non-malicious but poorly behaved clients, we
// return one cookie to us. To accommodate non-malicious but poorly behaved clients, we
// allow for checking a limited number of cookies, up to MAX_COOKIES_TO_CHECK or until we
// find the first one with COOKIE_NAME.
static const int MAX_COOKIES_TO_CHECK = 5;
Status AuthenticateCookie(
const AuthenticationHash& hash, const string& cookie_header, string* username) {
const AuthenticationHash& hash, const string& cookie_header,
string* username, string* rand) {
// The 'Cookie' header allows sending multiple name/value pairs separated by ';'.
vector<string> cookies = strings::Split(cookie_header, ";");
if (cookies.size() > MAX_COOKIES_TO_CHECK) {
@@ -125,6 +128,11 @@ Status AuthenticateCookie(
if (!TryStripPrefixString(cookie_value_split[0], USERNAME_KEY, username)) {
return Status("The cookie username value has an invalid format.");
}
if (rand != nullptr) {
if (!TryStripPrefixString(cookie_value_split[2], RAND_KEY, rand)) {
return Status("The cookie rand value has an invalid format.");
}
}
// We've successfully authenticated.
return Status::OK();
} else {
@@ -135,12 +143,18 @@ Status AuthenticateCookie(
return Status(Substitute("Did not find expected cookie name: $0", COOKIE_NAME));
}
string GenerateCookie(const string& username, const AuthenticationHash& hash) {
string GenerateCookie(const string& username, const AuthenticationHash& hash,
std::string* srand) {
// Its okay to use rand() here even though its a weak RNG because being able to guess
// the random numbers generated won't help an attacker. The important thing is that
// we're using a strong RNG to create the key and a strong HMAC function.
int cookie_rand = rand();
string cookie_rand_s = std::to_string(cookie_rand);
if (srand != nullptr) {
*srand = cookie_rand_s;
}
string cookie_value = StrCat(USERNAME_KEY, username, COOKIE_SEPARATOR, TIMESTAMP_KEY,
MonotonicMillis(), COOKIE_SEPARATOR, RAND_KEY, rand());
MonotonicMillis(), COOKIE_SEPARATOR, RAND_KEY, cookie_rand_s);
uint8_t signature[AuthenticationHash::HashLen()];
Status compute_status =
hash.Compute(reinterpret_cast<const uint8_t*>(cookie_value.data()),
@@ -155,12 +169,13 @@ string GenerateCookie(const string& username, const AuthenticationHash& hash) {
SHA256_BASE64_LEN, /* do_padding */ true);
base64_signature[SHA256_BASE64_LEN] = '\0';
const char* secure_flag = ";Secure";
if (!FLAGS_cookie_require_secure) {
secure_flag = "";
}
return Substitute("$0=$1$2$3;HttpOnly;Max-Age=$4$5", COOKIE_NAME, base64_signature,
COOKIE_SEPARATOR, cookie_value, FLAGS_max_cookie_lifetime_s, secure_flag);
const char* secure_flag = FLAGS_cookie_require_secure ? ";Secure" : "";
const char* samesite_flag = FLAGS_samesite_strict ? ";SameSite=Strict" : "";
// Add SameSite=Strict to notify the browser it should avoid sending the cookie with
// requests from other domains.
return Substitute("$0=$1$2$3;HttpOnly;Max-Age=$4$5$6", COOKIE_NAME, base64_signature,
COOKIE_SEPARATOR, cookie_value, FLAGS_max_cookie_lifetime_s, secure_flag,
samesite_flag);
}
string GetDeleteCookie() {

14
be/src/rpc/authentication-util.h
View File

@@ -25,13 +25,15 @@ class AuthenticationHash;
// Takes a single 'key=value' pair from a 'Cookie' header and attempts to verify its
// signature with 'hash'. If verification is successful and the cookie is still valid,
// sets 'username' to the corresponding username and returns OK.
Status AuthenticateCookie(const AuthenticationHash& hash,
const std::string& cookie_header, std::string* username);
// sets 'username' and 'rand' (if specified) to the corresponding values and returns OK.
Status AuthenticateCookie(
const AuthenticationHash& hash, const std::string& cookie_header,
std::string* username, std::string* rand = nullptr);
// Generates and returns a cookie containing the username set on 'connection_context' and
// a signature generated with 'hash'.
std::string GenerateCookie(const std::string& username, const AuthenticationHash& hash);
// a signature generated with 'hash'. If specified, sets 'rand' to the 'r=' cookie value.
std::string GenerateCookie(const std::string& username, const AuthenticationHash& hash,
std::string* rand = nullptr);
// Returns a empty cookie. Returned in a 'Set-Cookie' when cookie auth fails to indicate
// to the client that the cookie should be deleted.
@@ -51,4 +53,6 @@ bool IsTrustedDomain(const std::string& origin, const std::string& trusted_domai
Status BasicAuthExtractCredentials(
const string& token, string& username, string& password);
constexpr int RAND_MAX_LENGTH = 10;
} // namespace impala

24
be/src/util/logging-support.cc
View File

@@ -149,7 +149,12 @@ void GetJavaLogLevels(Document* document) {
// Callback handler for /set_java_loglevel.
void SetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* document) {
const auto& args = req.parsed_args;
if (req.request_method != "POST") {
AddDocumentMember("Use form input to update java log levels", "error", document);
return;
}
Webserver::ArgumentMap args = Webserver::GetVars(req.post_data);
Webserver::ArgumentMap::const_iterator classname = args.find("class");
Webserver::ArgumentMap::const_iterator level = args.find("level");
if (classname == args.end() || classname->second.empty() ||
@@ -179,6 +184,11 @@ void SetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* documen
// Callback handler for /reset_java_loglevel.
void ResetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* document) {
if (req.request_method != "POST") {
AddDocumentMember("Use form input to reset java log levels", "error", document);
return;
}
Status status = ResetJavaLogLevels();
if (!status.ok()) {
AddDocumentMember(status.GetDetail(), "error", document);
@@ -202,7 +212,12 @@ void GetGlogLevel(Document* document) {
// Callback handler for /set_glog_level
void SetGlogLevelCallback(const Webserver::WebRequest& req, Document* document) {
const auto& args = req.parsed_args;
if (req.request_method != "POST") {
AddDocumentMember("Use form input to update glog level", "error", document);
return;
}
Webserver::ArgumentMap args = Webserver::GetVars(req.post_data);
Webserver::ArgumentMap::const_iterator glog_level = args.find("glog");
if (glog_level == args.end() || glog_level->second.empty()) {
AddDocumentMember("Bad glog level input. Valid inputs are integers in the "
@@ -221,6 +236,11 @@ void SetGlogLevelCallback(const Webserver::WebRequest& req, Document* document)
// Callback handler for /reset_glog_level
void ResetGlogLevelCallback(const Webserver::WebRequest& req, Document* document) {
if (req.request_method != "POST") {
AddDocumentMember("Use form input to reset glog level", "error", document);
return;
}
string new_log_level = google::SetCommandLineOption("v",
to_string(FLAGS_v_original_value).data());
VLOG(1) << "New glog level set: " << new_log_level;

313
be/src/util/webserver-test.cc
View File

@@ -21,6 +21,7 @@
#include <boost/filesystem.hpp>
#include <boost/lexical_cast.hpp>
#include <gutil/strings/substitute.h>
#include <map>
#include <openssl/crypto.h>
#include <openssl/ssl.h>
#include <regex>
@@ -65,49 +66,74 @@ const string TO_ESCAPE_KEY = "ToEscape";
const string TO_ESCAPE_VALUE = "<script language='javascript'>";
const string ESCAPED_VALUE = "&lt;script language=&apos;javascript&apos;&gt;";
// Adapted from:
// http://stackoverflow.com/questions/10982717/get-html-without-header-with-boostasio
struct HttpRequest {
string url_path = "/";
string host = "localhost";
int32_t port = FLAGS_webserver_port;
map<string, string> headers = {};
// Adapted from:
// http://stackoverflow.com/questions/10982717/get-html-without-header-with-boostasio
Status Do(ostream* out, int expected_code, const string& method) {
try {
tcp::iostream request_stream;
request_stream.connect(host, lexical_cast<string>(port));
if (!request_stream) return Status("Could not connect request_stream");
request_stream << method << " " << url_path << " HTTP/1.1\r\n";
request_stream << "Host: " << host << ":" << port << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Cache-Control: no-cache\r\n";
if (method == "POST") {
request_stream << "Content-Length: 0\r\n";
}
for (const std::pair<string, string>& header : headers) {
request_stream << header.first << ": " << header.second << "\r\n";
}
request_stream << "Connection: close\r\n\r\n";
request_stream.flush();
string line1;
getline(request_stream, line1);
if (!request_stream) return Status("No response");
stringstream response_stream(line1);
string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
string status_message;
getline(response_stream, status_message);
if (!response_stream || http_version.substr(0,5) != "HTTP/") {
return Status("Malformed response");
}
if (status_code != expected_code) {
return Status(Substitute("Unexpected status code: $0", status_code));
}
(*out) << request_stream.rdbuf();
return Status::OK();
} catch (const std::exception& e){
return Status(e.what());
}
}
Status Get(ostream* out, int expected_code = 200) {
return Do(out, expected_code, "GET");
}
Status Post(ostream* out, int expected_code = 200) {
return Do(out, expected_code, "POST");
}
};
Status HttpGet(const string& host, const int32_t& port, const string& url_path,
ostream* out, int expected_code = 200, const string& method = "GET") {
try {
tcp::iostream request_stream;
request_stream.connect(host, lexical_cast<string>(port));
if (!request_stream) return Status("Could not connect request_stream");
request_stream << method << " " << url_path << " HTTP/1.1\r\n";
request_stream << "Host: " << host << ":" << port << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Cache-Control: no-cache\r\n";
request_stream << "Connection: close\r\n\r\n";
request_stream.flush();
string line1;
getline(request_stream, line1);
if (!request_stream) return Status("No response");
stringstream response_stream(line1);
string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
string status_message;
getline(response_stream,status_message);
if (!response_stream || http_version.substr(0,5) != "HTTP/") {
return Status("Malformed response");
}
if (status_code != expected_code) {
return Status(Substitute("Unexpected status code: $0", status_code));
}
(*out) << request_stream.rdbuf();
return Status::OK();
} catch (const std::exception& e){
return Status(e.what());
}
return HttpRequest{url_path, host, port}.Do(out, expected_code, method);
}
TEST(Webserver, SmokeTest) {
@@ -120,6 +146,30 @@ TEST(Webserver, SmokeTest) {
ASSERT_OK(HttpGet("localhost", FLAGS_webserver_port, "/", &contents));
}
void PostOnlyCallback(bool* success, const Webserver::WebRequest& req,
Document* document) {
*success = req.request_method == "POST";
}
TEST(Webserver, PostTest) {
MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics);
const string POST_TEST_PATH = "/post-test";
bool success = false;
Webserver::UrlCallback callback = bind<void>(PostOnlyCallback, &success , _1, _2);
webserver.RegisterUrlCallback(POST_TEST_PATH, "raw_text.tmpl", callback, false);
ASSERT_OK(webserver.Start());
stringstream contents;
HttpRequest req{POST_TEST_PATH};
ASSERT_OK(req.Get(&contents));
ASSERT_FALSE(success) << "GET unexpectedly succeeded";
ASSERT_OK(req.Post(&contents));
ASSERT_TRUE(success) << "POST unexpectedly failed";
}
void AssertArgsCallback(bool* success, const Webserver::WebRequest& req,
Document* document) {
const auto& args = req.parsed_args;
@@ -402,7 +452,68 @@ void CheckAuthMetrics(MetricGroup* metrics, int num_negotiate_success,
ASSERT_EQ(cookie_failure_metric->GetValue(), num_cookie_failure);
}
TEST(Webserver, TestWithSpnego) {
void curl_version(string* curl_output, bool* curl_7_64_or_above = nullptr) {
// TODO(todd) IMPALA-8987: import curl into native-toolchain and test this with
// authentication.
RunShellProcess("curl --version", curl_output);
// Detect curl version. We only care about the major and minor.
std::regex curl_version_regex = std::regex("curl ([0-9]+)\\.([0-9]+)\\.[0-9]+");
std::smatch match_result;
ASSERT_TRUE(std::regex_search(*curl_output, match_result, curl_version_regex));
ASSERT_EQ(match_result.size(), 3);
int curl_major_version = std::stoi(match_result[1]);
int curl_minor_version = std::stoi(match_result[2]);
if (curl_7_64_or_above) {
*curl_7_64_or_above = curl_major_version > 7 ||
(curl_major_version == 7 && curl_minor_version >= 64);
}
cout << "Detected curl version " << std::to_string(curl_major_version) << "."
<< std::to_string(curl_minor_version)
<< (curl_7_64_or_above == nullptr ? " " :
(*curl_7_64_or_above ? " is at least 7.64" : " is below 7.64")) << endl;
}
string curl(const string& curl_options, int32_t port = FLAGS_webserver_port) {
string cmd = Substitute("curl -v -f $0 'http://127.0.0.1:$1'", curl_options, port);
cout << cmd << endl;
return cmd;
}
class CookieJar {
public:
CookieJar() : dir_(filesystem::unique_path()), path_(dir_ / "cookiejar") {
filesystem::create_directories(dir_);
cout << "Storing cookies in " << path_ << endl;
}
~CookieJar() {
filesystem::remove_all(dir_);
}
const filesystem::path& path() { return path_; }
string token() {
const char* rand_key = "&r=";
string rand, line;
ifstream cookie_file(path_.string());
while (cookie_file) {
getline(cookie_file, line);
size_t rand_idx = line.rfind(rand_key);
if (rand_idx != string::npos) {
// Relies on the random value being the last element in the cookie.
rand = line.substr(rand_idx + strlen(rand_key));
break;
}
}
return rand;
}
private:
const filesystem::path dir_, path_;
};
void EmptyCallback(const Webserver::WebRequest& req, Document* document) { }
TEST(Webserver, TestGetWithSpnego) {
MiniKdc kdc(MiniKdcOptions{});
KUDU_ASSERT_OK(kdc.Start());
kdc.SetKrb5Environment();
@@ -420,6 +531,7 @@ TEST(Webserver, TestWithSpnego) {
MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics);
ASSERT_OK(webserver.Start());
webserver.RegisterUrlCallback("/", "raw_text.tmpl", EmptyCallback, false);
// Don't expect HTTP requests to work without Kerberos credentials.
stringstream contents;
@@ -428,46 +540,21 @@ TEST(Webserver, TestWithSpnego) {
// There should be one failed auth attempt.
CheckAuthMetrics(&metrics, 0, 1, 0, 0);
// TODO(todd) IMPALA-8987: import curl into native-toolchain and test this with
// authentication.
string curl_output;
RunShellProcess("curl --version", &curl_output);
// Detect curl version. We only care about the major and minor.
std::regex curl_version_regex = std::regex("curl ([0-9]+)\\.([0-9]+)\\.[0-9]+");
std::smatch match_result;
ASSERT_TRUE(std::regex_search(curl_output, match_result, curl_version_regex));
ASSERT_EQ(match_result.size(), 3);
int curl_major_version = std::stoi(match_result[1]);
int curl_minor_version = std::stoi(match_result[2]);
bool curl_7_64_or_above = true;
if (curl_major_version < 7 ||
(curl_major_version == 7 && curl_minor_version < 64)) {
curl_7_64_or_above = false;
}
LOG(INFO) << "Detected curl version " << std::to_string(curl_major_version) << "."
<< std::to_string(curl_minor_version) << " "
<< (curl_7_64_or_above ? "is at least 7.64" : "is below 7.64");
bool curl_7_64_or_above;
curl_version(&curl_output, &curl_7_64_or_above);
if (curl_output.find("GSS-API") != string::npos
&& curl_output.find("SPNEGO") != string::npos) {
// Test that OPTIONS works with and without having kinit-ed.
string options_cmd =
Substitute("curl -X OPTIONS -v --negotiate -u : 'http://127.0.0.1:$0'",
FLAGS_webserver_port);
string options_cmd = curl("-X OPTIONS --negotiate -u :");
system(options_cmd.c_str());
KUDU_ASSERT_OK(kdc.Kinit("alice"));
system(options_cmd.c_str());
// Test that GET works with cookies.
filesystem::path cookie_dir = filesystem::unique_path();
filesystem::create_directories(cookie_dir);
filesystem::path cookie_path = cookie_dir / "cookiejar";
LOG(INFO) << "Storing cookies in " << cookie_path;
CookieJar cookie;
string curl_cmd =
Substitute("curl -c $0 -b $0 -X GET -v --negotiate -u : 'http://127.0.0.1:$1'",
cookie_path.string(), FLAGS_webserver_port);
curl(Substitute("-c $0 -b $0 --negotiate -u :", cookie.path().string()));
// Run the command twice, the first time we should authenticate with SPNEGO, the
// second time with a cookie.
system(Substitute("$0 && $0", curl_cmd).c_str());
@@ -486,11 +573,56 @@ TEST(Webserver, TestWithSpnego) {
// webserver uses a different HMAC key. See above note about curl 7.64.0 or above.
system(curl_cmd.c_str());
CheckAuthMetrics(&metrics2, 1, (curl_7_64_or_above ? 0 : 1), 0, 1);
filesystem::remove_all(cookie_dir);
} else {
LOG(INFO) << "Skipping test, curl was not present or did not have the required "
<< "features: " << curl_output;
cout << "Skipping test, curl was not present or did not have the required "
<< "features: " << curl_output << endl;
}
}
TEST(Webserver, TestPostWithSpnego) {
MiniKdc kdc(MiniKdcOptions{});
KUDU_ASSERT_OK(kdc.Start());
kdc.SetKrb5Environment();
string kt_path;
KUDU_ASSERT_OK(kdc.CreateServiceKeytab("HTTP/127.0.0.1", &kt_path));
CHECK_ERR(setenv("KRB5_KTNAME", kt_path.c_str(), 1));
KUDU_ASSERT_OK(kdc.CreateUserPrincipal("alice"));
gflags::FlagSaver saver;
FLAGS_webserver_require_spnego = true;
FLAGS_webserver_ldap_passwords_in_clear_ok = true;
FLAGS_cookie_require_secure = false;
MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics);
ASSERT_OK(webserver.Start());
webserver.RegisterUrlCallback("/", "raw_text.tmpl", EmptyCallback, false);
string curl_output;
curl_version(&curl_output);
if (curl_output.find("GSS-API") != string::npos
&& curl_output.find("SPNEGO") != string::npos) {
// POST fails without a header
ASSERT_NE(system(curl("-d '' --negotiate -u :").c_str()), 0);
// POST succeeds with X-Requested-By header
ASSERT_EQ(system(curl("-d '' -H 'X-Requested-By: me' --negotiate -u :").c_str()), 0);
CookieJar cookie;
// GET with SPNEGO succeeds and returns a cookie.
ASSERT_EQ(system(curl("--negotiate -u : -c " + cookie.path().string()).c_str()), 0);
// Verify we got a cookie and can read the random token.
string token = cookie.token();
ASSERT_FALSE(token.empty());
// Post with the cookie fails due to CSRF protection.
ASSERT_NE(system(curl("-d '' -b " + cookie.path().string()).c_str()), 0);
// Include the cookie's random token as csrf_token and request should succeed.
ASSERT_EQ(system(curl(Substitute(
"-b $0 -d 'csrf_token=$1'", cookie.path().string(), token)).c_str()), 0);
} else {
cout << "Skipping test, curl was not present or did not have the required "
<< "features: " << curl_output << endl;
}
}
@@ -500,17 +632,46 @@ TEST(Webserver, StartWithPasswordFileTest) {
auto password =
ScopedFlagSetter<string>::Make(&FLAGS_webserver_password_file, password_file.str());
gflags::FlagSaver saver;
FLAGS_cookie_require_secure = false;
MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics);
if (FIPS_mode()) {
ASSERT_FALSE(webserver.Start().ok());
} else {
ASSERT_OK(webserver.Start());
webserver.RegisterUrlCallback("/", "raw_text.tmpl", EmptyCallback, false);
// Don't expect HTTP requests to work without a password
stringstream contents;
ASSERT_ERROR_MSG(HttpGet("localhost", FLAGS_webserver_port, "/", &contents),
"Unexpected status code: 401");
// Succeeds with user and password
string curl_output;
curl_version(&curl_output);
ASSERT_EQ(system(curl("--digest -u test:test").c_str()), 0);
// POST is rejected without header
ASSERT_NE(system(curl("-d '' --digest -u test:test").c_str()), 0);
ASSERT_EQ(system(
curl("-d '' --digest -u test:test -H 'X-Requested-By: me'").c_str()), 0);
CookieJar cookie;
// GET with user and password succeeds and returns a cookie.
ASSERT_EQ(system(curl(Substitute("--digest -u test:test -c $0",
cookie.path().string())).c_str()), 0);
// Verify we got a cookie and can read the random token.
string token = cookie.token();
ASSERT_FALSE(token.empty());
// Post with the cookie fails due to CSRF protection.
ASSERT_NE(system(curl(Substitute("--digest -u test:test -b $0 -d ''",
cookie.path().string())).c_str()), 0);
// Include the cookie's random token as csrf_token and request should succeed.
ASSERT_EQ(system(curl(Substitute("--digest -u test:test -b $0 -d 'csrf_token=$1'",
cookie.path().string(), token)).c_str()), 0);
}
}

111
be/src/util/webserver.cc
View File

@@ -321,7 +321,7 @@ Webserver::Webserver(const string& interface, const int port, MetricGroup* metri
total_basic_auth_failure_ =
metrics->AddCounter("impala.webserver.total-basic-auth-failure", 0);
}
if (use_cookies_ && (auth_mode_ == AuthMode::SPNEGO || auth_mode_ == AuthMode::LDAP)) {
if (use_cookies_ && auth_mode_ != AuthMode::NONE) {
total_cookie_auth_success_ =
metrics->AddCounter("impala.webserver.total-cookie-auth-success", 0);
total_cookie_auth_failure_ =
@@ -552,13 +552,18 @@ void Webserver::Init() {
"$0://$1:$2", IsSecure() ? "https" : "http", hostname_, http_address_.port);
}
void Webserver::GetCommonJson(
Document* document, const struct sq_connection* connection, const WebRequest& req) {
void Webserver::GetCommonJson(Document* document, const struct sq_connection* connection,
const WebRequest& req, const std::string& csrf_token) {
DCHECK(document != nullptr);
Value obj(kObjectType);
obj.AddMember("process-name",
rapidjson::StringRef(google::ProgramInvocationShortName()),
document->GetAllocator());
if (!csrf_token.empty()) {
obj.AddMember("csrf_token",
Value(csrf_token.c_str(), document->GetAllocator()),
document->GetAllocator());
}
// If Apacke Knox is being used to proxy connections to the webserver, the
// 'x-forwarded-context' header will be present.
@@ -637,6 +642,16 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
vector<string> response_headers;
bool authenticated = false;
// Random value from cookie that we'll also use as a csrf_token to implement the
// "Double Submit Cookie" and custom header (X-Requested-By) patterns for preventing
// cross-site request forgery (CSRF).
std::string cookie_rand_value;
// With JWTs we can skip CSRF protection because browsers won't send "Authorization:
// Bearer" headers automatically.
bool check_csrf_protection = true;
// Flags if we have a valid cookie to test for CSRF.
bool cookie_authenticated = false;
// Try authenticating with JWT token first, if enabled.
if (use_jwt_) {
const char* auth_value = nullptr;
@@ -654,6 +669,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (JWTTokenAuth(jwt_token, connection, request_info)) {
total_jwt_token_auth_success_->Increment(1);
authenticated = true;
check_csrf_protection = false;
// TODO: cookies are not added, but are not needed right now
} else {
LOG(INFO) << "Invalid JWT token provided: " << jwt_token;
@@ -662,17 +678,25 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
}
}
}
if (!authenticated) {
authenticated = auth_mode_ != AuthMode::SPNEGO && auth_mode_ != AuthMode::LDAP;
if (!authenticated && auth_mode_ == AuthMode::NONE) {
// With AuthMode::NONE, any protection can be bypassed. We sometimes initialize a 2nd
// Metrics webserver using AuthMode::NONE, and metrics counters are not named
// uniquely to work with two webservers using cookies so we skip using cookies.
authenticated = true;
check_csrf_protection = false;
}
// Try authenticating with a cookie, if enabled.
if (!authenticated && use_cookies_) {
const char* cookie_header = sq_get_header(connection, "Cookie");
string username;
if (cookie_header != nullptr) {
Status cookie_status = AuthenticateCookie(hash_, cookie_header, &username);
Status cookie_status =
AuthenticateCookie(hash_, cookie_header, &username, &cookie_rand_value);
if (cookie_status.ok()) {
authenticated = true;
cookie_authenticated = true;
request_info->remote_user = strdup(username.c_str());
total_cookie_auth_success_->Increment(1);
} else {
@@ -684,6 +708,14 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
}
}
if (!authenticated && auth_mode_ == AuthMode::HTPASSWD) {
// Squeasel already handled HTPASSWD authentication. We still enable CSRF protection
// as browsers automatically include HTPASSWD credentials in requests, so add and use
// cookies to avoid requiring the custom header.
authenticated = true;
AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
}
// Connections originating from trusted domains should not require authentication.
// Returns a cookie on the first successful auth attempt. This check is performed after
// checking for cookie to avoid subsequent reverse DNS lookups which can be
@@ -699,7 +731,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (TrustedDomainCheck(origin, connection, request_info)) {
total_trusted_domain_check_success_->Increment(1);
authenticated = true;
AddCookie(request_info, &response_headers);
AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
}
}
}
@@ -712,7 +744,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (GetUsernameFromAuthHeader(connection, request_info, err_msg)) {
total_trusted_auth_header_check_success_->Increment(1);
authenticated = true;
AddCookie(request_info, &response_headers);
AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} else {
LOG(ERROR) << "Found trusted auth header but " << err_msg;
}
@@ -725,7 +757,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
HandleSpnego(connection, request_info, &response_headers);
if (spnego_result == SQ_CONTINUE_HANDLING) {
// Spnego negotiation was successful.
AddCookie(request_info, &response_headers);
AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} else {
// Spnego negotiation is incomplete or failed, stop processing the request.
return spnego_result;
@@ -736,7 +768,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (basic_status.ok()) {
// Basic auth was successful.
total_basic_auth_success_->Increment(1);
AddCookie(request_info, &response_headers);
AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} else {
total_basic_auth_failure_->Increment(1);
if (!sq_get_header(connection, "Authorization")) {
@@ -831,6 +863,44 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
req.post_data.append(buf, n);
rem -= n;
}
if (check_csrf_protection) {
// Always require 1) a csrf_token and cookie or 2) X-Requested-By header.
if (cookie_authenticated) {
std::vector<char> csrf_token(RAND_MAX_LENGTH+1, '\0');
int csrf_len = sq_get_var(req.post_data.c_str(), req.post_data.size(),
"csrf_token", csrf_token.data(), csrf_token.size());
if (csrf_len == -1) {
LOG(WARNING) << "CSRF protection: rejected POST without CSRF token";
sq_printf(connection, "HTTP/1.1 403 Forbidden\r\n");
return SQ_HANDLED_CLOSE_CONNECTION;
} else if (csrf_len == -2) {
LOG(WARNING) << "CSRF protection: CSRF token is too long";
sq_printf(connection, "HTTP/1.1 403 Forbidden\r\n");
return SQ_HANDLED_CLOSE_CONNECTION;
}
DCHECK(csrf_len >= 0 && csrf_len < csrf_token.size());
// Prevent CSRF for POSTs using the Double Submit Cookie pattern only if cookie
// authentication succeeded.
if (cookie_rand_value != csrf_token.data()) {
LOG(WARNING) << "CSRF protection: rejected POST with token mismatch: "
<< cookie_rand_value << " != " << csrf_token.data();
const char* msg = "please refresh the page and try again";
SendResponse(connection, "403 Forbidden", "text/plain", msg, response_headers);
return SQ_HANDLED_CLOSE_CONNECTION;
}
} else {
// Require a custom header matching csrf_token in the POST body.
const char* csrf_header = sq_get_header(connection, "X-Requested-By");
if (csrf_header == nullptr) {
const char* msg = "rejected POST missing X-Requested-By header";
LOG(WARNING) << "CSRF protection: " << msg;
SendResponse(connection, "403 Forbidden", "text/plain", msg, response_headers);
return SQ_HANDLED_CLOSE_CONNECTION;
}
}
}
}
// The output of this page is accumulated into this stringstream.
@@ -842,7 +912,8 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
content_type = PLAIN;
url_handler->raw_callback()(req, &output, &response);
} else {
RenderUrlWithTemplate(connection, req, *url_handler, &output, &content_type);
RenderUrlWithTemplate(
connection, req, *url_handler, &output, &content_type, cookie_rand_value);
}
VLOG(3) << "Rendering page " << request_info->uri << " took "
@@ -986,8 +1057,8 @@ Status Webserver::HandleBasic(struct sq_connection* connection,
return Status::Expected("Failed to authenticate to LDAP.");
}
void Webserver::AddCookie(
struct sq_request_info* request_info, vector<string>* response_headers) {
void Webserver::AddCookie(const char* user, vector<string>* response_headers,
string* cookie_rand_value) {
if (use_cookies_) {
// If cookie auth failed and we generated a 'delete cookie' header, remove it.
auto eq = [](const string& header) { return header.rfind("Set-Cookie", 0) == 0; };
@@ -996,17 +1067,17 @@ void Webserver::AddCookie(
response_headers->erase(it);
}
// Generate a cookie to return.
response_headers->push_back(
Substitute("Set-Cookie: $0", GenerateCookie(request_info->remote_user, hash_)));
response_headers->push_back(Substitute("Set-Cookie: $0",
GenerateCookie(user, hash_, cookie_rand_value)));
}
}
void Webserver::RenderUrlWithTemplate(const struct sq_connection* connection,
const WebRequest& req, const UrlHandler& url_handler, stringstream* output,
ContentType* content_type) {
ContentType* content_type, const std::string& csrf_token) {
Document document;
document.SetObject();
GetCommonJson(&document, connection, req);
GetCommonJson(&document, connection, req, csrf_token);
const auto& arguments = req.parsed_args;
url_handler.callback()(req, &document);
@@ -1092,4 +1163,10 @@ Webserver::AuthMode Webserver::GetConfiguredAuthMode() {
}
return AuthMode::NONE;
}
Webserver::ArgumentMap Webserver::GetVars(const std::string& data) {
ArgumentMap vars;
BuildArgumentMap(data, &vars);
return vars;
}
}

14
be/src/util/webserver.h
View File

@@ -133,6 +133,9 @@ class Webserver {
/// Returns the authentication mode configured by the startup flags.
static AuthMode GetConfiguredAuthMode();
/// Parses form-uri-encoded data and returns key/value pairs.
static ArgumentMap GetVars(const std::string& data);
private:
/// Contains all information relevant to rendering one Url. Each Url has one callback
/// that produces the output to render. The callback either produces a Json document
@@ -212,7 +215,8 @@ class Webserver {
struct sq_request_info* request_info, std::vector<std::string>* response_headers);
// Adds a 'Set-Cookie' header to 'response_headers', if cookie support is enabled.
void AddCookie(struct sq_request_info* request_info, vector<string>* response_headers);
// Returns the random value portion of the cookie in 'rand' for use in CSRF prevention.
void AddCookie(const char* user, vector<string>* response_headers, string* rand);
// Get username from Authorization header.
bool GetUsernameFromAuthHeader(struct sq_connection* connection,
@@ -225,7 +229,8 @@ class Webserver {
/// pretty-printed.
void RenderUrlWithTemplate(const struct sq_connection* connection,
const WebRequest& arguments, const UrlHandler& url_handler,
std::stringstream* output, ContentType* content_type);
std::stringstream* output, ContentType* content_type,
const std::string& csrf_token);
/// Called when an error is encountered, e.g. when a handler for a URI cannot be found.
void ErrorHandler(const WebRequest& req, rapidjson::Document* document);
@@ -233,12 +238,13 @@ class Webserver {
/// Builds a map of argument name to argument value from a typical URL argument
/// string (that is, "key1=value1&key2=value2.."). If no value is given for a
/// key, it is entered into the map as (key, "").
void BuildArgumentMap(const std::string& args, ArgumentMap* output);
static void BuildArgumentMap(const std::string& args, ArgumentMap* output);
/// Adds a __common__ object to document with common data that every webpage might want
/// to read (e.g. the names of links to write to the navbar).
void GetCommonJson(rapidjson::Document* document,
const struct sq_connection* connection, const WebRequest& req);
const struct sq_connection* connection, const WebRequest& req,
const std::string& csrf_token);
/// Lock guarding the path_handlers_ map
boost::shared_mutex url_handlers_lock_;

15
fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
View File

@@ -18,28 +18,23 @@
package org.apache.impala.customcluster;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import com.google.common.collect.Lists;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.hive.service.rpc.thrift.*;
import org.apache.impala.util.Metrics;
import org.apache.impala.testutil.WebClient;
import org.apache.thrift.transport.THttpClient;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
/**
@@ -47,7 +42,7 @@ import org.junit.Test;
* JWT authentication is being used.
*/
public class JwtHttpTest {
Metrics metrics_ = new Metrics();
WebClient client_ = new WebClient();
/* Since we don't have Java version of JWT library, we use pre-calculated JWT token.
* The token and JWK set used in this test case were generated by using BE unit-test
@@ -87,7 +82,7 @@ public class JwtHttpTest {
// JWKS file.
CustomClusterRunner.StartImpalaCluster();
if (createJWKSForWebServer_) deleteTempJWKSFromWebServerRootDir();
metrics_.Close();
client_.Close();
}
private void createTempJWKSInWebServerRootDir(String srcFilename) {
@@ -145,11 +140,11 @@ public class JwtHttpTest {
private void verifyJwtAuthMetrics(long expectedAuthSuccess, long expectedAuthFailure)
throws Exception {
long actualAuthSuccess =
(long) metrics_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess);
long actualAuthFailure =
(long) metrics_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-failure");
assertEquals(expectedAuthFailure, actualAuthFailure);
}

11
fe/src/test/java/org/apache/impala/customcluster/JwtWebserverTest.java
View File

@@ -19,7 +19,6 @@ package org.apache.impala.customcluster;
import static org.apache.impala.testutil.LdapUtil.*;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import java.io.File;
@@ -30,7 +29,7 @@ import java.util.HashMap;
import java.util.Map;
import com.google.common.collect.Range;
import org.apache.impala.util.Metrics;
import org.apache.impala.testutil.WebClient;
import org.junit.After;
import org.junit.Test;
@@ -40,7 +39,7 @@ import org.junit.Test;
public class JwtWebserverTest {
private static final Range<Long> zero = Range.closed(0L, 0L);
Metrics metrics_ = new Metrics(TEST_USER_1, TEST_PASSWORD_1);
WebClient client_ = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
public void setUp(String extraArgs, String startArgs) throws Exception {
Map<String, String> env = new HashMap<>();
@@ -54,17 +53,17 @@ public class JwtWebserverTest {
public void cleanUp() throws Exception {
// Leave a cluster running with the default configuration.
CustomClusterRunner.StartImpalaCluster();
metrics_.Close();
client_.Close();
}
private void verifyJwtAuthMetrics(
Range<Long> expectedAuthSuccess, Range<Long> expectedAuthFailure) throws Exception {
long actualAuthSuccess =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-success");
(long) client_.getMetric("impala.webserver.total-jwt-token-auth-success");
assertTrue("Expected: " + expectedAuthSuccess + ", Actual: " + actualAuthSuccess,
expectedAuthSuccess.contains(actualAuthSuccess));
long actualAuthFailure =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-failure");
(long) client_.getMetric("impala.webserver.total-jwt-token-auth-failure");
assertTrue("Expected: " + expectedAuthFailure + ", Actual: " + actualAuthFailure,
expectedAuthFailure.contains(actualAuthFailure));
}

23
fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
View File

@@ -34,10 +34,9 @@ import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.hive.service.rpc.thrift.*;
import org.apache.impala.util.Metrics;
import org.apache.impala.testutil.WebClient;
import org.apache.thrift.transport.THttpClient;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;
@@ -54,7 +53,7 @@ public class LdapHS2Test {
@ClassRule
public static CreateLdapServerRule serverRule = new CreateLdapServerRule();
Metrics metrics = new Metrics();
WebClient client_ = new WebClient();
public void setUp(String extraArgs) throws Exception {
String uri =
@@ -112,26 +111,26 @@ public class LdapHS2Test {
private void verifyMetrics(long expectedBasicAuthSuccess, long expectedBasicAuthFailure)
throws Exception {
long actualBasicAuthSuccess = (long) metrics.getMetric(
long actualBasicAuthSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertEquals(expectedBasicAuthSuccess, actualBasicAuthSuccess);
long actualBasicAuthFailure = (long) metrics.getMetric(
long actualBasicAuthFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertEquals(expectedBasicAuthFailure, actualBasicAuthFailure);
}
private void verifyCookieMetrics(
long expectedCookieAuthSuccess, long expectedCookieAuthFailure) throws Exception {
long actualCookieAuthSuccess = (long) metrics.getMetric(
long actualCookieAuthSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertEquals(expectedCookieAuthSuccess, actualCookieAuthSuccess);
long actualCookieAuthFailure = (long) metrics.getMetric(
long actualCookieAuthFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertEquals(expectedCookieAuthFailure, actualCookieAuthFailure);
}
private void verifyTrustedDomainMetrics(long expectedAuthSuccess) throws Exception {
long actualAuthSuccess = (long) metrics
long actualAuthSuccess = (long) client_
.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-trusted-domain-check-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess);
@@ -139,7 +138,7 @@ public class LdapHS2Test {
private void verifyTrustedAuthHeaderMetrics(long expectedAuthSuccess) throws Exception {
long actualAuthSuccess =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-trusted-auth-header-check-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess);
}
@@ -147,11 +146,11 @@ public class LdapHS2Test {
private void verifyJwtAuthMetrics(long expectedAuthSuccess, long expectedAuthFailure)
throws Exception {
long actualAuthSuccess =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess);
long actualAuthFailure =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-failure");
assertEquals(expectedAuthFailure, actualAuthFailure);
}
@@ -511,7 +510,7 @@ public class LdapHS2Test {
// Case 4: Verify that there are no changes in metrics for trusted auth
// header check if the X-Trusted-Proxy-Auth-Header header is not present
long successMetricBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-trusted-auth-header-check-success");
headers.put("Authorization", "Basic VGVzdDFMZGFwOjEyMzQ1");
headers.remove("X-Trusted-Proxy-Auth-Header");

12
fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
View File

@@ -28,7 +28,7 @@ import org.apache.directory.server.annotations.CreateLdapServer;
import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.impala.util.Metrics;
import org.apache.impala.testutil.WebClient;
import org.junit.Assume;
import org.junit.ClassRule;
@@ -47,7 +47,7 @@ public class LdapImpalaShellTest {
// Includes a special character to test HTTP path encoding.
protected static final String delegateUser_ = "proxyUser$";
private Metrics metrics = new Metrics();
private WebClient client_ = new WebClient();
public void setUp(String extraArgs) throws Exception {
String uri =
@@ -76,20 +76,20 @@ public class LdapImpalaShellTest {
private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = (long) metrics.getMetric(
long actualBasicSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = (long) metrics.getMetric(
long actualBasicFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = (long) metrics.getMetric(
long actualCookieSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = (long) metrics.getMetric(
long actualCookieFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure));

26
fe/src/test/java/org/apache/impala/customcluster/LdapImpylaHttpTest.java
View File

@@ -22,8 +22,6 @@ import static org.junit.Assert.assertTrue;
import com.google.common.collect.Lists;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
@@ -33,7 +31,7 @@ import org.apache.directory.server.annotations.CreateLdapServer;
import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.impala.util.Metrics;
import org.apache.impala.testutil.WebClient;
import org.apache.log4j.Logger;
import com.google.common.collect.Range;
import org.junit.After;
@@ -72,7 +70,7 @@ public class LdapImpylaHttpTest {
// Includes a special character to test HTTP path encoding.
private static final String delegateUser_ = "proxyUser$";
Metrics metrics = new Metrics();
WebClient client_ = new WebClient();
@Before
public void setUp() throws Exception {
@@ -96,20 +94,20 @@ public class LdapImpylaHttpTest {
private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = (long) metrics.getMetric(
long actualBasicSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = (long) metrics.getMetric(
long actualBasicFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = (long) metrics.getMetric(
long actualCookieSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = (long) metrics.getMetric(
long actualCookieFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure));
@@ -131,10 +129,10 @@ public class LdapImpylaHttpTest {
// 2. Invalid username password combination. Should fail.
long successBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-success");
long successCookieAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-cookie-auth-success");
String[] invalidCmd = buildCommand("foo", "bar", null, null);
RunShellCommand.Run(
@@ -146,7 +144,7 @@ public class LdapImpylaHttpTest {
// 3. Without username and password. Should fail.
long failedBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-failure");
String[] noAuthCmd = {"impala-python", helper_, "--query", query_};
RunShellCommand.Run(
@@ -168,7 +166,7 @@ public class LdapImpylaHttpTest {
// 5. Valid username, password, and HTTP cookie names.
// Should succeed with cookie authentication.
successBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-success");
String[] validCookieNamesCmd =
buildCommand(testUser_, testPassword_, null, "impala.auth");
@@ -181,10 +179,10 @@ public class LdapImpylaHttpTest {
// 6. Valid username and password, but HTTP cookie names don't consist of
// "impala.auth". Should succeed with cookie authentication failures.
successBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-success");
successCookieAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend."
(long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-cookie-auth-success");
String[] nonAuthCookieNamesCmd = buildCommand(testUser_, testPassword_, null,
"impala.session.id");

14
fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java
View File

@@ -34,9 +34,7 @@ import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.impala.testutil.ImpalaJdbcClient;
import org.apache.impala.util.Metrics;
import org.junit.AfterClass;
import org.junit.Before;
import org.apache.impala.testutil.WebClient;
import org.junit.ClassRule;
import org.junit.Test;
@@ -61,7 +59,7 @@ public class LdapJdbcTest extends JdbcTestBase {
private static final Range<Long> zero = Range.closed(0L, 0L);
private static final Range<Long> one = Range.closed(1L, 1L);
Metrics metrics = new Metrics();
WebClient client_ = new WebClient();
public LdapJdbcTest(String connectionType) { super(connectionType); }
@@ -87,20 +85,20 @@ public class LdapJdbcTest extends JdbcTestBase {
private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = (long) metrics.getMetric(
long actualBasicSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = (long) metrics.getMetric(
long actualBasicFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = (long) metrics.getMetric(
long actualCookieSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = (long) metrics.getMetric(
long actualCookieFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure));

238
fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java
View File

@@ -21,12 +21,15 @@ import static org.apache.impala.testutil.LdapUtil.*;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@@ -40,10 +43,14 @@ import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.hive.service.rpc.thrift.*;
import org.apache.impala.util.Metrics;
import org.apache.log4j.Logger;
import org.apache.http.cookie.Cookie;
import org.apache.http.message.BasicHeader;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.NameValuePair;
import org.apache.impala.testutil.WebClient;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.transport.THttpClient;
import org.json.simple.JSONObject;
import org.junit.After;
import org.junit.ClassRule;
import org.junit.Test;
@@ -54,13 +61,12 @@ import org.junit.Test;
transports = { @CreateTransport(protocol = "LDAP", address = "localhost") })
@ApplyLdifFiles({"users.ldif"})
public class LdapWebserverTest {
private static final Logger LOG = Logger.getLogger(LdapWebserverTest.class);
@ClassRule
public static CreateLdapServerRule serverRule = new CreateLdapServerRule();
private static final Range<Long> zero = Range.closed(0L, 0L);
Metrics metrics_ = new Metrics(TEST_USER_1, TEST_PASSWORD_1);
WebClient client_ = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
public void setUp(String extraArgs, String startArgs) throws Exception {
String uri =
@@ -75,38 +81,38 @@ public class LdapWebserverTest {
env.put("IMPALA_WEBSERVER_USERNAME", TEST_USER_1);
env.put("IMPALA_WEBSERVER_PASSWORD", TEST_PASSWORD_1);
int ret = CustomClusterRunner.StartImpalaCluster(impalaArgs, env, startArgs);
assertEquals(ret, 0);
assertEquals(0, ret);
}
@After
public void cleanUp() throws IOException {
metrics_.Close();
client_.Close();
}
private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess =
(long) metrics_.getMetric("impala.webserver.total-basic-auth-success");
(long) client_.getMetric("impala.webserver.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure =
(long) metrics_.getMetric("impala.webserver.total-basic-auth-failure");
(long) client_.getMetric("impala.webserver.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess =
(long) metrics_.getMetric("impala.webserver.total-cookie-auth-success");
(long) client_.getMetric("impala.webserver.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure =
(long) metrics_.getMetric("impala.webserver.total-cookie-auth-failure");
(long) client_.getMetric("impala.webserver.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure));
}
private void verifyTrustedDomainMetrics(Range<Long> expectedSuccess) throws Exception {
long actualSuccess = (long) metrics_
long actualSuccess = (long) client_
.getMetric("impala.webserver.total-trusted-domain-check-success");
assertTrue("Expected: " + expectedSuccess + ", Actual: " + actualSuccess,
expectedSuccess.contains(actualSuccess));
@@ -114,7 +120,7 @@ public class LdapWebserverTest {
private void verifyTrustedAuthHeaderMetrics(Range<Long> expectedSuccess)
throws Exception {
long actualSuccess = (long) metrics_.getMetric(
long actualSuccess = (long) client_.getMetric(
"impala.webserver.total-trusted-auth-header-check-success");
assertTrue("Expected: " + expectedSuccess + ", Actual: " + actualSuccess,
expectedSuccess.contains(actualSuccess));
@@ -123,11 +129,11 @@ public class LdapWebserverTest {
private void verifyJwtAuthMetrics(
Range<Long> expectedAuthSuccess, Range<Long> expectedAuthFailure) throws Exception {
long actualAuthSuccess =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-success");
(long) client_.getMetric("impala.webserver.total-jwt-token-auth-success");
assertTrue("Expected: " + expectedAuthSuccess + ", Actual: " + actualAuthSuccess,
expectedAuthSuccess.contains(actualAuthSuccess));
long actualAuthFailure =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-failure");
(long) client_.getMetric("impala.webserver.total-jwt-token-auth-failure");
assertTrue("Expected: " + expectedAuthFailure + ", Actual: " + actualAuthFailure,
expectedAuthFailure.contains(actualAuthFailure));
}
@@ -140,14 +146,14 @@ public class LdapWebserverTest {
verifyMetrics(Range.atLeast(1L), zero, Range.atLeast(1L), zero);
// Attempt to access the webserver without a username/password.
Metrics noUsername = new Metrics();
WebClient noUsername = new WebClient();
String result = noUsername.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is one unsuccessful auth attempt.
verifyMetrics(Range.atLeast(1L), Range.closed(1L, 1L), Range.atLeast(1L), zero);
// Attempt to access the webserver with invalid username/password.
Metrics invalidUserPass = new Metrics("invalid", "invalid");
WebClient invalidUserPass = new WebClient("invalid", "invalid");
result = invalidUserPass.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is now two unsuccessful auth attempts.
@@ -174,23 +180,23 @@ public class LdapWebserverTest {
// Access the webserver with a user that passes the group filter but not the user
// filter, should fail.
Metrics metricsUser2 = new Metrics(TEST_USER_2, TEST_PASSWORD_2);
String result = metricsUser2.readContent("/");
WebClient user2 = new WebClient(TEST_USER_2, TEST_PASSWORD_2);
String result = user2.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is one unsuccessful auth attempt.
verifyMetrics(Range.atLeast(1L), Range.closed(1L, 1L), Range.atLeast(1L), zero);
// Access the webserver with a user that passes the user filter but not the group
// filter, should fail.
Metrics metricsUser3 = new Metrics(TEST_USER_3, TEST_PASSWORD_3);
result = metricsUser3.readContent("/");
WebClient user3 = new WebClient(TEST_USER_3, TEST_PASSWORD_3);
result = user3.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is now two unsuccessful auth attempts.
verifyMetrics(Range.atLeast(1L), Range.closed(2L, 2L), Range.atLeast(1L), zero);
// Access the webserver with a user that doesn't pass either filter, should fail.
Metrics metricsUser4 = new Metrics(TEST_USER_4, TEST_PASSWORD_4);
result = metricsUser4.readContent("/");
WebClient user4 = new WebClient(TEST_USER_4, TEST_PASSWORD_4);
result = user4.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is now three unsuccessful auth attempts.
verifyMetrics(Range.atLeast(1L), Range.closed(3L, 3L), Range.atLeast(1L), zero);
@@ -205,17 +211,17 @@ public class LdapWebserverTest {
// Use 'per_impalad_args' to turn the metrics webserver on only for the first impalad.
setUp("", "--per_impalad_args=--metrics_webserver_port=25030");
// Attempt to access the regular webserver without a username/password, should fail.
Metrics noUsername = new Metrics();
WebClient noUsername = new WebClient();
String result = noUsername.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Attempt to access the regular webserver with invalid username/password.
Metrics invalidUserPass = new Metrics("invalid", "invalid");
WebClient invalidUserPass = new WebClient("invalid", "invalid");
result = invalidUserPass.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Attempt to access the metrics webserver without a username/password.
Metrics noUsernameMetrics = new Metrics(25030);
WebClient noUsernameMetrics = new WebClient(25030);
// Should succeed for the metrics endpoints.
for (String endpoint :
new String[] {"/metrics", "/jsonmetrics", "/metrics_prometheus", "/healthz"}) {
@@ -266,7 +272,7 @@ public class LdapWebserverTest {
// Case 6: Verify that there are no changes in metrics for trusted domain
// check if the X-Forwarded-For header is not present
long successMetricBefore = (long) metrics_
long successMetricBefore = (long) client_
.getMetric("impala.webserver.total-trusted-domain-check-success");
attemptConnection("Basic VGVzdDFMZGFwOjEyMzQ1", null, false);
verifyTrustedDomainMetrics(Range.closed(successMetricBefore, successMetricBefore));
@@ -295,7 +301,7 @@ public class LdapWebserverTest {
// Case 4: Verify that there are no changes in metrics for trusted auth header
// check if the trusted auth header is not present.
long successMetricBefore = (long) metrics_.getMetric(
long successMetricBefore = (long) client_.getMetric(
"impala.webserver.total-trusted-auth-header-check-success");
attemptConnection("Basic VGVzdDFMZGFwOjEyMzQ1", null, false);
verifyTrustedAuthHeaderMetrics(
@@ -368,29 +374,197 @@ public class LdapWebserverTest {
String cancelQueryUrl = String.format("/cancel_query?query_id=%s", queryId);
String textProfileUrl = String.format("/query_profile_plain_text?query_id=%s",
queryId);
metrics_.readContent(cancelQueryUrl);
String response = metrics_.readContent(textProfileUrl);
client_.readContent(cancelQueryUrl);
String response = client_.readContent(textProfileUrl);
String cancelStatus = String.format("Cancelled from Impala&apos;s debug web interface"
+ " by user: &apos;%s&apos; at", TEST_USER_1);
assertTrue(response.contains(cancelStatus));
// Wait for logs to flush
TimeUnit.SECONDS.sleep(6);
response = metrics_.readContent("/logs");
response = client_.readContent("/logs");
assertTrue(response.contains(cancelStatus));
// Session closing from the WebUI does not produce the cause message in the profile,
// so we will skip checking the runtime profile.
String sessionId = PrintId(openResp.getSessionHandle().getSessionId());
String closeSessionUrl = String.format("/close_session?session_id=%s", sessionId);
metrics_.readContent(closeSessionUrl);
client_.readContent(closeSessionUrl);
// Wait for logs to flush
TimeUnit.SECONDS.sleep(6);
String closeStatus = String.format("Session closed from Impala&apos;s debug web"
+ " interface by user: &apos;%s&apos; at", TEST_USER_1);
response = metrics_.readContent("/logs");
response = client_.readContent("/logs");
assertTrue(response.contains(closeStatus));
}
/*
* Test that we can set glog level.
*/
@Test
public void testSetGLogLevel() throws Exception {
setUp("", "");
// Validate defaults
JSONObject json = client_.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
// Test GET set_glog_level returns an error
json = client_.jsonGet("/set_glog_level?glog=0&json");
assertEquals("1", json.get("glog_level"));
assertEquals("Use form input to update glog level", json.get("error"));
// Test GET reset_glog_level returns an error
json = client_.jsonGet("/reset_glog_level?json");
assertEquals("1", json.get("glog_level"));
assertEquals("Use form input to reset glog level", json.get("error"));
// Clients persist state like 400 errors and cookies. Use new client for each test.
BasicHeader[] headers = { new BasicHeader("X-Requested-By", "anything") };
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("glog", "0"));
// Test POST set_glog_level fails
WebClient client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
String body = client.post("/set_glog_level?json", null, params, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST reset_glog_level fails
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
body = client.post("/reset_glog_level?json", null, null, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST set_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/set_glog_level?json", headers, params);
assertEquals("0", json.get("glog_level"));
// Test POST reset_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/reset_glog_level?json", headers, null);
assertEquals("1", json.get("glog_level"));
// Test POST set_glog_level with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
body = client.post("/set_glog_level?json", null, params, 403);
assertEquals("", body);
// Test POST reset_glog_level with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
body = client.post("/reset_glog_level?json", null, null, 403);
assertEquals("", body);
// Create a new client, get a cookie, and add csrf_token based on the cookie
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
String rand = getRandToken(client.getCookies());
params.add(new BasicNameValuePair("csrf_token", rand));
// Test POST set_glog_level with cookie and csrf_token succeeds
json = client.jsonPost("/set_glog_level?json", null, params);
assertEquals("0", json.get("glog_level"));
// Test POST reset_glog_level with cookie and csrf_token succeeds
json = client.jsonPost("/reset_glog_level?json", null, params);
assertEquals("1", json.get("glog_level"));
}
/*
* Test that we can set java log level.
*/
@Test
public void testSetJavaLogLevel() throws Exception {
setUp("", "");
// Validate defaults
JSONObject json = client_.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
// Test GET set_java_loglevel does nothing
json = client_.jsonGet("/set_java_loglevel?class=org.apache&level=WARN&json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
assertEquals("Use form input to update java log levels", json.get("error"));
// Test GET reset_java_loglevel does nothing
json = client_.jsonGet("/reset_java_loglevel?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
assertEquals("Use form input to reset java log levels", json.get("error"));
// Clients persist state like 400 errors and cookies. Use new client for each test.
BasicHeader[] headers = { new BasicHeader("X-Requested-By", "anything") };
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("class", "org.apache"));
params.add(new BasicNameValuePair("level", "WARN"));
// Test POST set_java_loglevel fails
WebClient client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
String body = client.post("/set_java_loglevel?json", null, params, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST reset_java_loglevel fails
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
body = client.post("/reset_java_loglevel?json", null, null, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST set_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/set_java_loglevel?json", headers, params);
assertEquals("org.apache : WARN\norg.apache.impala : DEBUG\n",
json.get("get_java_loglevel_result"));
// Test POST reset_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/reset_java_loglevel?json", headers, null);
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
// Test POST set_java_loglevel with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
body = client.post("/set_java_loglevel?json", null, params, 403);
assertEquals("", body);
// Test POST reset_java_loglevel with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
body = client.post("/reset_java_loglevel?json", null, null, 403);
assertEquals("", body);
// Create a new client, get a cookie, and add csrf_token based on the cookie
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
String rand = getRandToken(client.getCookies());
params.add(new BasicNameValuePair("csrf_token", rand));
// Test POST set_java_loglevel with cookie and csrf_token succeeds
json = client.jsonPost("/set_java_loglevel?json", null, params);
assertEquals("org.apache : WARN\norg.apache.impala : DEBUG\n",
json.get("get_java_loglevel_result"));
// Test POST reset_java_loglevel with cookie and csrf_token succeeds
json = client.jsonPost("/reset_java_loglevel?json", null, params);
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
}
private String getRandToken(List<Cookie> cookies) {
for (Cookie cookie : cookies) {
String[] tokens = cookie.getValue().split("&");
for (String token : tokens) {
if (token.charAt(0) == 'r' && token.charAt(1) == '=') {
String rand = token.substring(2);
assertTrue("Expected number: " + rand, rand.matches("^[1-9][0-9]*$"));
return rand;
}
}
}
fail("Expected cookie to contain random number");
return "";
}
// Helper method to make a get call to the webserver using the input basic
// auth token, x-forward-for and X-Trusted-Proxy-Auth-Header token.
private void attemptConnection(String basic_auth_token, String xff_address,

16
fe/src/test/java/org/apache/impala/service/JdbcTest.java
View File

@@ -36,7 +36,7 @@ import java.util.Map;
import org.apache.impala.testutil.ImpalaJdbcClient;
import org.apache.impala.testutil.TestUtils;
import org.apache.impala.util.Metrics;
import org.apache.impala.testutil.WebClient;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.Test;
@@ -615,7 +615,7 @@ public class JdbcTest extends JdbcTestBase {
@Test
public void testConcurrentSessionMixedIdleTimeout() throws Exception {
// Test for concurrent idle sessions' expiration with mixed timeout durations.
Metrics metrics = new Metrics();
WebClient client = new WebClient();
List<Integer> timeoutPeriods = Arrays.asList(0, 3, 15);
List<Connection> connections = new ArrayList<>();
@@ -626,9 +626,9 @@ public class JdbcTest extends JdbcTestBase {
createConnection(ImpalaJdbcClient.getNoAuthConnectionStr(connectionType_)));
}
Long numOpenSessions = (Long)metrics.getMetric(
Long numOpenSessions = (Long)client.getMetric(
"impala-server.num-open-hiveserver2-sessions");
Long numExpiredSessions = (Long)metrics.getMetric(
Long numExpiredSessions = (Long)client.getMetric(
"impala-server.num-sessions-expired");
for (int i = 0; i < connections.size(); ++i) {
@@ -642,9 +642,9 @@ public class JdbcTest extends JdbcTestBase {
lastTimeSessionActive.add(System.currentTimeMillis() / 1000);
}
assertEquals(numOpenSessions, (Long)metrics.getMetric(
assertEquals(numOpenSessions, (Long)client.getMetric(
"impala-server.num-open-hiveserver2-sessions"));
assertEquals(numExpiredSessions, (Long)metrics.getMetric(
assertEquals(numExpiredSessions, (Long)client.getMetric(
"impala-server.num-sessions-expired"));
for (int timeout : timeoutPeriods) {
@@ -679,9 +679,9 @@ public class JdbcTest extends JdbcTestBase {
}
}
assertEquals(numOpenSessions, (Long)metrics.getMetric(
assertEquals(numOpenSessions, (Long)client.getMetric(
"impala-server.num-open-hiveserver2-sessions"));
assertEquals(numExpiredSessions, (Long)metrics.getMetric(
assertEquals(numExpiredSessions, (Long)client.getMetric(
"impala-server.num-sessions-expired"));
}

112
fe/src/test/java/org/apache/impala/util/Metrics.java → fe/src/test/java/org/apache/impala/testutil/WebClient.java
View File

@@ -15,57 +15,69 @@
// specific language governing permissions and limitations
// under the License.
package org.apache.impala.util;
package org.apache.impala.testutil;
import java.io.IOException;
import java.util.List;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.AuthCache;
import org.apache.http.impl.client.BasicAuthCache;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.cookie.Cookie;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.NameValuePair;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.impl.client.BasicAuthCache;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
/**
* Utility class for retrieving metrics from the Impala webserver.
* Utility class for interacting with the Impala webserver.
*/
public class Metrics {
public class WebClient {
private final static String WEBSERVER_HOST = "localhost";
private final static int DEFAULT_WEBSERVER_PORT = 25000;
private final static String JSON_METRICS = "/jsonmetrics?json";
private CloseableHttpClient httpClient_;
private BasicCookieStore cookieStore_;
private int port_;
private String username_;
private String password_;
public Metrics() { this("", "", DEFAULT_WEBSERVER_PORT); }
public WebClient() { this("", "", DEFAULT_WEBSERVER_PORT); }
public Metrics(int port) { this("", "", port); }
public WebClient(int port) { this("", "", port); }
public Metrics(String username, String password) {
public WebClient(String username, String password) {
this(username, password, DEFAULT_WEBSERVER_PORT);
}
public Metrics(String username, String password, int port) {
public WebClient(String username, String password, int port) {
this.username_ = username;
this.password_ = password;
this.port_ = port;
httpClient_ = HttpClients.createDefault();
cookieStore_ = new BasicCookieStore();
}
public void Close() throws IOException { httpClient_.close(); }
public List<Cookie> getCookies() { return cookieStore_.getCookies(); }
/**
* Returns the metric for the given metric id from the Impala web server.
* @param metricId identifier of the metric we want to retrieve
@@ -73,20 +85,79 @@ public class Metrics {
* The caller needs to cast it to the appropriate type, e.g. Long, String, etc.
*/
public Object getMetric(String metricId) throws Exception {
String content = readContent(JSON_METRICS);
if (content == null) return null;
JSONObject json = toJson(content);
JSONObject json = jsonGet(JSON_METRICS);
if (json == null) return null;
return json.get(metricId);
}
/**
* Does a GET request at path of the Impala web server and returns response.
* @param path URI path to query
* @return A JSON object, or null if not parseable as JSON
* @throws Exception if the request fails or JSON is not an object
*/
public JSONObject jsonGet(String path) throws Exception {
return toJson(readContent(path));
}
/**
* Does a POST request at path of the Impala web server and returns response.
* @param path URI path to query
* @param headers Headers to include in the request
* @param params Parameters to include in the POST
* @return A JSON object, or null if not parseable as JSON
* @throws Exception if the request fails or JSON is not an object
*/
public JSONObject jsonPost(String path, Header[] headers, List<NameValuePair> params)
throws Exception {
return toJson(post(path, headers, params, 200));
}
/**
* Retrieves the page at 'path' and returns its contents.
*/
public String readContent(String path) throws IOException {
HttpHost targetHost = new HttpHost(WEBSERVER_HOST, port_, "http");
HttpHost target = new HttpHost(WEBSERVER_HOST, port_, "http");
HttpClientContext context = getContext(target);
HttpGet get = new HttpGet(path);
try (CloseableHttpResponse response = httpClient_.execute(target, get, context)) {
return EntityUtils.toString(response.getEntity());
}
}
/**
* Does a POST request at path of the Impala web server and returns response.
* If the response does not include the expected status code, returns null.
* @param path URI path to query
* @param headers Headers to include in the POST; can be null
* @param params Parameters to include in the POST; can be null
* @param code Response status code expected
* @return Response string
* @throws IOException if the request fails
*/
public String post(String path, Header[] headers, List<NameValuePair> params, int code)
throws IOException {
HttpHost target = new HttpHost(WEBSERVER_HOST, port_, "http");
HttpClientContext context = getContext(target);
HttpPost post = new HttpPost(path);
if (headers != null) {
post.setHeaders(headers);
}
if (params != null) {
post.setEntity(new UrlEncodedFormEntity(params));
}
try (CloseableHttpResponse response = httpClient_.execute(target, post, context)) {
if (response.getStatusLine().getStatusCode() != code) {
return null;
}
return EntityUtils.toString(response.getEntity());
}
}
private HttpClientContext getContext(HttpHost targetHost) {
HttpClientContext context = HttpClientContext.create();
if (!username_.equals("")) {
CredentialsProvider credsProvider = new BasicCredentialsProvider();
@@ -97,20 +168,11 @@ public class Metrics {
context.setCredentialsProvider(credsProvider);
context.setAuthCache(authCache);
}
String ret = "";
HttpGet httpGet = new HttpGet(path);
CloseableHttpResponse response = httpClient_.execute(targetHost, httpGet, context);
try {
ret = EntityUtils.toString(response.getEntity());
} finally {
response.close();
}
return ret;
context.setCookieStore(cookieStore_);
return context;
}
private static JSONObject toJson(String text) throws Exception {
private static JSONObject toJson(String text) throws ParseException {
JSONParser parser = new JSONParser();
Object obj = parser.parse(text);

84
tests/webserver/test_web_pages.py
View File

@@ -183,6 +183,27 @@ class TestWebPage(ImpalaTestSuite):
assert 'Content-Security-Policy' in response.headers, "CSP header missing"
return responses
def post_and_check_status(self, url, data={}, string_to_search="", ports_to_test=None):
"""Helper method that posts to a given url, then asserts the return code is ok and
the response contains the expected string."""
if ports_to_test is None:
ports_to_test = self.TEST_PORTS_WITH_SS
responses = []
for port in ports_to_test:
input_url = url.format(port)
response = requests.head(input_url)
assert response.status_code == requests.codes.ok, "URL: {0} Str:'{1}'\nResp:{2}"\
.format(input_url, string_to_search, response.text)
response = requests.post(input_url, data=data)
assert response.status_code == requests.codes.ok, "URL: {0} Str:'{1}'\nResp:{2}"\
.format(input_url, string_to_search, response.text)
assert string_to_search in response.text, "URL: {0} Str:'{1}'\nResp:{2}".format(
input_url, string_to_search, response.text)
responses.append(response)
assert 'Content-Security-Policy' in response.headers, "CSP header missing"
return responses
def get_debug_page(self, page_url, port=25000):
"""Returns the content of the debug page 'page_url' as json."""
responses = self.get_and_check_status(page_url + "?json", ports_to_test=[port])
@@ -195,6 +216,11 @@ class TestWebPage(ImpalaTestSuite):
return self.get_and_check_status(url, string_to_search,
ports_to_test=self.TEST_PORTS_WITHOUT_SS)
def post_and_check_status_jvm(self, url, data={}, string_to_search=""):
"""Calls post_and_check_status() for impalad and catalogd only"""
return self.post_and_check_status(url, data, string_to_search,
ports_to_test=self.TEST_PORTS_WITHOUT_SS)
def test_content_type(self):
"""Checks that an appropriate content-type is set for various types of pages."""
# Mapping from each page to its MIME type.
@@ -211,52 +237,46 @@ class TestWebPage(ImpalaTestSuite):
malformed inputs. This however does not test that the log level changes are actually
in effect."""
# Check that the log_level end points are accessible.
self.get_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL)
self.get_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL)
self.get_and_check_status(self.SET_GLOG_LOGLEVEL_URL)
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL)
self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL)
self.post_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL)
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL)
self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL)
# Set the log level of a class to TRACE and confirm the setting is in place
set_loglevel_url = (self.SET_JAVA_LOGLEVEL_URL + "?class" +
"=org.apache.impala.catalog.HdfsTable&level=trace")
self.get_and_check_status_jvm(
set_loglevel_url,
"org.apache.impala.catalog.HdfsTable : TRACE")
self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL,
{"class": "org.apache.impala.catalog.HdfsTable", "level": "trace"},
"org.apache.impala.catalog.HdfsTable : TRACE")
# Reset Java logging levels
self.get_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL, "Java log levels reset.")
self.post_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL, {},
"Java log levels reset.")
# Set a new glog level and make sure the setting has been applied.
set_glog_url = (self.SET_GLOG_LOGLEVEL_URL + "?glog=3")
self.get_and_check_status(set_glog_url, "val(3)")
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": 3}, "val(3)")
# Try resetting the glog logging defaults again.
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, "Current backend log level: ")
self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, {},
"Current backend log level: ")
# Same as above, for set log level request
set_loglevel_url = (self.SET_JAVA_LOGLEVEL_URL + "?class=")
self.get_and_check_status_jvm(set_loglevel_url)
# Try to set the log level with an empty class input
self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL, {"class": ""})
# Empty input for setting a glog level request
set_glog_url = (self.SET_GLOG_LOGLEVEL_URL + "?glog=")
self.get_and_check_status(set_glog_url)
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": ""})
# Try setting a non-existent log level on a valid class. In such cases,
# log4j automatically sets it as DEBUG. This is the behavior of
# Level.toLevel() method.
set_loglevel_url = (self.SET_JAVA_LOGLEVEL_URL + "?class" +
"=org.apache.impala.catalog.HdfsTable&level=foo&")
self.get_and_check_status_jvm(
set_loglevel_url,
"org.apache.impala.catalog.HdfsTable : DEBUG")
self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL,
{"class": "org.apache.impala.catalog.HdfsTable", "level": "foo"},
"org.apache.impala.catalog.HdfsTable : DEBUG")
# Try setting an invalid glog level.
set_glog_url = self.SET_GLOG_LOGLEVEL_URL + "?glog=foo"
self.get_and_check_status(set_glog_url, "Bad glog level input")
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": "foo"},
"Bad glog level input")
# Try a non-existent endpoint on log_level URL.
bad_loglevel_url = self.SET_GLOG_LOGLEVEL_URL + "?badurl=foo"
self.get_and_check_status(bad_loglevel_url)
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"badurl": "foo"})
@pytest.mark.execute_serially
def test_uda_with_log_level(self):
@@ -264,15 +284,15 @@ class TestWebPage(ImpalaTestSuite):
to 3. Running this test serially not to interfere with other tests setting the log
level."""
# Check that the log_level end points are accessible.
self.get_and_check_status(self.SET_GLOG_LOGLEVEL_URL)
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL)
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL)
self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL)
# Set log level to 3.
set_glog_url = (self.SET_GLOG_LOGLEVEL_URL + "?glog=3")
self.get_and_check_status(set_glog_url, "val(3)")
self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": 3}, "val(3)")
# Check that Impala doesn't crash when running a query that aggregates.
self.client.execute("select avg(int_col) from functional.alltypessmall")
# Reset log level.
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, "Current backend log level: ")
self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, {},
"Current backend log level: ")
def test_catalog(self, cluster_properties, unique_database):
"""Tests the /catalog and /catalog_object endpoints."""

3
www/form-hidden-inputs.tmpl
View File

@@ -1,3 +1,6 @@
{{# __common__.csrf_token }}
<input type='hidden' name='csrf_token' value='{{ __common__.csrf_token }}' />
{{/ __common__.csrf_token }}
{{# __common__.hostname }}
<input type='hidden' name='scheme' value='{{ __common__.scheme }}' />
<input type='hidden' name='host' value='{{ __common__.hostname }}' />

8
www/log_level.tmpl
View File

@@ -31,7 +31,7 @@ under the License.
<h5>Current frontend log level:</h5>
<span style="white-space: pre-line">{{get_java_loglevel_result}}</span>
<br>
<form action="set_java_loglevel">{{>www/form-hidden-inputs.tmpl}}
<form action="set_java_loglevel" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="form-group" name="level">
<input type="text" class="form-control" name="class" placeholder="e.g. org.apache.impala.analysis.Analyzer">
<br>
@@ -51,7 +51,7 @@ under the License.
</div>
</div>
</form>
<form action="reset_java_loglevel">{{>www/form-hidden-inputs.tmpl}}
<form action="reset_java_loglevel" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="col-xs-20">
<button type="submit" class="btn btn-primary btn-sm">Reset Frontend Log Levels</button>
<strong>{{reset_java_loglevel_result}}</strong>
@@ -62,7 +62,7 @@ under the License.
<h2>Backend log level configuration (glog)</h2>
<h5>Current backend log level: <span id="glog_text"></span></h5>
<form action="set_glog_level">{{>www/form-hidden-inputs.tmpl}}
<form action="set_glog_level" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="form-group" name="level">
<div class="col-xs-20">
<label>Log level:</label>
@@ -76,7 +76,7 @@ under the License.
</div>
</div>
</form>
<form action="reset_glog_level">{{>www/form-hidden-inputs.tmpl}}
<form action="reset_glog_level" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="col-xs-20">
<button type="submit" class="btn btn-primary btn-sm">Reset Backend Log Level</button>
<strong>&nbsp &nbsp Default backend log level: {{default_glog_level}}</strong>