jprdonnelly/impala
1
0
Fork 0
mirror of https://github.com/apache/impala.git synced 2025-12-19 18:12:08 -05:00
Code Issues Projects Releases Wiki Activity

IMPALA-11856: Use POST requests to set log level

Browse Source 
Set and reset loglevel handlers now require a POST. Implements
Cross-Site Request Forgery (CSRF) prevention in Impala's webserver using
the Double Submit Cookie pattern - where POST requests must include a
csrf_token field in their post with the random value from the cookie -
or a custom header.

CSRF attacks rely on the browser always sending a cookie or
'Authorization: Basic' header.
- With cookies, attacks don't have access to default form values or the
  original cookie, so we can include the cookie's random value in the
  form as a cross-check. As the cookie is cryptographically signed, they
  also can't be replaced with one that would match an attack's forms.
- When not using cookies, a custom header (X-Requested-By) is required
  as CSRFs are unable to add custom headers. This approach is also used
  by Jersey; see
  http://blog.alutam.com/2011/09/14/jersey-and-cross-site-request-forgery-csrf

In a broader implementation this would require a separate cookie so it
can be used to protect logins as well, but login is handled external to
Impala so we re-use the cookie the page already has.

Cookies are now generated for the HTPASSWD authentication method.
Authenticating via JWT still omits cookies because the JWT is already
provided via custom header (preventing CSRF) and disabling
authentication (NONE) means anyone could directly send a request so CSRF
protection is meaningless.

We also start an additional Webserver instance with authentication NONE
when metrics_webserver_port > 0, and the Webserver metric
"impala.webserver.total-cookie-auth-success" can only be registered
once. Additional changes would be necessary to make metric names unique
in Webserver (based on port); for the moment we avoid that by ensuring
all metrics counters are only instantiated for Webservers that use
authentication.

Cookie generation and authentication were updated to provide access to
the random value.

Adds flag to enable SameSite=Strict for defense in depth as mentioned in
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis.
This can be enabled if another CSRF attack method is found.

Verified that this prevents CSRF attacks by disabling SameSite=Strict
and visiting (via https://security.love/CSRF-PoC-Genorator):
```
<html>
  <form enctype="application/x-www-form-urlencoded" method="POST"
        action="http://localhost:45000/set_glog_level">
    <table>
      <tr>
        <td>glog</td>
        <td><input type="text" value="1" name="glog"></td>
      </tr>
    </table>
    <input type="submit" value="http://localhost:45000/set_glog_level">
  </form>
</html>
```

Adds tests for the webserver with basic authentication, LDAP, and SPNEGO
that authorization fails on POST unless
- using a cookie and csrf_token is correctly set in the POST body
- the X-Requested-By header is set

Change-Id: I4be8694492b8ba16737f644ac8c56d8124f19693
Reviewed-on: http://gerrit.cloudera.org:8080/19199
Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
This commit is contained in:
Michael Smith
2022-11-03 09:23:54 -07:00
committed by Impala Public Jenkins
parent 026f6993ed
commit 86d33a0a3d
18 changed files with 802 additions and 271 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
be/src/rpc/authentication-util.cc
View File

@@ -35,6 +35,8 @@ DEFINE_bool(cookie_require_secure, true,
DEFINE_int64(max_cookie_lifetime_s, 24 * 60 * 60, DEFINE_int64(max_cookie_lifetime_s, 24 * 60 * 60,
"Maximum amount of time in seconds that an authentication cookie will remain valid. " "Maximum amount of time in seconds that an authentication cookie will remain valid. "
"Setting to 0 disables use of cookies. Defaults to 1 day."); "Setting to 0 disables use of cookies. Defaults to 1 day.");
DEFINE_bool(samesite_strict, false,
"(Advanced) If true, authentication cookies will include SameSite=Strict.");
using namespace strings; using namespace strings;
@@ -59,13 +61,14 @@ static const int SHA256_BASE64_LEN =
CalculateBase64EscapedLen(AuthenticationHash::HashLen(), /* do_padding */ true); CalculateBase64EscapedLen(AuthenticationHash::HashLen(), /* do_padding */ true);
// Since we only return cookies with a single name, well behaved clients should only ever // Since we only return cookies with a single name, well behaved clients should only ever
// return one cookie to us. To accomodate non-malicious but poorly behaved clients, we // return one cookie to us. To accommodate non-malicious but poorly behaved clients, we
// allow for checking a limited number of cookies, up to MAX_COOKIES_TO_CHECK or until we // allow for checking a limited number of cookies, up to MAX_COOKIES_TO_CHECK or until we
// find the first one with COOKIE_NAME. // find the first one with COOKIE_NAME.
static const int MAX_COOKIES_TO_CHECK = 5; static const int MAX_COOKIES_TO_CHECK = 5;
Status AuthenticateCookie( Status AuthenticateCookie(
const AuthenticationHash& hash, const string& cookie_header, string* username) { const AuthenticationHash& hash, const string& cookie_header,
string* username, string* rand) {
// The 'Cookie' header allows sending multiple name/value pairs separated by ';'. // The 'Cookie' header allows sending multiple name/value pairs separated by ';'.
vector<string> cookies = strings::Split(cookie_header, ";"); vector<string> cookies = strings::Split(cookie_header, ";");
if (cookies.size() > MAX_COOKIES_TO_CHECK) { if (cookies.size() > MAX_COOKIES_TO_CHECK) {
@@ -125,6 +128,11 @@ Status AuthenticateCookie(
if (!TryStripPrefixString(cookie_value_split[0], USERNAME_KEY, username)) { if (!TryStripPrefixString(cookie_value_split[0], USERNAME_KEY, username)) {
return Status("The cookie username value has an invalid format."); return Status("The cookie username value has an invalid format.");
} }
if (rand != nullptr) {
if (!TryStripPrefixString(cookie_value_split[2], RAND_KEY, rand)) {
return Status("The cookie rand value has an invalid format.");
}
}
// We've successfully authenticated. // We've successfully authenticated.
return Status::OK(); return Status::OK();
} else { } else {
@@ -135,12 +143,18 @@ Status AuthenticateCookie(
return Status(Substitute("Did not find expected cookie name: $0", COOKIE_NAME)); return Status(Substitute("Did not find expected cookie name: $0", COOKIE_NAME));
} }
string GenerateCookie(const string& username, const AuthenticationHash& hash) { string GenerateCookie(const string& username, const AuthenticationHash& hash,
std::string* srand) {
// Its okay to use rand() here even though its a weak RNG because being able to guess // Its okay to use rand() here even though its a weak RNG because being able to guess
// the random numbers generated won't help an attacker. The important thing is that // the random numbers generated won't help an attacker. The important thing is that
// we're using a strong RNG to create the key and a strong HMAC function. // we're using a strong RNG to create the key and a strong HMAC function.
int cookie_rand = rand();
string cookie_rand_s = std::to_string(cookie_rand);
if (srand != nullptr) {
*srand = cookie_rand_s;
}
string cookie_value = StrCat(USERNAME_KEY, username, COOKIE_SEPARATOR, TIMESTAMP_KEY, string cookie_value = StrCat(USERNAME_KEY, username, COOKIE_SEPARATOR, TIMESTAMP_KEY,
MonotonicMillis(), COOKIE_SEPARATOR, RAND_KEY, rand()); MonotonicMillis(), COOKIE_SEPARATOR, RAND_KEY, cookie_rand_s);
uint8_t signature[AuthenticationHash::HashLen()]; uint8_t signature[AuthenticationHash::HashLen()];
Status compute_status = Status compute_status =
hash.Compute(reinterpret_cast<const uint8_t*>(cookie_value.data()), hash.Compute(reinterpret_cast<const uint8_t*>(cookie_value.data()),
@@ -155,12 +169,13 @@ string GenerateCookie(const string& username, const AuthenticationHash& hash) {
SHA256_BASE64_LEN, /* do_padding */ true); SHA256_BASE64_LEN, /* do_padding */ true);
base64_signature[SHA256_BASE64_LEN] = '\0'; base64_signature[SHA256_BASE64_LEN] = '\0';
const char* secure_flag = ";Secure"; const char* secure_flag = FLAGS_cookie_require_secure ? ";Secure" : "";
if (!FLAGS_cookie_require_secure) { const char* samesite_flag = FLAGS_samesite_strict ? ";SameSite=Strict" : "";
secure_flag = ""; // Add SameSite=Strict to notify the browser it should avoid sending the cookie with
} // requests from other domains.
return Substitute("$0=$1$2$3;HttpOnly;Max-Age=$4$5", COOKIE_NAME, base64_signature, return Substitute("$0=$1$2$3;HttpOnly;Max-Age=$4$5$6", COOKIE_NAME, base64_signature,
COOKIE_SEPARATOR, cookie_value, FLAGS_max_cookie_lifetime_s, secure_flag); COOKIE_SEPARATOR, cookie_value, FLAGS_max_cookie_lifetime_s, secure_flag,
samesite_flag);
} }
string GetDeleteCookie() { string GetDeleteCookie() {

14
be/src/rpc/authentication-util.h
View File

@@ -25,13 +25,15 @@ class AuthenticationHash;
// Takes a single 'key=value' pair from a 'Cookie' header and attempts to verify its // Takes a single 'key=value' pair from a 'Cookie' header and attempts to verify its
// signature with 'hash'. If verification is successful and the cookie is still valid, // signature with 'hash'. If verification is successful and the cookie is still valid,
// sets 'username' to the corresponding username and returns OK. // sets 'username' and 'rand' (if specified) to the corresponding values and returns OK.
Status AuthenticateCookie(const AuthenticationHash& hash, Status AuthenticateCookie(
const std::string& cookie_header, std::string* username); const AuthenticationHash& hash, const std::string& cookie_header,
std::string* username, std::string* rand = nullptr);
// Generates and returns a cookie containing the username set on 'connection_context' and // Generates and returns a cookie containing the username set on 'connection_context' and
// a signature generated with 'hash'. // a signature generated with 'hash'. If specified, sets 'rand' to the 'r=' cookie value.
std::string GenerateCookie(const std::string& username, const AuthenticationHash& hash); std::string GenerateCookie(const std::string& username, const AuthenticationHash& hash,
std::string* rand = nullptr);
// Returns a empty cookie. Returned in a 'Set-Cookie' when cookie auth fails to indicate // Returns a empty cookie. Returned in a 'Set-Cookie' when cookie auth fails to indicate
// to the client that the cookie should be deleted. // to the client that the cookie should be deleted.
@@ -51,4 +53,6 @@ bool IsTrustedDomain(const std::string& origin, const std::string& trusted_domai
Status BasicAuthExtractCredentials( Status BasicAuthExtractCredentials(
const string& token, string& username, string& password); const string& token, string& username, string& password);
constexpr int RAND_MAX_LENGTH = 10;
} // namespace impala } // namespace impala

24
be/src/util/logging-support.cc
View File

@@ -149,7 +149,12 @@ void GetJavaLogLevels(Document* document) {
// Callback handler for /set_java_loglevel. // Callback handler for /set_java_loglevel.
void SetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* document) { void SetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* document) {
const auto& args = req.parsed_args; if (req.request_method != "POST") {
AddDocumentMember("Use form input to update java log levels", "error", document);
return;
}
Webserver::ArgumentMap args = Webserver::GetVars(req.post_data);
Webserver::ArgumentMap::const_iterator classname = args.find("class"); Webserver::ArgumentMap::const_iterator classname = args.find("class");
Webserver::ArgumentMap::const_iterator level = args.find("level"); Webserver::ArgumentMap::const_iterator level = args.find("level");
if (classname == args.end() || classname->second.empty() || if (classname == args.end() || classname->second.empty() ||
@@ -179,6 +184,11 @@ void SetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* documen
// Callback handler for /reset_java_loglevel. // Callback handler for /reset_java_loglevel.
void ResetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* document) { void ResetJavaLogLevelCallback(const Webserver::WebRequest& req, Document* document) {
if (req.request_method != "POST") {
AddDocumentMember("Use form input to reset java log levels", "error", document);
return;
}
Status status = ResetJavaLogLevels(); Status status = ResetJavaLogLevels();
if (!status.ok()) { if (!status.ok()) {
AddDocumentMember(status.GetDetail(), "error", document); AddDocumentMember(status.GetDetail(), "error", document);
@@ -202,7 +212,12 @@ void GetGlogLevel(Document* document) {
// Callback handler for /set_glog_level // Callback handler for /set_glog_level
void SetGlogLevelCallback(const Webserver::WebRequest& req, Document* document) { void SetGlogLevelCallback(const Webserver::WebRequest& req, Document* document) {
const auto& args = req.parsed_args; if (req.request_method != "POST") {
AddDocumentMember("Use form input to update glog level", "error", document);
return;
}
Webserver::ArgumentMap args = Webserver::GetVars(req.post_data);
Webserver::ArgumentMap::const_iterator glog_level = args.find("glog"); Webserver::ArgumentMap::const_iterator glog_level = args.find("glog");
if (glog_level == args.end() || glog_level->second.empty()) { if (glog_level == args.end() || glog_level->second.empty()) {
AddDocumentMember("Bad glog level input. Valid inputs are integers in the " AddDocumentMember("Bad glog level input. Valid inputs are integers in the "
@@ -221,6 +236,11 @@ void SetGlogLevelCallback(const Webserver::WebRequest& req, Document* document)
// Callback handler for /reset_glog_level // Callback handler for /reset_glog_level
void ResetGlogLevelCallback(const Webserver::WebRequest& req, Document* document) { void ResetGlogLevelCallback(const Webserver::WebRequest& req, Document* document) {
if (req.request_method != "POST") {
AddDocumentMember("Use form input to reset glog level", "error", document);
return;
}
string new_log_level = google::SetCommandLineOption("v", string new_log_level = google::SetCommandLineOption("v",
to_string(FLAGS_v_original_value).data()); to_string(FLAGS_v_original_value).data());
VLOG(1) << "New glog level set: " << new_log_level; VLOG(1) << "New glog level set: " << new_log_level;

313
be/src/util/webserver-test.cc
View File

@@ -21,6 +21,7 @@
#include <boost/filesystem.hpp> #include <boost/filesystem.hpp>
#include <boost/lexical_cast.hpp> #include <boost/lexical_cast.hpp>
#include <gutil/strings/substitute.h> #include <gutil/strings/substitute.h>
#include <map>
#include <openssl/crypto.h> #include <openssl/crypto.h>
#include <openssl/ssl.h> #include <openssl/ssl.h>
#include <regex> #include <regex>
@@ -65,49 +66,74 @@ const string TO_ESCAPE_KEY = "ToEscape";
const string TO_ESCAPE_VALUE = "<script language='javascript'>"; const string TO_ESCAPE_VALUE = "<script language='javascript'>";
const string ESCAPED_VALUE = "&lt;script language=&apos;javascript&apos;&gt;"; const string ESCAPED_VALUE = "&lt;script language=&apos;javascript&apos;&gt;";
// Adapted from: struct HttpRequest {
// http://stackoverflow.com/questions/10982717/get-html-without-header-with-boostasio string url_path = "/";
string host = "localhost";
int32_t port = FLAGS_webserver_port;
map<string, string> headers = {};
// Adapted from:
// http://stackoverflow.com/questions/10982717/get-html-without-header-with-boostasio
Status Do(ostream* out, int expected_code, const string& method) {
try {
tcp::iostream request_stream;
request_stream.connect(host, lexical_cast<string>(port));
if (!request_stream) return Status("Could not connect request_stream");
request_stream << method << " " << url_path << " HTTP/1.1\r\n";
request_stream << "Host: " << host << ":" << port << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Cache-Control: no-cache\r\n";
if (method == "POST") {
request_stream << "Content-Length: 0\r\n";
}
for (const std::pair<string, string>& header : headers) {
request_stream << header.first << ": " << header.second << "\r\n";
}
request_stream << "Connection: close\r\n\r\n";
request_stream.flush();
string line1;
getline(request_stream, line1);
if (!request_stream) return Status("No response");
stringstream response_stream(line1);
string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
string status_message;
getline(response_stream, status_message);
if (!response_stream || http_version.substr(0,5) != "HTTP/") {
return Status("Malformed response");
}
if (status_code != expected_code) {
return Status(Substitute("Unexpected status code: $0", status_code));
}
(*out) << request_stream.rdbuf();
return Status::OK();
} catch (const std::exception& e){
return Status(e.what());
}
}
Status Get(ostream* out, int expected_code = 200) {
return Do(out, expected_code, "GET");
}
Status Post(ostream* out, int expected_code = 200) {
return Do(out, expected_code, "POST");
}
};
Status HttpGet(const string& host, const int32_t& port, const string& url_path, Status HttpGet(const string& host, const int32_t& port, const string& url_path,
ostream* out, int expected_code = 200, const string& method = "GET") { ostream* out, int expected_code = 200, const string& method = "GET") {
try { return HttpRequest{url_path, host, port}.Do(out, expected_code, method);
tcp::iostream request_stream;
request_stream.connect(host, lexical_cast<string>(port));
if (!request_stream) return Status("Could not connect request_stream");
request_stream << method << " " << url_path << " HTTP/1.1\r\n";
request_stream << "Host: " << host << ":" << port << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Cache-Control: no-cache\r\n";
request_stream << "Connection: close\r\n\r\n";
request_stream.flush();
string line1;
getline(request_stream, line1);
if (!request_stream) return Status("No response");
stringstream response_stream(line1);
string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
string status_message;
getline(response_stream,status_message);
if (!response_stream || http_version.substr(0,5) != "HTTP/") {
return Status("Malformed response");
}
if (status_code != expected_code) {
return Status(Substitute("Unexpected status code: $0", status_code));
}
(*out) << request_stream.rdbuf();
return Status::OK();
} catch (const std::exception& e){
return Status(e.what());
}
} }
TEST(Webserver, SmokeTest) { TEST(Webserver, SmokeTest) {
@@ -120,6 +146,30 @@ TEST(Webserver, SmokeTest) {
ASSERT_OK(HttpGet("localhost", FLAGS_webserver_port, "/", &contents)); ASSERT_OK(HttpGet("localhost", FLAGS_webserver_port, "/", &contents));
} }
void PostOnlyCallback(bool* success, const Webserver::WebRequest& req,
Document* document) {
*success = req.request_method == "POST";
}
TEST(Webserver, PostTest) {
MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics);
const string POST_TEST_PATH = "/post-test";
bool success = false;
Webserver::UrlCallback callback = bind<void>(PostOnlyCallback, &success , _1, _2);
webserver.RegisterUrlCallback(POST_TEST_PATH, "raw_text.tmpl", callback, false);
ASSERT_OK(webserver.Start());
stringstream contents;
HttpRequest req{POST_TEST_PATH};
ASSERT_OK(req.Get(&contents));
ASSERT_FALSE(success) << "GET unexpectedly succeeded";
ASSERT_OK(req.Post(&contents));
ASSERT_TRUE(success) << "POST unexpectedly failed";
}
void AssertArgsCallback(bool* success, const Webserver::WebRequest& req, void AssertArgsCallback(bool* success, const Webserver::WebRequest& req,
Document* document) { Document* document) {
const auto& args = req.parsed_args; const auto& args = req.parsed_args;
@@ -402,7 +452,68 @@ void CheckAuthMetrics(MetricGroup* metrics, int num_negotiate_success,
ASSERT_EQ(cookie_failure_metric->GetValue(), num_cookie_failure); ASSERT_EQ(cookie_failure_metric->GetValue(), num_cookie_failure);
} }
TEST(Webserver, TestWithSpnego) { void curl_version(string* curl_output, bool* curl_7_64_or_above = nullptr) {
// TODO(todd) IMPALA-8987: import curl into native-toolchain and test this with
// authentication.
RunShellProcess("curl --version", curl_output);
// Detect curl version. We only care about the major and minor.
std::regex curl_version_regex = std::regex("curl ([0-9]+)\\.([0-9]+)\\.[0-9]+");
std::smatch match_result;
ASSERT_TRUE(std::regex_search(*curl_output, match_result, curl_version_regex));
ASSERT_EQ(match_result.size(), 3);
int curl_major_version = std::stoi(match_result[1]);
int curl_minor_version = std::stoi(match_result[2]);
if (curl_7_64_or_above) {
*curl_7_64_or_above = curl_major_version > 7 ||
(curl_major_version == 7 && curl_minor_version >= 64);
}
cout << "Detected curl version " << std::to_string(curl_major_version) << "."
<< std::to_string(curl_minor_version)
<< (curl_7_64_or_above == nullptr ? " " :
(*curl_7_64_or_above ? " is at least 7.64" : " is below 7.64")) << endl;
}
string curl(const string& curl_options, int32_t port = FLAGS_webserver_port) {
string cmd = Substitute("curl -v -f $0 'http://127.0.0.1:$1'", curl_options, port);
cout << cmd << endl;
return cmd;
}
class CookieJar {
public:
CookieJar() : dir_(filesystem::unique_path()), path_(dir_ / "cookiejar") {
filesystem::create_directories(dir_);
cout << "Storing cookies in " << path_ << endl;
}
~CookieJar() {
filesystem::remove_all(dir_);
}
const filesystem::path& path() { return path_; }
string token() {
const char* rand_key = "&r=";
string rand, line;
ifstream cookie_file(path_.string());
while (cookie_file) {
getline(cookie_file, line);
size_t rand_idx = line.rfind(rand_key);
if (rand_idx != string::npos) {
// Relies on the random value being the last element in the cookie.
rand = line.substr(rand_idx + strlen(rand_key));
break;
}
}
return rand;
}
private:
const filesystem::path dir_, path_;
};
void EmptyCallback(const Webserver::WebRequest& req, Document* document) { }
TEST(Webserver, TestGetWithSpnego) {
MiniKdc kdc(MiniKdcOptions{}); MiniKdc kdc(MiniKdcOptions{});
KUDU_ASSERT_OK(kdc.Start()); KUDU_ASSERT_OK(kdc.Start());
kdc.SetKrb5Environment(); kdc.SetKrb5Environment();
@@ -420,6 +531,7 @@ TEST(Webserver, TestWithSpnego) {
MetricGroup metrics("webserver-test"); MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics); Webserver webserver("", FLAGS_webserver_port, &metrics);
ASSERT_OK(webserver.Start()); ASSERT_OK(webserver.Start());
webserver.RegisterUrlCallback("/", "raw_text.tmpl", EmptyCallback, false);
// Don't expect HTTP requests to work without Kerberos credentials. // Don't expect HTTP requests to work without Kerberos credentials.
stringstream contents; stringstream contents;
@@ -428,46 +540,21 @@ TEST(Webserver, TestWithSpnego) {
// There should be one failed auth attempt. // There should be one failed auth attempt.
CheckAuthMetrics(&metrics, 0, 1, 0, 0); CheckAuthMetrics(&metrics, 0, 1, 0, 0);
// TODO(todd) IMPALA-8987: import curl into native-toolchain and test this with
// authentication.
string curl_output; string curl_output;
RunShellProcess("curl --version", &curl_output); bool curl_7_64_or_above;
curl_version(&curl_output, &curl_7_64_or_above);
// Detect curl version. We only care about the major and minor.
std::regex curl_version_regex = std::regex("curl ([0-9]+)\\.([0-9]+)\\.[0-9]+");
std::smatch match_result;
ASSERT_TRUE(std::regex_search(curl_output, match_result, curl_version_regex));
ASSERT_EQ(match_result.size(), 3);
int curl_major_version = std::stoi(match_result[1]);
int curl_minor_version = std::stoi(match_result[2]);
bool curl_7_64_or_above = true;
if (curl_major_version < 7 ||
(curl_major_version == 7 && curl_minor_version < 64)) {
curl_7_64_or_above = false;
}
LOG(INFO) << "Detected curl version " << std::to_string(curl_major_version) << "."
<< std::to_string(curl_minor_version) << " "
<< (curl_7_64_or_above ? "is at least 7.64" : "is below 7.64");
if (curl_output.find("GSS-API") != string::npos if (curl_output.find("GSS-API") != string::npos
&& curl_output.find("SPNEGO") != string::npos) { && curl_output.find("SPNEGO") != string::npos) {
// Test that OPTIONS works with and without having kinit-ed. // Test that OPTIONS works with and without having kinit-ed.
string options_cmd = string options_cmd = curl("-X OPTIONS --negotiate -u :");
Substitute("curl -X OPTIONS -v --negotiate -u : 'http://127.0.0.1:$0'",
FLAGS_webserver_port);
system(options_cmd.c_str()); system(options_cmd.c_str());
KUDU_ASSERT_OK(kdc.Kinit("alice")); KUDU_ASSERT_OK(kdc.Kinit("alice"));
system(options_cmd.c_str()); system(options_cmd.c_str());
// Test that GET works with cookies. // Test that GET works with cookies.
filesystem::path cookie_dir = filesystem::unique_path(); CookieJar cookie;
filesystem::create_directories(cookie_dir);
filesystem::path cookie_path = cookie_dir / "cookiejar";
LOG(INFO) << "Storing cookies in " << cookie_path;
string curl_cmd = string curl_cmd =
Substitute("curl -c $0 -b $0 -X GET -v --negotiate -u : 'http://127.0.0.1:$1'", curl(Substitute("-c $0 -b $0 --negotiate -u :", cookie.path().string()));
cookie_path.string(), FLAGS_webserver_port);
// Run the command twice, the first time we should authenticate with SPNEGO, the // Run the command twice, the first time we should authenticate with SPNEGO, the
// second time with a cookie. // second time with a cookie.
system(Substitute("$0 && $0", curl_cmd).c_str()); system(Substitute("$0 && $0", curl_cmd).c_str());
@@ -486,11 +573,56 @@ TEST(Webserver, TestWithSpnego) {
// webserver uses a different HMAC key. See above note about curl 7.64.0 or above. // webserver uses a different HMAC key. See above note about curl 7.64.0 or above.
system(curl_cmd.c_str()); system(curl_cmd.c_str());
CheckAuthMetrics(&metrics2, 1, (curl_7_64_or_above ? 0 : 1), 0, 1); CheckAuthMetrics(&metrics2, 1, (curl_7_64_or_above ? 0 : 1), 0, 1);
filesystem::remove_all(cookie_dir);
} else { } else {
LOG(INFO) << "Skipping test, curl was not present or did not have the required " cout << "Skipping test, curl was not present or did not have the required "
<< "features: " << curl_output; << "features: " << curl_output << endl;
}
}
TEST(Webserver, TestPostWithSpnego) {
MiniKdc kdc(MiniKdcOptions{});
KUDU_ASSERT_OK(kdc.Start());
kdc.SetKrb5Environment();
string kt_path;
KUDU_ASSERT_OK(kdc.CreateServiceKeytab("HTTP/127.0.0.1", &kt_path));
CHECK_ERR(setenv("KRB5_KTNAME", kt_path.c_str(), 1));
KUDU_ASSERT_OK(kdc.CreateUserPrincipal("alice"));
gflags::FlagSaver saver;
FLAGS_webserver_require_spnego = true;
FLAGS_webserver_ldap_passwords_in_clear_ok = true;
FLAGS_cookie_require_secure = false;
MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics);
ASSERT_OK(webserver.Start());
webserver.RegisterUrlCallback("/", "raw_text.tmpl", EmptyCallback, false);
string curl_output;
curl_version(&curl_output);
if (curl_output.find("GSS-API") != string::npos
&& curl_output.find("SPNEGO") != string::npos) {
// POST fails without a header
ASSERT_NE(system(curl("-d '' --negotiate -u :").c_str()), 0);
// POST succeeds with X-Requested-By header
ASSERT_EQ(system(curl("-d '' -H 'X-Requested-By: me' --negotiate -u :").c_str()), 0);
CookieJar cookie;
// GET with SPNEGO succeeds and returns a cookie.
ASSERT_EQ(system(curl("--negotiate -u : -c " + cookie.path().string()).c_str()), 0);
// Verify we got a cookie and can read the random token.
string token = cookie.token();
ASSERT_FALSE(token.empty());
// Post with the cookie fails due to CSRF protection.
ASSERT_NE(system(curl("-d '' -b " + cookie.path().string()).c_str()), 0);
// Include the cookie's random token as csrf_token and request should succeed.
ASSERT_EQ(system(curl(Substitute(
"-b $0 -d 'csrf_token=$1'", cookie.path().string(), token)).c_str()), 0);
} else {
cout << "Skipping test, curl was not present or did not have the required "
<< "features: " << curl_output << endl;
} }
} }
@@ -500,17 +632,46 @@ TEST(Webserver, StartWithPasswordFileTest) {
auto password = auto password =
ScopedFlagSetter<string>::Make(&FLAGS_webserver_password_file, password_file.str()); ScopedFlagSetter<string>::Make(&FLAGS_webserver_password_file, password_file.str());
gflags::FlagSaver saver;
FLAGS_cookie_require_secure = false;
MetricGroup metrics("webserver-test"); MetricGroup metrics("webserver-test");
Webserver webserver("", FLAGS_webserver_port, &metrics); Webserver webserver("", FLAGS_webserver_port, &metrics);
if (FIPS_mode()) { if (FIPS_mode()) {
ASSERT_FALSE(webserver.Start().ok()); ASSERT_FALSE(webserver.Start().ok());
} else { } else {
ASSERT_OK(webserver.Start()); ASSERT_OK(webserver.Start());
webserver.RegisterUrlCallback("/", "raw_text.tmpl", EmptyCallback, false);
// Don't expect HTTP requests to work without a password // Don't expect HTTP requests to work without a password
stringstream contents; stringstream contents;
ASSERT_ERROR_MSG(HttpGet("localhost", FLAGS_webserver_port, "/", &contents), ASSERT_ERROR_MSG(HttpGet("localhost", FLAGS_webserver_port, "/", &contents),
"Unexpected status code: 401"); "Unexpected status code: 401");
// Succeeds with user and password
string curl_output;
curl_version(&curl_output);
ASSERT_EQ(system(curl("--digest -u test:test").c_str()), 0);
// POST is rejected without header
ASSERT_NE(system(curl("-d '' --digest -u test:test").c_str()), 0);
ASSERT_EQ(system(
curl("-d '' --digest -u test:test -H 'X-Requested-By: me'").c_str()), 0);
CookieJar cookie;
// GET with user and password succeeds and returns a cookie.
ASSERT_EQ(system(curl(Substitute("--digest -u test:test -c $0",
cookie.path().string())).c_str()), 0);
// Verify we got a cookie and can read the random token.
string token = cookie.token();
ASSERT_FALSE(token.empty());
// Post with the cookie fails due to CSRF protection.
ASSERT_NE(system(curl(Substitute("--digest -u test:test -b $0 -d ''",
cookie.path().string())).c_str()), 0);
// Include the cookie's random token as csrf_token and request should succeed.
ASSERT_EQ(system(curl(Substitute("--digest -u test:test -b $0 -d 'csrf_token=$1'",
cookie.path().string(), token)).c_str()), 0);
} }
} }

111
be/src/util/webserver.cc
View File

@@ -321,7 +321,7 @@ Webserver::Webserver(const string& interface, const int port, MetricGroup* metri
total_basic_auth_failure_ = total_basic_auth_failure_ =
metrics->AddCounter("impala.webserver.total-basic-auth-failure", 0); metrics->AddCounter("impala.webserver.total-basic-auth-failure", 0);
} }
if (use_cookies_ && (auth_mode_ == AuthMode::SPNEGO || auth_mode_ == AuthMode::LDAP)) { if (use_cookies_ && auth_mode_ != AuthMode::NONE) {
total_cookie_auth_success_ = total_cookie_auth_success_ =
metrics->AddCounter("impala.webserver.total-cookie-auth-success", 0); metrics->AddCounter("impala.webserver.total-cookie-auth-success", 0);
total_cookie_auth_failure_ = total_cookie_auth_failure_ =
@@ -552,13 +552,18 @@ void Webserver::Init() {
"$0://$1:$2", IsSecure() ? "https" : "http", hostname_, http_address_.port); "$0://$1:$2", IsSecure() ? "https" : "http", hostname_, http_address_.port);
} }
void Webserver::GetCommonJson( void Webserver::GetCommonJson(Document* document, const struct sq_connection* connection,
Document* document, const struct sq_connection* connection, const WebRequest& req) { const WebRequest& req, const std::string& csrf_token) {
DCHECK(document != nullptr); DCHECK(document != nullptr);
Value obj(kObjectType); Value obj(kObjectType);
obj.AddMember("process-name", obj.AddMember("process-name",
rapidjson::StringRef(google::ProgramInvocationShortName()), rapidjson::StringRef(google::ProgramInvocationShortName()),
document->GetAllocator()); document->GetAllocator());
if (!csrf_token.empty()) {
obj.AddMember("csrf_token",
Value(csrf_token.c_str(), document->GetAllocator()),
document->GetAllocator());
}
// If Apacke Knox is being used to proxy connections to the webserver, the // If Apacke Knox is being used to proxy connections to the webserver, the
// 'x-forwarded-context' header will be present. // 'x-forwarded-context' header will be present.
@@ -637,6 +642,16 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
vector<string> response_headers; vector<string> response_headers;
bool authenticated = false; bool authenticated = false;
// Random value from cookie that we'll also use as a csrf_token to implement the
// "Double Submit Cookie" and custom header (X-Requested-By) patterns for preventing
// cross-site request forgery (CSRF).
std::string cookie_rand_value;
// With JWTs we can skip CSRF protection because browsers won't send "Authorization:
// Bearer" headers automatically.
bool check_csrf_protection = true;
// Flags if we have a valid cookie to test for CSRF.
bool cookie_authenticated = false;
// Try authenticating with JWT token first, if enabled. // Try authenticating with JWT token first, if enabled.
if (use_jwt_) { if (use_jwt_) {
const char* auth_value = nullptr; const char* auth_value = nullptr;
@@ -654,6 +669,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (JWTTokenAuth(jwt_token, connection, request_info)) { if (JWTTokenAuth(jwt_token, connection, request_info)) {
total_jwt_token_auth_success_->Increment(1); total_jwt_token_auth_success_->Increment(1);
authenticated = true; authenticated = true;
check_csrf_protection = false;
// TODO: cookies are not added, but are not needed right now // TODO: cookies are not added, but are not needed right now
} else { } else {
LOG(INFO) << "Invalid JWT token provided: " << jwt_token; LOG(INFO) << "Invalid JWT token provided: " << jwt_token;
@@ -662,17 +678,25 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
} }
} }
} }
if (!authenticated) {
authenticated = auth_mode_ != AuthMode::SPNEGO && auth_mode_ != AuthMode::LDAP; if (!authenticated && auth_mode_ == AuthMode::NONE) {
// With AuthMode::NONE, any protection can be bypassed. We sometimes initialize a 2nd
// Metrics webserver using AuthMode::NONE, and metrics counters are not named
// uniquely to work with two webservers using cookies so we skip using cookies.
authenticated = true;
check_csrf_protection = false;
} }
// Try authenticating with a cookie, if enabled. // Try authenticating with a cookie, if enabled.
if (!authenticated && use_cookies_) { if (!authenticated && use_cookies_) {
const char* cookie_header = sq_get_header(connection, "Cookie"); const char* cookie_header = sq_get_header(connection, "Cookie");
string username; string username;
if (cookie_header != nullptr) { if (cookie_header != nullptr) {
Status cookie_status = AuthenticateCookie(hash_, cookie_header, &username); Status cookie_status =
AuthenticateCookie(hash_, cookie_header, &username, &cookie_rand_value);
if (cookie_status.ok()) { if (cookie_status.ok()) {
authenticated = true; authenticated = true;
cookie_authenticated = true;
request_info->remote_user = strdup(username.c_str()); request_info->remote_user = strdup(username.c_str());
total_cookie_auth_success_->Increment(1); total_cookie_auth_success_->Increment(1);
} else { } else {
@@ -684,6 +708,14 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
} }
} }
if (!authenticated && auth_mode_ == AuthMode::HTPASSWD) {
// Squeasel already handled HTPASSWD authentication. We still enable CSRF protection
// as browsers automatically include HTPASSWD credentials in requests, so add and use
// cookies to avoid requiring the custom header.
authenticated = true;
AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
}
// Connections originating from trusted domains should not require authentication. // Connections originating from trusted domains should not require authentication.
// Returns a cookie on the first successful auth attempt. This check is performed after // Returns a cookie on the first successful auth attempt. This check is performed after
// checking for cookie to avoid subsequent reverse DNS lookups which can be // checking for cookie to avoid subsequent reverse DNS lookups which can be
@@ -699,7 +731,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (TrustedDomainCheck(origin, connection, request_info)) { if (TrustedDomainCheck(origin, connection, request_info)) {
total_trusted_domain_check_success_->Increment(1); total_trusted_domain_check_success_->Increment(1);
authenticated = true; authenticated = true;
AddCookie(request_info, &response_headers); AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} }
} }
} }
@@ -712,7 +744,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (GetUsernameFromAuthHeader(connection, request_info, err_msg)) { if (GetUsernameFromAuthHeader(connection, request_info, err_msg)) {
total_trusted_auth_header_check_success_->Increment(1); total_trusted_auth_header_check_success_->Increment(1);
authenticated = true; authenticated = true;
AddCookie(request_info, &response_headers); AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} else { } else {
LOG(ERROR) << "Found trusted auth header but " << err_msg; LOG(ERROR) << "Found trusted auth header but " << err_msg;
} }
@@ -725,7 +757,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
HandleSpnego(connection, request_info, &response_headers); HandleSpnego(connection, request_info, &response_headers);
if (spnego_result == SQ_CONTINUE_HANDLING) { if (spnego_result == SQ_CONTINUE_HANDLING) {
// Spnego negotiation was successful. // Spnego negotiation was successful.
AddCookie(request_info, &response_headers); AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} else { } else {
// Spnego negotiation is incomplete or failed, stop processing the request. // Spnego negotiation is incomplete or failed, stop processing the request.
return spnego_result; return spnego_result;
@@ -736,7 +768,7 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
if (basic_status.ok()) { if (basic_status.ok()) {
// Basic auth was successful. // Basic auth was successful.
total_basic_auth_success_->Increment(1); total_basic_auth_success_->Increment(1);
AddCookie(request_info, &response_headers); AddCookie(request_info->remote_user, &response_headers, &cookie_rand_value);
} else { } else {
total_basic_auth_failure_->Increment(1); total_basic_auth_failure_->Increment(1);
if (!sq_get_header(connection, "Authorization")) { if (!sq_get_header(connection, "Authorization")) {
@@ -831,6 +863,44 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
req.post_data.append(buf, n); req.post_data.append(buf, n);
rem -= n; rem -= n;
} }
if (check_csrf_protection) {
// Always require 1) a csrf_token and cookie or 2) X-Requested-By header.
if (cookie_authenticated) {
std::vector<char> csrf_token(RAND_MAX_LENGTH+1, '\0');
int csrf_len = sq_get_var(req.post_data.c_str(), req.post_data.size(),
"csrf_token", csrf_token.data(), csrf_token.size());
if (csrf_len == -1) {
LOG(WARNING) << "CSRF protection: rejected POST without CSRF token";
sq_printf(connection, "HTTP/1.1 403 Forbidden\r\n");
return SQ_HANDLED_CLOSE_CONNECTION;
} else if (csrf_len == -2) {
LOG(WARNING) << "CSRF protection: CSRF token is too long";
sq_printf(connection, "HTTP/1.1 403 Forbidden\r\n");
return SQ_HANDLED_CLOSE_CONNECTION;
}
DCHECK(csrf_len >= 0 && csrf_len < csrf_token.size());
// Prevent CSRF for POSTs using the Double Submit Cookie pattern only if cookie
// authentication succeeded.
if (cookie_rand_value != csrf_token.data()) {
LOG(WARNING) << "CSRF protection: rejected POST with token mismatch: "
<< cookie_rand_value << " != " << csrf_token.data();
const char* msg = "please refresh the page and try again";
SendResponse(connection, "403 Forbidden", "text/plain", msg, response_headers);
return SQ_HANDLED_CLOSE_CONNECTION;
}
} else {
// Require a custom header matching csrf_token in the POST body.
const char* csrf_header = sq_get_header(connection, "X-Requested-By");
if (csrf_header == nullptr) {
const char* msg = "rejected POST missing X-Requested-By header";
LOG(WARNING) << "CSRF protection: " << msg;
SendResponse(connection, "403 Forbidden", "text/plain", msg, response_headers);
return SQ_HANDLED_CLOSE_CONNECTION;
}
}
}
} }
// The output of this page is accumulated into this stringstream. // The output of this page is accumulated into this stringstream.
@@ -842,7 +912,8 @@ sq_callback_result_t Webserver::BeginRequestCallback(struct sq_connection* conne
content_type = PLAIN; content_type = PLAIN;
url_handler->raw_callback()(req, &output, &response); url_handler->raw_callback()(req, &output, &response);
} else { } else {
RenderUrlWithTemplate(connection, req, *url_handler, &output, &content_type); RenderUrlWithTemplate(
connection, req, *url_handler, &output, &content_type, cookie_rand_value);
} }
VLOG(3) << "Rendering page " << request_info->uri << " took " VLOG(3) << "Rendering page " << request_info->uri << " took "
@@ -986,8 +1057,8 @@ Status Webserver::HandleBasic(struct sq_connection* connection,
return Status::Expected("Failed to authenticate to LDAP."); return Status::Expected("Failed to authenticate to LDAP.");
} }
void Webserver::AddCookie( void Webserver::AddCookie(const char* user, vector<string>* response_headers,
struct sq_request_info* request_info, vector<string>* response_headers) { string* cookie_rand_value) {
if (use_cookies_) { if (use_cookies_) {
// If cookie auth failed and we generated a 'delete cookie' header, remove it. // If cookie auth failed and we generated a 'delete cookie' header, remove it.
auto eq = [](const string& header) { return header.rfind("Set-Cookie", 0) == 0; }; auto eq = [](const string& header) { return header.rfind("Set-Cookie", 0) == 0; };
@@ -996,17 +1067,17 @@ void Webserver::AddCookie(
response_headers->erase(it); response_headers->erase(it);
} }
// Generate a cookie to return. // Generate a cookie to return.
response_headers->push_back( response_headers->push_back(Substitute("Set-Cookie: $0",
Substitute("Set-Cookie: $0", GenerateCookie(request_info->remote_user, hash_))); GenerateCookie(user, hash_, cookie_rand_value)));
} }
} }
void Webserver::RenderUrlWithTemplate(const struct sq_connection* connection, void Webserver::RenderUrlWithTemplate(const struct sq_connection* connection,
const WebRequest& req, const UrlHandler& url_handler, stringstream* output, const WebRequest& req, const UrlHandler& url_handler, stringstream* output,
ContentType* content_type) { ContentType* content_type, const std::string& csrf_token) {
Document document; Document document;
document.SetObject(); document.SetObject();
GetCommonJson(&document, connection, req); GetCommonJson(&document, connection, req, csrf_token);
const auto& arguments = req.parsed_args; const auto& arguments = req.parsed_args;
url_handler.callback()(req, &document); url_handler.callback()(req, &document);
@@ -1092,4 +1163,10 @@ Webserver::AuthMode Webserver::GetConfiguredAuthMode() {
} }
return AuthMode::NONE; return AuthMode::NONE;
} }
Webserver::ArgumentMap Webserver::GetVars(const std::string& data) {
ArgumentMap vars;
BuildArgumentMap(data, &vars);
return vars;
}
} }

14
be/src/util/webserver.h
View File

@@ -133,6 +133,9 @@ class Webserver {
/// Returns the authentication mode configured by the startup flags. /// Returns the authentication mode configured by the startup flags.
static AuthMode GetConfiguredAuthMode(); static AuthMode GetConfiguredAuthMode();
/// Parses form-uri-encoded data and returns key/value pairs.
static ArgumentMap GetVars(const std::string& data);
private: private:
/// Contains all information relevant to rendering one Url. Each Url has one callback /// Contains all information relevant to rendering one Url. Each Url has one callback
/// that produces the output to render. The callback either produces a Json document /// that produces the output to render. The callback either produces a Json document
@@ -212,7 +215,8 @@ class Webserver {
struct sq_request_info* request_info, std::vector<std::string>* response_headers); struct sq_request_info* request_info, std::vector<std::string>* response_headers);
// Adds a 'Set-Cookie' header to 'response_headers', if cookie support is enabled. // Adds a 'Set-Cookie' header to 'response_headers', if cookie support is enabled.
void AddCookie(struct sq_request_info* request_info, vector<string>* response_headers); // Returns the random value portion of the cookie in 'rand' for use in CSRF prevention.
void AddCookie(const char* user, vector<string>* response_headers, string* rand);
// Get username from Authorization header. // Get username from Authorization header.
bool GetUsernameFromAuthHeader(struct sq_connection* connection, bool GetUsernameFromAuthHeader(struct sq_connection* connection,
@@ -225,7 +229,8 @@ class Webserver {
/// pretty-printed. /// pretty-printed.
void RenderUrlWithTemplate(const struct sq_connection* connection, void RenderUrlWithTemplate(const struct sq_connection* connection,
const WebRequest& arguments, const UrlHandler& url_handler, const WebRequest& arguments, const UrlHandler& url_handler,
std::stringstream* output, ContentType* content_type); std::stringstream* output, ContentType* content_type,
const std::string& csrf_token);
/// Called when an error is encountered, e.g. when a handler for a URI cannot be found. /// Called when an error is encountered, e.g. when a handler for a URI cannot be found.
void ErrorHandler(const WebRequest& req, rapidjson::Document* document); void ErrorHandler(const WebRequest& req, rapidjson::Document* document);
@@ -233,12 +238,13 @@ class Webserver {
/// Builds a map of argument name to argument value from a typical URL argument /// Builds a map of argument name to argument value from a typical URL argument
/// string (that is, "key1=value1&key2=value2.."). If no value is given for a /// string (that is, "key1=value1&key2=value2.."). If no value is given for a
/// key, it is entered into the map as (key, ""). /// key, it is entered into the map as (key, "").
void BuildArgumentMap(const std::string& args, ArgumentMap* output); static void BuildArgumentMap(const std::string& args, ArgumentMap* output);
/// Adds a __common__ object to document with common data that every webpage might want /// Adds a __common__ object to document with common data that every webpage might want
/// to read (e.g. the names of links to write to the navbar). /// to read (e.g. the names of links to write to the navbar).
void GetCommonJson(rapidjson::Document* document, void GetCommonJson(rapidjson::Document* document,
const struct sq_connection* connection, const WebRequest& req); const struct sq_connection* connection, const WebRequest& req,
const std::string& csrf_token);
/// Lock guarding the path_handlers_ map /// Lock guarding the path_handlers_ map
boost::shared_mutex url_handlers_lock_; boost::shared_mutex url_handlers_lock_;

15
fe/src/test/java/org/apache/impala/customcluster/JwtHttpTest.java
View File

@@ -18,28 +18,23 @@
package org.apache.impala.customcluster; package org.apache.impala.customcluster;
import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail; import static org.junit.Assert.fail;
import com.google.common.collect.Lists;
import java.io.File; import java.io.File;
import java.io.IOException; import java.io.IOException;
import java.nio.file.Files; import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.nio.file.StandardCopyOption; import java.nio.file.StandardCopyOption;
import java.util.Arrays;
import java.util.HashMap; import java.util.HashMap;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import org.apache.hive.service.rpc.thrift.*; import org.apache.hive.service.rpc.thrift.*;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.THttpClient;
import org.apache.thrift.protocol.TBinaryProtocol; import org.apache.thrift.protocol.TBinaryProtocol;
import org.junit.After; import org.junit.After;
import org.junit.Before;
import org.junit.Test; import org.junit.Test;
/** /**
@@ -47,7 +42,7 @@ import org.junit.Test;
* JWT authentication is being used. * JWT authentication is being used.
*/ */
public class JwtHttpTest { public class JwtHttpTest {
Metrics metrics_ = new Metrics(); WebClient client_ = new WebClient();
/* Since we don't have Java version of JWT library, we use pre-calculated JWT token. /* Since we don't have Java version of JWT library, we use pre-calculated JWT token.
* The token and JWK set used in this test case were generated by using BE unit-test * The token and JWK set used in this test case were generated by using BE unit-test
@@ -87,7 +82,7 @@ public class JwtHttpTest {
// JWKS file. // JWKS file.
CustomClusterRunner.StartImpalaCluster(); CustomClusterRunner.StartImpalaCluster();
if (createJWKSForWebServer_) deleteTempJWKSFromWebServerRootDir(); if (createJWKSForWebServer_) deleteTempJWKSFromWebServerRootDir();
metrics_.Close(); client_.Close();
} }
private void createTempJWKSInWebServerRootDir(String srcFilename) { private void createTempJWKSInWebServerRootDir(String srcFilename) {
@@ -145,11 +140,11 @@ public class JwtHttpTest {
private void verifyJwtAuthMetrics(long expectedAuthSuccess, long expectedAuthFailure) private void verifyJwtAuthMetrics(long expectedAuthSuccess, long expectedAuthFailure)
throws Exception { throws Exception {
long actualAuthSuccess = long actualAuthSuccess =
(long) metrics_.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-success"); + "total-jwt-token-auth-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess); assertEquals(expectedAuthSuccess, actualAuthSuccess);
long actualAuthFailure = long actualAuthFailure =
(long) metrics_.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-failure"); + "total-jwt-token-auth-failure");
assertEquals(expectedAuthFailure, actualAuthFailure); assertEquals(expectedAuthFailure, actualAuthFailure);
} }

11
fe/src/test/java/org/apache/impala/customcluster/JwtWebserverTest.java
View File

@@ -19,7 +19,6 @@ package org.apache.impala.customcluster;
import static org.apache.impala.testutil.LdapUtil.*; import static org.apache.impala.testutil.LdapUtil.*;
import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertTrue;
import java.io.File; import java.io.File;
@@ -30,7 +29,7 @@ import java.util.HashMap;
import java.util.Map; import java.util.Map;
import com.google.common.collect.Range; import com.google.common.collect.Range;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.junit.After; import org.junit.After;
import org.junit.Test; import org.junit.Test;
@@ -40,7 +39,7 @@ import org.junit.Test;
public class JwtWebserverTest { public class JwtWebserverTest {
private static final Range<Long> zero = Range.closed(0L, 0L); private static final Range<Long> zero = Range.closed(0L, 0L);
Metrics metrics_ = new Metrics(TEST_USER_1, TEST_PASSWORD_1); WebClient client_ = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
public void setUp(String extraArgs, String startArgs) throws Exception { public void setUp(String extraArgs, String startArgs) throws Exception {
Map<String, String> env = new HashMap<>(); Map<String, String> env = new HashMap<>();
@@ -54,17 +53,17 @@ public class JwtWebserverTest {
public void cleanUp() throws Exception { public void cleanUp() throws Exception {
// Leave a cluster running with the default configuration. // Leave a cluster running with the default configuration.
CustomClusterRunner.StartImpalaCluster(); CustomClusterRunner.StartImpalaCluster();
metrics_.Close(); client_.Close();
} }
private void verifyJwtAuthMetrics( private void verifyJwtAuthMetrics(
Range<Long> expectedAuthSuccess, Range<Long> expectedAuthFailure) throws Exception { Range<Long> expectedAuthSuccess, Range<Long> expectedAuthFailure) throws Exception {
long actualAuthSuccess = long actualAuthSuccess =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-success"); (long) client_.getMetric("impala.webserver.total-jwt-token-auth-success");
assertTrue("Expected: " + expectedAuthSuccess + ", Actual: " + actualAuthSuccess, assertTrue("Expected: " + expectedAuthSuccess + ", Actual: " + actualAuthSuccess,
expectedAuthSuccess.contains(actualAuthSuccess)); expectedAuthSuccess.contains(actualAuthSuccess));
long actualAuthFailure = long actualAuthFailure =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-failure"); (long) client_.getMetric("impala.webserver.total-jwt-token-auth-failure");
assertTrue("Expected: " + expectedAuthFailure + ", Actual: " + actualAuthFailure, assertTrue("Expected: " + expectedAuthFailure + ", Actual: " + actualAuthFailure,
expectedAuthFailure.contains(actualAuthFailure)); expectedAuthFailure.contains(actualAuthFailure));
} }

23
fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
View File

@@ -34,10 +34,9 @@ import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles; import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule; import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.hive.service.rpc.thrift.*; import org.apache.hive.service.rpc.thrift.*;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.THttpClient;
import org.apache.thrift.protocol.TBinaryProtocol; import org.apache.thrift.protocol.TBinaryProtocol;
import org.junit.Before;
import org.junit.ClassRule; import org.junit.ClassRule;
import org.junit.Test; import org.junit.Test;
@@ -54,7 +53,7 @@ public class LdapHS2Test {
@ClassRule @ClassRule
public static CreateLdapServerRule serverRule = new CreateLdapServerRule(); public static CreateLdapServerRule serverRule = new CreateLdapServerRule();
Metrics metrics = new Metrics(); WebClient client_ = new WebClient();
public void setUp(String extraArgs) throws Exception { public void setUp(String extraArgs) throws Exception {
String uri = String uri =
@@ -112,26 +111,26 @@ public class LdapHS2Test {
private void verifyMetrics(long expectedBasicAuthSuccess, long expectedBasicAuthFailure) private void verifyMetrics(long expectedBasicAuthSuccess, long expectedBasicAuthFailure)
throws Exception { throws Exception {
long actualBasicAuthSuccess = (long) metrics.getMetric( long actualBasicAuthSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertEquals(expectedBasicAuthSuccess, actualBasicAuthSuccess); assertEquals(expectedBasicAuthSuccess, actualBasicAuthSuccess);
long actualBasicAuthFailure = (long) metrics.getMetric( long actualBasicAuthFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertEquals(expectedBasicAuthFailure, actualBasicAuthFailure); assertEquals(expectedBasicAuthFailure, actualBasicAuthFailure);
} }
private void verifyCookieMetrics( private void verifyCookieMetrics(
long expectedCookieAuthSuccess, long expectedCookieAuthFailure) throws Exception { long expectedCookieAuthSuccess, long expectedCookieAuthFailure) throws Exception {
long actualCookieAuthSuccess = (long) metrics.getMetric( long actualCookieAuthSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertEquals(expectedCookieAuthSuccess, actualCookieAuthSuccess); assertEquals(expectedCookieAuthSuccess, actualCookieAuthSuccess);
long actualCookieAuthFailure = (long) metrics.getMetric( long actualCookieAuthFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertEquals(expectedCookieAuthFailure, actualCookieAuthFailure); assertEquals(expectedCookieAuthFailure, actualCookieAuthFailure);
} }
private void verifyTrustedDomainMetrics(long expectedAuthSuccess) throws Exception { private void verifyTrustedDomainMetrics(long expectedAuthSuccess) throws Exception {
long actualAuthSuccess = (long) metrics long actualAuthSuccess = (long) client_
.getMetric("impala.thrift-server.hiveserver2-http-frontend." .getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-trusted-domain-check-success"); + "total-trusted-domain-check-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess); assertEquals(expectedAuthSuccess, actualAuthSuccess);
@@ -139,7 +138,7 @@ public class LdapHS2Test {
private void verifyTrustedAuthHeaderMetrics(long expectedAuthSuccess) throws Exception { private void verifyTrustedAuthHeaderMetrics(long expectedAuthSuccess) throws Exception {
long actualAuthSuccess = long actualAuthSuccess =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-trusted-auth-header-check-success"); + "total-trusted-auth-header-check-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess); assertEquals(expectedAuthSuccess, actualAuthSuccess);
} }
@@ -147,11 +146,11 @@ public class LdapHS2Test {
private void verifyJwtAuthMetrics(long expectedAuthSuccess, long expectedAuthFailure) private void verifyJwtAuthMetrics(long expectedAuthSuccess, long expectedAuthFailure)
throws Exception { throws Exception {
long actualAuthSuccess = long actualAuthSuccess =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-success"); + "total-jwt-token-auth-success");
assertEquals(expectedAuthSuccess, actualAuthSuccess); assertEquals(expectedAuthSuccess, actualAuthSuccess);
long actualAuthFailure = long actualAuthFailure =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-jwt-token-auth-failure"); + "total-jwt-token-auth-failure");
assertEquals(expectedAuthFailure, actualAuthFailure); assertEquals(expectedAuthFailure, actualAuthFailure);
} }
@@ -511,7 +510,7 @@ public class LdapHS2Test {
// Case 4: Verify that there are no changes in metrics for trusted auth // Case 4: Verify that there are no changes in metrics for trusted auth
// header check if the X-Trusted-Proxy-Auth-Header header is not present // header check if the X-Trusted-Proxy-Auth-Header header is not present
long successMetricBefore = long successMetricBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-trusted-auth-header-check-success"); + "total-trusted-auth-header-check-success");
headers.put("Authorization", "Basic VGVzdDFMZGFwOjEyMzQ1"); headers.put("Authorization", "Basic VGVzdDFMZGFwOjEyMzQ1");
headers.remove("X-Trusted-Proxy-Auth-Header"); headers.remove("X-Trusted-Proxy-Auth-Header");

12
fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
View File

@@ -28,7 +28,7 @@ import org.apache.directory.server.annotations.CreateLdapServer;
import org.apache.directory.server.annotations.CreateTransport; import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles; import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule; import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.junit.Assume; import org.junit.Assume;
import org.junit.ClassRule; import org.junit.ClassRule;
@@ -47,7 +47,7 @@ public class LdapImpalaShellTest {
// Includes a special character to test HTTP path encoding. // Includes a special character to test HTTP path encoding.
protected static final String delegateUser_ = "proxyUser$"; protected static final String delegateUser_ = "proxyUser$";
private Metrics metrics = new Metrics(); private WebClient client_ = new WebClient();
public void setUp(String extraArgs) throws Exception { public void setUp(String extraArgs) throws Exception {
String uri = String uri =
@@ -76,20 +76,20 @@ public class LdapImpalaShellTest {
private void verifyMetrics(Range<Long> expectedBasicSuccess, private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess, Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception { Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = (long) metrics.getMetric( long actualBasicSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess, assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess)); expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = (long) metrics.getMetric( long actualBasicFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure, assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure)); expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = (long) metrics.getMetric( long actualCookieSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess, assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess)); expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = (long) metrics.getMetric( long actualCookieFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure, assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure)); expectedCookieFailure.contains(actualCookieFailure));

26
fe/src/test/java/org/apache/impala/customcluster/LdapImpylaHttpTest.java
View File

@@ -22,8 +22,6 @@ import static org.junit.Assert.assertTrue;
import com.google.common.collect.Lists; import com.google.common.collect.Lists;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Arrays; import java.util.Arrays;
import java.util.List; import java.util.List;
@@ -33,7 +31,7 @@ import org.apache.directory.server.annotations.CreateLdapServer;
import org.apache.directory.server.annotations.CreateTransport; import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles; import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule; import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.apache.log4j.Logger; import org.apache.log4j.Logger;
import com.google.common.collect.Range; import com.google.common.collect.Range;
import org.junit.After; import org.junit.After;
@@ -72,7 +70,7 @@ public class LdapImpylaHttpTest {
// Includes a special character to test HTTP path encoding. // Includes a special character to test HTTP path encoding.
private static final String delegateUser_ = "proxyUser$"; private static final String delegateUser_ = "proxyUser$";
Metrics metrics = new Metrics(); WebClient client_ = new WebClient();
@Before @Before
public void setUp() throws Exception { public void setUp() throws Exception {
@@ -96,20 +94,20 @@ public class LdapImpylaHttpTest {
private void verifyMetrics(Range<Long> expectedBasicSuccess, private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess, Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception { Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = (long) metrics.getMetric( long actualBasicSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess, assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess)); expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = (long) metrics.getMetric( long actualBasicFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure, assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure)); expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = (long) metrics.getMetric( long actualCookieSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess, assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess)); expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = (long) metrics.getMetric( long actualCookieFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure, assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure)); expectedCookieFailure.contains(actualCookieFailure));
@@ -131,10 +129,10 @@ public class LdapImpylaHttpTest {
// 2. Invalid username password combination. Should fail. // 2. Invalid username password combination. Should fail.
long successBasicAuthBefore = long successBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-success"); + "total-basic-auth-success");
long successCookieAuthBefore = long successCookieAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-cookie-auth-success"); + "total-cookie-auth-success");
String[] invalidCmd = buildCommand("foo", "bar", null, null); String[] invalidCmd = buildCommand("foo", "bar", null, null);
RunShellCommand.Run( RunShellCommand.Run(
@@ -146,7 +144,7 @@ public class LdapImpylaHttpTest {
// 3. Without username and password. Should fail. // 3. Without username and password. Should fail.
long failedBasicAuthBefore = long failedBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-failure"); + "total-basic-auth-failure");
String[] noAuthCmd = {"impala-python", helper_, "--query", query_}; String[] noAuthCmd = {"impala-python", helper_, "--query", query_};
RunShellCommand.Run( RunShellCommand.Run(
@@ -168,7 +166,7 @@ public class LdapImpylaHttpTest {
// 5. Valid username, password, and HTTP cookie names. // 5. Valid username, password, and HTTP cookie names.
// Should succeed with cookie authentication. // Should succeed with cookie authentication.
successBasicAuthBefore = successBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-success"); + "total-basic-auth-success");
String[] validCookieNamesCmd = String[] validCookieNamesCmd =
buildCommand(testUser_, testPassword_, null, "impala.auth"); buildCommand(testUser_, testPassword_, null, "impala.auth");
@@ -181,10 +179,10 @@ public class LdapImpylaHttpTest {
// 6. Valid username and password, but HTTP cookie names don't consist of // 6. Valid username and password, but HTTP cookie names don't consist of
// "impala.auth". Should succeed with cookie authentication failures. // "impala.auth". Should succeed with cookie authentication failures.
successBasicAuthBefore = successBasicAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-basic-auth-success"); + "total-basic-auth-success");
successCookieAuthBefore = successCookieAuthBefore =
(long) metrics.getMetric("impala.thrift-server.hiveserver2-http-frontend." (long) client_.getMetric("impala.thrift-server.hiveserver2-http-frontend."
+ "total-cookie-auth-success"); + "total-cookie-auth-success");
String[] nonAuthCookieNamesCmd = buildCommand(testUser_, testPassword_, null, String[] nonAuthCookieNamesCmd = buildCommand(testUser_, testPassword_, null,
"impala.session.id"); "impala.session.id");

14
fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java
View File

@@ -34,9 +34,7 @@ import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles; import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule; import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.impala.testutil.ImpalaJdbcClient; import org.apache.impala.testutil.ImpalaJdbcClient;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.ClassRule; import org.junit.ClassRule;
import org.junit.Test; import org.junit.Test;
@@ -61,7 +59,7 @@ public class LdapJdbcTest extends JdbcTestBase {
private static final Range<Long> zero = Range.closed(0L, 0L); private static final Range<Long> zero = Range.closed(0L, 0L);
private static final Range<Long> one = Range.closed(1L, 1L); private static final Range<Long> one = Range.closed(1L, 1L);
Metrics metrics = new Metrics(); WebClient client_ = new WebClient();
public LdapJdbcTest(String connectionType) { super(connectionType); } public LdapJdbcTest(String connectionType) { super(connectionType); }
@@ -87,20 +85,20 @@ public class LdapJdbcTest extends JdbcTestBase {
private void verifyMetrics(Range<Long> expectedBasicSuccess, private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess, Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception { Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = (long) metrics.getMetric( long actualBasicSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess, assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess)); expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = (long) metrics.getMetric( long actualBasicFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure, assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure)); expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = (long) metrics.getMetric( long actualCookieSuccess = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess, assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess)); expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = (long) metrics.getMetric( long actualCookieFailure = (long) client_.getMetric(
"impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure"); "impala.thrift-server.hiveserver2-http-frontend.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure, assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure)); expectedCookieFailure.contains(actualCookieFailure));

238
fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java
View File

@@ -21,12 +21,15 @@ import static org.apache.impala.testutil.LdapUtil.*;
import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue; import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import java.io.File; import java.io.File;
import java.io.IOException; import java.io.IOException;
import java.net.URL; import java.net.URL;
import java.net.URLConnection; import java.net.URLConnection;
import java.util.ArrayList;
import java.util.HashMap; import java.util.HashMap;
import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
@@ -40,10 +43,14 @@ import org.apache.directory.server.annotations.CreateTransport;
import org.apache.directory.server.core.annotations.ApplyLdifFiles; import org.apache.directory.server.core.annotations.ApplyLdifFiles;
import org.apache.directory.server.core.integ.CreateLdapServerRule; import org.apache.directory.server.core.integ.CreateLdapServerRule;
import org.apache.hive.service.rpc.thrift.*; import org.apache.hive.service.rpc.thrift.*;
import org.apache.impala.util.Metrics; import org.apache.http.cookie.Cookie;
import org.apache.log4j.Logger; import org.apache.http.message.BasicHeader;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.NameValuePair;
import org.apache.impala.testutil.WebClient;
import org.apache.thrift.protocol.TBinaryProtocol; import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.transport.THttpClient; import org.apache.thrift.transport.THttpClient;
import org.json.simple.JSONObject;
import org.junit.After; import org.junit.After;
import org.junit.ClassRule; import org.junit.ClassRule;
import org.junit.Test; import org.junit.Test;
@@ -54,13 +61,12 @@ import org.junit.Test;
transports = { @CreateTransport(protocol = "LDAP", address = "localhost") }) transports = { @CreateTransport(protocol = "LDAP", address = "localhost") })
@ApplyLdifFiles({"users.ldif"}) @ApplyLdifFiles({"users.ldif"})
public class LdapWebserverTest { public class LdapWebserverTest {
private static final Logger LOG = Logger.getLogger(LdapWebserverTest.class);
@ClassRule @ClassRule
public static CreateLdapServerRule serverRule = new CreateLdapServerRule(); public static CreateLdapServerRule serverRule = new CreateLdapServerRule();
private static final Range<Long> zero = Range.closed(0L, 0L); private static final Range<Long> zero = Range.closed(0L, 0L);
Metrics metrics_ = new Metrics(TEST_USER_1, TEST_PASSWORD_1); WebClient client_ = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
public void setUp(String extraArgs, String startArgs) throws Exception { public void setUp(String extraArgs, String startArgs) throws Exception {
String uri = String uri =
@@ -75,38 +81,38 @@ public class LdapWebserverTest {
env.put("IMPALA_WEBSERVER_USERNAME", TEST_USER_1); env.put("IMPALA_WEBSERVER_USERNAME", TEST_USER_1);
env.put("IMPALA_WEBSERVER_PASSWORD", TEST_PASSWORD_1); env.put("IMPALA_WEBSERVER_PASSWORD", TEST_PASSWORD_1);
int ret = CustomClusterRunner.StartImpalaCluster(impalaArgs, env, startArgs); int ret = CustomClusterRunner.StartImpalaCluster(impalaArgs, env, startArgs);
assertEquals(ret, 0); assertEquals(0, ret);
} }
@After @After
public void cleanUp() throws IOException { public void cleanUp() throws IOException {
metrics_.Close(); client_.Close();
} }
private void verifyMetrics(Range<Long> expectedBasicSuccess, private void verifyMetrics(Range<Long> expectedBasicSuccess,
Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess, Range<Long> expectedBasicFailure, Range<Long> expectedCookieSuccess,
Range<Long> expectedCookieFailure) throws Exception { Range<Long> expectedCookieFailure) throws Exception {
long actualBasicSuccess = long actualBasicSuccess =
(long) metrics_.getMetric("impala.webserver.total-basic-auth-success"); (long) client_.getMetric("impala.webserver.total-basic-auth-success");
assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess, assertTrue("Expected: " + expectedBasicSuccess + ", Actual: " + actualBasicSuccess,
expectedBasicSuccess.contains(actualBasicSuccess)); expectedBasicSuccess.contains(actualBasicSuccess));
long actualBasicFailure = long actualBasicFailure =
(long) metrics_.getMetric("impala.webserver.total-basic-auth-failure"); (long) client_.getMetric("impala.webserver.total-basic-auth-failure");
assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure, assertTrue("Expected: " + expectedBasicFailure + ", Actual: " + actualBasicFailure,
expectedBasicFailure.contains(actualBasicFailure)); expectedBasicFailure.contains(actualBasicFailure));
long actualCookieSuccess = long actualCookieSuccess =
(long) metrics_.getMetric("impala.webserver.total-cookie-auth-success"); (long) client_.getMetric("impala.webserver.total-cookie-auth-success");
assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess, assertTrue("Expected: " + expectedCookieSuccess + ", Actual: " + actualCookieSuccess,
expectedCookieSuccess.contains(actualCookieSuccess)); expectedCookieSuccess.contains(actualCookieSuccess));
long actualCookieFailure = long actualCookieFailure =
(long) metrics_.getMetric("impala.webserver.total-cookie-auth-failure"); (long) client_.getMetric("impala.webserver.total-cookie-auth-failure");
assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure, assertTrue("Expected: " + expectedCookieFailure + ", Actual: " + actualCookieFailure,
expectedCookieFailure.contains(actualCookieFailure)); expectedCookieFailure.contains(actualCookieFailure));
} }
private void verifyTrustedDomainMetrics(Range<Long> expectedSuccess) throws Exception { private void verifyTrustedDomainMetrics(Range<Long> expectedSuccess) throws Exception {
long actualSuccess = (long) metrics_ long actualSuccess = (long) client_
.getMetric("impala.webserver.total-trusted-domain-check-success"); .getMetric("impala.webserver.total-trusted-domain-check-success");
assertTrue("Expected: " + expectedSuccess + ", Actual: " + actualSuccess, assertTrue("Expected: " + expectedSuccess + ", Actual: " + actualSuccess,
expectedSuccess.contains(actualSuccess)); expectedSuccess.contains(actualSuccess));
@@ -114,7 +120,7 @@ public class LdapWebserverTest {
private void verifyTrustedAuthHeaderMetrics(Range<Long> expectedSuccess) private void verifyTrustedAuthHeaderMetrics(Range<Long> expectedSuccess)
throws Exception { throws Exception {
long actualSuccess = (long) metrics_.getMetric( long actualSuccess = (long) client_.getMetric(
"impala.webserver.total-trusted-auth-header-check-success"); "impala.webserver.total-trusted-auth-header-check-success");
assertTrue("Expected: " + expectedSuccess + ", Actual: " + actualSuccess, assertTrue("Expected: " + expectedSuccess + ", Actual: " + actualSuccess,
expectedSuccess.contains(actualSuccess)); expectedSuccess.contains(actualSuccess));
@@ -123,11 +129,11 @@ public class LdapWebserverTest {
private void verifyJwtAuthMetrics( private void verifyJwtAuthMetrics(
Range<Long> expectedAuthSuccess, Range<Long> expectedAuthFailure) throws Exception { Range<Long> expectedAuthSuccess, Range<Long> expectedAuthFailure) throws Exception {
long actualAuthSuccess = long actualAuthSuccess =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-success"); (long) client_.getMetric("impala.webserver.total-jwt-token-auth-success");
assertTrue("Expected: " + expectedAuthSuccess + ", Actual: " + actualAuthSuccess, assertTrue("Expected: " + expectedAuthSuccess + ", Actual: " + actualAuthSuccess,
expectedAuthSuccess.contains(actualAuthSuccess)); expectedAuthSuccess.contains(actualAuthSuccess));
long actualAuthFailure = long actualAuthFailure =
(long) metrics_.getMetric("impala.webserver.total-jwt-token-auth-failure"); (long) client_.getMetric("impala.webserver.total-jwt-token-auth-failure");
assertTrue("Expected: " + expectedAuthFailure + ", Actual: " + actualAuthFailure, assertTrue("Expected: " + expectedAuthFailure + ", Actual: " + actualAuthFailure,
expectedAuthFailure.contains(actualAuthFailure)); expectedAuthFailure.contains(actualAuthFailure));
} }
@@ -140,14 +146,14 @@ public class LdapWebserverTest {
verifyMetrics(Range.atLeast(1L), zero, Range.atLeast(1L), zero); verifyMetrics(Range.atLeast(1L), zero, Range.atLeast(1L), zero);
// Attempt to access the webserver without a username/password. // Attempt to access the webserver without a username/password.
Metrics noUsername = new Metrics(); WebClient noUsername = new WebClient();
String result = noUsername.readContent("/"); String result = noUsername.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is one unsuccessful auth attempt. // Check that there is one unsuccessful auth attempt.
verifyMetrics(Range.atLeast(1L), Range.closed(1L, 1L), Range.atLeast(1L), zero); verifyMetrics(Range.atLeast(1L), Range.closed(1L, 1L), Range.atLeast(1L), zero);
// Attempt to access the webserver with invalid username/password. // Attempt to access the webserver with invalid username/password.
Metrics invalidUserPass = new Metrics("invalid", "invalid"); WebClient invalidUserPass = new WebClient("invalid", "invalid");
result = invalidUserPass.readContent("/"); result = invalidUserPass.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is now two unsuccessful auth attempts. // Check that there is now two unsuccessful auth attempts.
@@ -174,23 +180,23 @@ public class LdapWebserverTest {
// Access the webserver with a user that passes the group filter but not the user // Access the webserver with a user that passes the group filter but not the user
// filter, should fail. // filter, should fail.
Metrics metricsUser2 = new Metrics(TEST_USER_2, TEST_PASSWORD_2); WebClient user2 = new WebClient(TEST_USER_2, TEST_PASSWORD_2);
String result = metricsUser2.readContent("/"); String result = user2.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is one unsuccessful auth attempt. // Check that there is one unsuccessful auth attempt.
verifyMetrics(Range.atLeast(1L), Range.closed(1L, 1L), Range.atLeast(1L), zero); verifyMetrics(Range.atLeast(1L), Range.closed(1L, 1L), Range.atLeast(1L), zero);
// Access the webserver with a user that passes the user filter but not the group // Access the webserver with a user that passes the user filter but not the group
// filter, should fail. // filter, should fail.
Metrics metricsUser3 = new Metrics(TEST_USER_3, TEST_PASSWORD_3); WebClient user3 = new WebClient(TEST_USER_3, TEST_PASSWORD_3);
result = metricsUser3.readContent("/"); result = user3.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is now two unsuccessful auth attempts. // Check that there is now two unsuccessful auth attempts.
verifyMetrics(Range.atLeast(1L), Range.closed(2L, 2L), Range.atLeast(1L), zero); verifyMetrics(Range.atLeast(1L), Range.closed(2L, 2L), Range.atLeast(1L), zero);
// Access the webserver with a user that doesn't pass either filter, should fail. // Access the webserver with a user that doesn't pass either filter, should fail.
Metrics metricsUser4 = new Metrics(TEST_USER_4, TEST_PASSWORD_4); WebClient user4 = new WebClient(TEST_USER_4, TEST_PASSWORD_4);
result = metricsUser4.readContent("/"); result = user4.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Check that there is now three unsuccessful auth attempts. // Check that there is now three unsuccessful auth attempts.
verifyMetrics(Range.atLeast(1L), Range.closed(3L, 3L), Range.atLeast(1L), zero); verifyMetrics(Range.atLeast(1L), Range.closed(3L, 3L), Range.atLeast(1L), zero);
@@ -205,17 +211,17 @@ public class LdapWebserverTest {
// Use 'per_impalad_args' to turn the metrics webserver on only for the first impalad. // Use 'per_impalad_args' to turn the metrics webserver on only for the first impalad.
setUp("", "--per_impalad_args=--metrics_webserver_port=25030"); setUp("", "--per_impalad_args=--metrics_webserver_port=25030");
// Attempt to access the regular webserver without a username/password, should fail. // Attempt to access the regular webserver without a username/password, should fail.
Metrics noUsername = new Metrics(); WebClient noUsername = new WebClient();
String result = noUsername.readContent("/"); String result = noUsername.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Attempt to access the regular webserver with invalid username/password. // Attempt to access the regular webserver with invalid username/password.
Metrics invalidUserPass = new Metrics("invalid", "invalid"); WebClient invalidUserPass = new WebClient("invalid", "invalid");
result = invalidUserPass.readContent("/"); result = invalidUserPass.readContent("/");
assertTrue(result, result.contains("Must authenticate with Basic authentication.")); assertTrue(result, result.contains("Must authenticate with Basic authentication."));
// Attempt to access the metrics webserver without a username/password. // Attempt to access the metrics webserver without a username/password.
Metrics noUsernameMetrics = new Metrics(25030); WebClient noUsernameMetrics = new WebClient(25030);
// Should succeed for the metrics endpoints. // Should succeed for the metrics endpoints.
for (String endpoint : for (String endpoint :
new String[] {"/metrics", "/jsonmetrics", "/metrics_prometheus", "/healthz"}) { new String[] {"/metrics", "/jsonmetrics", "/metrics_prometheus", "/healthz"}) {
@@ -266,7 +272,7 @@ public class LdapWebserverTest {
// Case 6: Verify that there are no changes in metrics for trusted domain // Case 6: Verify that there are no changes in metrics for trusted domain
// check if the X-Forwarded-For header is not present // check if the X-Forwarded-For header is not present
long successMetricBefore = (long) metrics_ long successMetricBefore = (long) client_
.getMetric("impala.webserver.total-trusted-domain-check-success"); .getMetric("impala.webserver.total-trusted-domain-check-success");
attemptConnection("Basic VGVzdDFMZGFwOjEyMzQ1", null, false); attemptConnection("Basic VGVzdDFMZGFwOjEyMzQ1", null, false);
verifyTrustedDomainMetrics(Range.closed(successMetricBefore, successMetricBefore)); verifyTrustedDomainMetrics(Range.closed(successMetricBefore, successMetricBefore));
@@ -295,7 +301,7 @@ public class LdapWebserverTest {
// Case 4: Verify that there are no changes in metrics for trusted auth header // Case 4: Verify that there are no changes in metrics for trusted auth header
// check if the trusted auth header is not present. // check if the trusted auth header is not present.
long successMetricBefore = (long) metrics_.getMetric( long successMetricBefore = (long) client_.getMetric(
"impala.webserver.total-trusted-auth-header-check-success"); "impala.webserver.total-trusted-auth-header-check-success");
attemptConnection("Basic VGVzdDFMZGFwOjEyMzQ1", null, false); attemptConnection("Basic VGVzdDFMZGFwOjEyMzQ1", null, false);
verifyTrustedAuthHeaderMetrics( verifyTrustedAuthHeaderMetrics(
@@ -368,29 +374,197 @@ public class LdapWebserverTest {
String cancelQueryUrl = String.format("/cancel_query?query_id=%s", queryId); String cancelQueryUrl = String.format("/cancel_query?query_id=%s", queryId);
String textProfileUrl = String.format("/query_profile_plain_text?query_id=%s", String textProfileUrl = String.format("/query_profile_plain_text?query_id=%s",
queryId); queryId);
metrics_.readContent(cancelQueryUrl); client_.readContent(cancelQueryUrl);
String response = metrics_.readContent(textProfileUrl); String response = client_.readContent(textProfileUrl);
String cancelStatus = String.format("Cancelled from Impala&apos;s debug web interface" String cancelStatus = String.format("Cancelled from Impala&apos;s debug web interface"
+ " by user: &apos;%s&apos; at", TEST_USER_1); + " by user: &apos;%s&apos; at", TEST_USER_1);
assertTrue(response.contains(cancelStatus)); assertTrue(response.contains(cancelStatus));
// Wait for logs to flush // Wait for logs to flush
TimeUnit.SECONDS.sleep(6); TimeUnit.SECONDS.sleep(6);
response = metrics_.readContent("/logs"); response = client_.readContent("/logs");
assertTrue(response.contains(cancelStatus)); assertTrue(response.contains(cancelStatus));
// Session closing from the WebUI does not produce the cause message in the profile, // Session closing from the WebUI does not produce the cause message in the profile,
// so we will skip checking the runtime profile. // so we will skip checking the runtime profile.
String sessionId = PrintId(openResp.getSessionHandle().getSessionId()); String sessionId = PrintId(openResp.getSessionHandle().getSessionId());
String closeSessionUrl = String.format("/close_session?session_id=%s", sessionId); String closeSessionUrl = String.format("/close_session?session_id=%s", sessionId);
metrics_.readContent(closeSessionUrl); client_.readContent(closeSessionUrl);
// Wait for logs to flush // Wait for logs to flush
TimeUnit.SECONDS.sleep(6); TimeUnit.SECONDS.sleep(6);
String closeStatus = String.format("Session closed from Impala&apos;s debug web" String closeStatus = String.format("Session closed from Impala&apos;s debug web"
+ " interface by user: &apos;%s&apos; at", TEST_USER_1); + " interface by user: &apos;%s&apos; at", TEST_USER_1);
response = metrics_.readContent("/logs"); response = client_.readContent("/logs");
assertTrue(response.contains(closeStatus)); assertTrue(response.contains(closeStatus));
} }
/*
* Test that we can set glog level.
*/
@Test
public void testSetGLogLevel() throws Exception {
setUp("", "");
// Validate defaults
JSONObject json = client_.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
// Test GET set_glog_level returns an error
json = client_.jsonGet("/set_glog_level?glog=0&json");
assertEquals("1", json.get("glog_level"));
assertEquals("Use form input to update glog level", json.get("error"));
// Test GET reset_glog_level returns an error
json = client_.jsonGet("/reset_glog_level?json");
assertEquals("1", json.get("glog_level"));
assertEquals("Use form input to reset glog level", json.get("error"));
// Clients persist state like 400 errors and cookies. Use new client for each test.
BasicHeader[] headers = { new BasicHeader("X-Requested-By", "anything") };
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("glog", "0"));
// Test POST set_glog_level fails
WebClient client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
String body = client.post("/set_glog_level?json", null, params, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST reset_glog_level fails
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
body = client.post("/reset_glog_level?json", null, null, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST set_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/set_glog_level?json", headers, params);
assertEquals("0", json.get("glog_level"));
// Test POST reset_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/reset_glog_level?json", headers, null);
assertEquals("1", json.get("glog_level"));
// Test POST set_glog_level with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
body = client.post("/set_glog_level?json", null, params, 403);
assertEquals("", body);
// Test POST reset_glog_level with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
body = client.post("/reset_glog_level?json", null, null, 403);
assertEquals("", body);
// Create a new client, get a cookie, and add csrf_token based on the cookie
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("1", json.get("glog_level"));
String rand = getRandToken(client.getCookies());
params.add(new BasicNameValuePair("csrf_token", rand));
// Test POST set_glog_level with cookie and csrf_token succeeds
json = client.jsonPost("/set_glog_level?json", null, params);
assertEquals("0", json.get("glog_level"));
// Test POST reset_glog_level with cookie and csrf_token succeeds
json = client.jsonPost("/reset_glog_level?json", null, params);
assertEquals("1", json.get("glog_level"));
}
/*
* Test that we can set java log level.
*/
@Test
public void testSetJavaLogLevel() throws Exception {
setUp("", "");
// Validate defaults
JSONObject json = client_.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
// Test GET set_java_loglevel does nothing
json = client_.jsonGet("/set_java_loglevel?class=org.apache&level=WARN&json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
assertEquals("Use form input to update java log levels", json.get("error"));
// Test GET reset_java_loglevel does nothing
json = client_.jsonGet("/reset_java_loglevel?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
assertEquals("Use form input to reset java log levels", json.get("error"));
// Clients persist state like 400 errors and cookies. Use new client for each test.
BasicHeader[] headers = { new BasicHeader("X-Requested-By", "anything") };
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("class", "org.apache"));
params.add(new BasicNameValuePair("level", "WARN"));
// Test POST set_java_loglevel fails
WebClient client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
String body = client.post("/set_java_loglevel?json", null, params, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST reset_java_loglevel fails
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
body = client.post("/reset_java_loglevel?json", null, null, 403);
assertEquals("rejected POST missing X-Requested-By header", body);
// Test POST set_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/set_java_loglevel?json", headers, params);
assertEquals("org.apache : WARN\norg.apache.impala : DEBUG\n",
json.get("get_java_loglevel_result"));
// Test POST reset_glog_level with X-Requested-By succeeds
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonPost("/reset_java_loglevel?json", headers, null);
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
// Test POST set_java_loglevel with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
body = client.post("/set_java_loglevel?json", null, params, 403);
assertEquals("", body);
// Test POST reset_java_loglevel with cookie gives 403
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
body = client.post("/reset_java_loglevel?json", null, null, 403);
assertEquals("", body);
// Create a new client, get a cookie, and add csrf_token based on the cookie
client = new WebClient(TEST_USER_1, TEST_PASSWORD_1);
json = client.jsonGet("/log_level?json");
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
String rand = getRandToken(client.getCookies());
params.add(new BasicNameValuePair("csrf_token", rand));
// Test POST set_java_loglevel with cookie and csrf_token succeeds
json = client.jsonPost("/set_java_loglevel?json", null, params);
assertEquals("org.apache : WARN\norg.apache.impala : DEBUG\n",
json.get("get_java_loglevel_result"));
// Test POST reset_java_loglevel with cookie and csrf_token succeeds
json = client.jsonPost("/reset_java_loglevel?json", null, params);
assertEquals("org.apache.impala : DEBUG\n", json.get("get_java_loglevel_result"));
}
private String getRandToken(List<Cookie> cookies) {
for (Cookie cookie : cookies) {
String[] tokens = cookie.getValue().split("&");
for (String token : tokens) {
if (token.charAt(0) == 'r' && token.charAt(1) == '=') {
String rand = token.substring(2);
assertTrue("Expected number: " + rand, rand.matches("^[1-9][0-9]*$"));
return rand;
}
}
}
fail("Expected cookie to contain random number");
return "";
}
// Helper method to make a get call to the webserver using the input basic // Helper method to make a get call to the webserver using the input basic
// auth token, x-forward-for and X-Trusted-Proxy-Auth-Header token. // auth token, x-forward-for and X-Trusted-Proxy-Auth-Header token.
private void attemptConnection(String basic_auth_token, String xff_address, private void attemptConnection(String basic_auth_token, String xff_address,

16
fe/src/test/java/org/apache/impala/service/JdbcTest.java
View File

@@ -36,7 +36,7 @@ import java.util.Map;
import org.apache.impala.testutil.ImpalaJdbcClient; import org.apache.impala.testutil.ImpalaJdbcClient;
import org.apache.impala.testutil.TestUtils; import org.apache.impala.testutil.TestUtils;
import org.apache.impala.util.Metrics; import org.apache.impala.testutil.WebClient;
import org.junit.AfterClass; import org.junit.AfterClass;
import org.junit.Before; import org.junit.Before;
import org.junit.Test; import org.junit.Test;
@@ -615,7 +615,7 @@ public class JdbcTest extends JdbcTestBase {
@Test @Test
public void testConcurrentSessionMixedIdleTimeout() throws Exception { public void testConcurrentSessionMixedIdleTimeout() throws Exception {
// Test for concurrent idle sessions' expiration with mixed timeout durations. // Test for concurrent idle sessions' expiration with mixed timeout durations.
Metrics metrics = new Metrics(); WebClient client = new WebClient();
List<Integer> timeoutPeriods = Arrays.asList(0, 3, 15); List<Integer> timeoutPeriods = Arrays.asList(0, 3, 15);
List<Connection> connections = new ArrayList<>(); List<Connection> connections = new ArrayList<>();
@@ -626,9 +626,9 @@ public class JdbcTest extends JdbcTestBase {
createConnection(ImpalaJdbcClient.getNoAuthConnectionStr(connectionType_))); createConnection(ImpalaJdbcClient.getNoAuthConnectionStr(connectionType_)));
} }
Long numOpenSessions = (Long)metrics.getMetric( Long numOpenSessions = (Long)client.getMetric(
"impala-server.num-open-hiveserver2-sessions"); "impala-server.num-open-hiveserver2-sessions");
Long numExpiredSessions = (Long)metrics.getMetric( Long numExpiredSessions = (Long)client.getMetric(
"impala-server.num-sessions-expired"); "impala-server.num-sessions-expired");
for (int i = 0; i < connections.size(); ++i) { for (int i = 0; i < connections.size(); ++i) {
@@ -642,9 +642,9 @@ public class JdbcTest extends JdbcTestBase {
lastTimeSessionActive.add(System.currentTimeMillis() / 1000); lastTimeSessionActive.add(System.currentTimeMillis() / 1000);
} }
assertEquals(numOpenSessions, (Long)metrics.getMetric( assertEquals(numOpenSessions, (Long)client.getMetric(
"impala-server.num-open-hiveserver2-sessions")); "impala-server.num-open-hiveserver2-sessions"));
assertEquals(numExpiredSessions, (Long)metrics.getMetric( assertEquals(numExpiredSessions, (Long)client.getMetric(
"impala-server.num-sessions-expired")); "impala-server.num-sessions-expired"));
for (int timeout : timeoutPeriods) { for (int timeout : timeoutPeriods) {
@@ -679,9 +679,9 @@ public class JdbcTest extends JdbcTestBase {
} }
} }
assertEquals(numOpenSessions, (Long)metrics.getMetric( assertEquals(numOpenSessions, (Long)client.getMetric(
"impala-server.num-open-hiveserver2-sessions")); "impala-server.num-open-hiveserver2-sessions"));
assertEquals(numExpiredSessions, (Long)metrics.getMetric( assertEquals(numExpiredSessions, (Long)client.getMetric(
"impala-server.num-sessions-expired")); "impala-server.num-sessions-expired"));
} }

112
fe/src/test/java/org/apache/impala/util/Metrics.java → fe/src/test/java/org/apache/impala/testutil/WebClient.java
View File

@@ -15,57 +15,69 @@
// specific language governing permissions and limitations // specific language governing permissions and limitations
// under the License. // under the License.
package org.apache.impala.util; package org.apache.impala.testutil;
import java.io.IOException; import java.io.IOException;
import java.util.List;
import org.apache.http.auth.AuthScope; import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials; import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.AuthCache; import org.apache.http.client.AuthCache;
import org.apache.http.impl.client.BasicAuthCache;
import org.apache.http.client.CredentialsProvider; import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext; import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.cookie.Cookie;
import org.apache.http.Header;
import org.apache.http.HttpHost; import org.apache.http.HttpHost;
import org.apache.http.NameValuePair;
import org.apache.http.impl.auth.BasicScheme; import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.impl.client.BasicAuthCache;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.BasicCredentialsProvider; import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients; import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils; import org.apache.http.util.EntityUtils;
import org.json.simple.JSONObject; import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser; import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
/** /**
* Utility class for retrieving metrics from the Impala webserver. * Utility class for interacting with the Impala webserver.
*/ */
public class Metrics { public class WebClient {
private final static String WEBSERVER_HOST = "localhost"; private final static String WEBSERVER_HOST = "localhost";
private final static int DEFAULT_WEBSERVER_PORT = 25000; private final static int DEFAULT_WEBSERVER_PORT = 25000;
private final static String JSON_METRICS = "/jsonmetrics?json"; private final static String JSON_METRICS = "/jsonmetrics?json";
private CloseableHttpClient httpClient_; private CloseableHttpClient httpClient_;
private BasicCookieStore cookieStore_;
private int port_; private int port_;
private String username_; private String username_;
private String password_; private String password_;
public Metrics() { this("", "", DEFAULT_WEBSERVER_PORT); } public WebClient() { this("", "", DEFAULT_WEBSERVER_PORT); }
public Metrics(int port) { this("", "", port); } public WebClient(int port) { this("", "", port); }
public Metrics(String username, String password) { public WebClient(String username, String password) {
this(username, password, DEFAULT_WEBSERVER_PORT); this(username, password, DEFAULT_WEBSERVER_PORT);
} }
public Metrics(String username, String password, int port) { public WebClient(String username, String password, int port) {
this.username_ = username; this.username_ = username;
this.password_ = password; this.password_ = password;
this.port_ = port; this.port_ = port;
httpClient_ = HttpClients.createDefault(); httpClient_ = HttpClients.createDefault();
cookieStore_ = new BasicCookieStore();
} }
public void Close() throws IOException { httpClient_.close(); } public void Close() throws IOException { httpClient_.close(); }
public List<Cookie> getCookies() { return cookieStore_.getCookies(); }
/** /**
* Returns the metric for the given metric id from the Impala web server. * Returns the metric for the given metric id from the Impala web server.
* @param metricId identifier of the metric we want to retrieve * @param metricId identifier of the metric we want to retrieve
@@ -73,20 +85,79 @@ public class Metrics {
* The caller needs to cast it to the appropriate type, e.g. Long, String, etc. * The caller needs to cast it to the appropriate type, e.g. Long, String, etc.
*/ */
public Object getMetric(String metricId) throws Exception { public Object getMetric(String metricId) throws Exception {
String content = readContent(JSON_METRICS); JSONObject json = jsonGet(JSON_METRICS);
if (content == null) return null;
JSONObject json = toJson(content);
if (json == null) return null; if (json == null) return null;
return json.get(metricId); return json.get(metricId);
} }
/**
* Does a GET request at path of the Impala web server and returns response.
* @param path URI path to query
* @return A JSON object, or null if not parseable as JSON
* @throws Exception if the request fails or JSON is not an object
*/
public JSONObject jsonGet(String path) throws Exception {
return toJson(readContent(path));
}
/**
* Does a POST request at path of the Impala web server and returns response.
* @param path URI path to query
* @param headers Headers to include in the request
* @param params Parameters to include in the POST
* @return A JSON object, or null if not parseable as JSON
* @throws Exception if the request fails or JSON is not an object
*/
public JSONObject jsonPost(String path, Header[] headers, List<NameValuePair> params)
throws Exception {
return toJson(post(path, headers, params, 200));
}
/** /**
* Retrieves the page at 'path' and returns its contents. * Retrieves the page at 'path' and returns its contents.
*/ */
public String readContent(String path) throws IOException { public String readContent(String path) throws IOException {
HttpHost targetHost = new HttpHost(WEBSERVER_HOST, port_, "http"); HttpHost target = new HttpHost(WEBSERVER_HOST, port_, "http");
HttpClientContext context = getContext(target);
HttpGet get = new HttpGet(path);
try (CloseableHttpResponse response = httpClient_.execute(target, get, context)) {
return EntityUtils.toString(response.getEntity());
}
}
/**
* Does a POST request at path of the Impala web server and returns response.
* If the response does not include the expected status code, returns null.
* @param path URI path to query
* @param headers Headers to include in the POST; can be null
* @param params Parameters to include in the POST; can be null
* @param code Response status code expected
* @return Response string
* @throws IOException if the request fails
*/
public String post(String path, Header[] headers, List<NameValuePair> params, int code)
throws IOException {
HttpHost target = new HttpHost(WEBSERVER_HOST, port_, "http");
HttpClientContext context = getContext(target);
HttpPost post = new HttpPost(path);
if (headers != null) {
post.setHeaders(headers);
}
if (params != null) {
post.setEntity(new UrlEncodedFormEntity(params));
}
try (CloseableHttpResponse response = httpClient_.execute(target, post, context)) {
if (response.getStatusLine().getStatusCode() != code) {
return null;
}
return EntityUtils.toString(response.getEntity());
}
}
private HttpClientContext getContext(HttpHost targetHost) {
HttpClientContext context = HttpClientContext.create(); HttpClientContext context = HttpClientContext.create();
if (!username_.equals("")) { if (!username_.equals("")) {
CredentialsProvider credsProvider = new BasicCredentialsProvider(); CredentialsProvider credsProvider = new BasicCredentialsProvider();
@@ -97,20 +168,11 @@ public class Metrics {
context.setCredentialsProvider(credsProvider); context.setCredentialsProvider(credsProvider);
context.setAuthCache(authCache); context.setAuthCache(authCache);
} }
context.setCookieStore(cookieStore_);
String ret = ""; return context;
HttpGet httpGet = new HttpGet(path);
CloseableHttpResponse response = httpClient_.execute(targetHost, httpGet, context);
try {
ret = EntityUtils.toString(response.getEntity());
} finally {
response.close();
}
return ret;
} }
private static JSONObject toJson(String text) throws Exception { private static JSONObject toJson(String text) throws ParseException {
JSONParser parser = new JSONParser(); JSONParser parser = new JSONParser();
Object obj = parser.parse(text); Object obj = parser.parse(text);

84
tests/webserver/test_web_pages.py
View File

@@ -183,6 +183,27 @@ class TestWebPage(ImpalaTestSuite):
assert 'Content-Security-Policy' in response.headers, "CSP header missing" assert 'Content-Security-Policy' in response.headers, "CSP header missing"
return responses return responses
def post_and_check_status(self, url, data={}, string_to_search="", ports_to_test=None):
"""Helper method that posts to a given url, then asserts the return code is ok and
the response contains the expected string."""
if ports_to_test is None:
ports_to_test = self.TEST_PORTS_WITH_SS
responses = []
for port in ports_to_test:
input_url = url.format(port)
response = requests.head(input_url)
assert response.status_code == requests.codes.ok, "URL: {0} Str:'{1}'\nResp:{2}"\
.format(input_url, string_to_search, response.text)
response = requests.post(input_url, data=data)
assert response.status_code == requests.codes.ok, "URL: {0} Str:'{1}'\nResp:{2}"\
.format(input_url, string_to_search, response.text)
assert string_to_search in response.text, "URL: {0} Str:'{1}'\nResp:{2}".format(
input_url, string_to_search, response.text)
responses.append(response)
assert 'Content-Security-Policy' in response.headers, "CSP header missing"
return responses
def get_debug_page(self, page_url, port=25000): def get_debug_page(self, page_url, port=25000):
"""Returns the content of the debug page 'page_url' as json.""" """Returns the content of the debug page 'page_url' as json."""
responses = self.get_and_check_status(page_url + "?json", ports_to_test=[port]) responses = self.get_and_check_status(page_url + "?json", ports_to_test=[port])
@@ -195,6 +216,11 @@ class TestWebPage(ImpalaTestSuite):
return self.get_and_check_status(url, string_to_search, return self.get_and_check_status(url, string_to_search,
ports_to_test=self.TEST_PORTS_WITHOUT_SS) ports_to_test=self.TEST_PORTS_WITHOUT_SS)
def post_and_check_status_jvm(self, url, data={}, string_to_search=""):
"""Calls post_and_check_status() for impalad and catalogd only"""
return self.post_and_check_status(url, data, string_to_search,
ports_to_test=self.TEST_PORTS_WITHOUT_SS)
def test_content_type(self): def test_content_type(self):
"""Checks that an appropriate content-type is set for various types of pages.""" """Checks that an appropriate content-type is set for various types of pages."""
# Mapping from each page to its MIME type. # Mapping from each page to its MIME type.
@@ -211,52 +237,46 @@ class TestWebPage(ImpalaTestSuite):
malformed inputs. This however does not test that the log level changes are actually malformed inputs. This however does not test that the log level changes are actually
in effect.""" in effect."""
# Check that the log_level end points are accessible. # Check that the log_level end points are accessible.
self.get_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL) self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL)
self.get_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL) self.post_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL)
self.get_and_check_status(self.SET_GLOG_LOGLEVEL_URL) self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL)
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL) self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL)
# Set the log level of a class to TRACE and confirm the setting is in place # Set the log level of a class to TRACE and confirm the setting is in place
set_loglevel_url = (self.SET_JAVA_LOGLEVEL_URL + "?class" + self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL,
"=org.apache.impala.catalog.HdfsTable&level=trace") {"class": "org.apache.impala.catalog.HdfsTable", "level": "trace"},
self.get_and_check_status_jvm( "org.apache.impala.catalog.HdfsTable : TRACE")
set_loglevel_url,
"org.apache.impala.catalog.HdfsTable : TRACE")
# Reset Java logging levels # Reset Java logging levels
self.get_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL, "Java log levels reset.") self.post_and_check_status_jvm(self.RESET_JAVA_LOGLEVEL_URL, {},
"Java log levels reset.")
# Set a new glog level and make sure the setting has been applied. # Set a new glog level and make sure the setting has been applied.
set_glog_url = (self.SET_GLOG_LOGLEVEL_URL + "?glog=3") self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": 3}, "val(3)")
self.get_and_check_status(set_glog_url, "val(3)")
# Try resetting the glog logging defaults again. # Try resetting the glog logging defaults again.
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, "Current backend log level: ") self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, {},
"Current backend log level: ")
# Same as above, for set log level request # Try to set the log level with an empty class input
set_loglevel_url = (self.SET_JAVA_LOGLEVEL_URL + "?class=") self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL, {"class": ""})
self.get_and_check_status_jvm(set_loglevel_url)
# Empty input for setting a glog level request # Empty input for setting a glog level request
set_glog_url = (self.SET_GLOG_LOGLEVEL_URL + "?glog=") self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": ""})
self.get_and_check_status(set_glog_url)
# Try setting a non-existent log level on a valid class. In such cases, # Try setting a non-existent log level on a valid class. In such cases,
# log4j automatically sets it as DEBUG. This is the behavior of # log4j automatically sets it as DEBUG. This is the behavior of
# Level.toLevel() method. # Level.toLevel() method.
set_loglevel_url = (self.SET_JAVA_LOGLEVEL_URL + "?class" + self.post_and_check_status_jvm(self.SET_JAVA_LOGLEVEL_URL,
"=org.apache.impala.catalog.HdfsTable&level=foo&") {"class": "org.apache.impala.catalog.HdfsTable", "level": "foo"},
self.get_and_check_status_jvm( "org.apache.impala.catalog.HdfsTable : DEBUG")
set_loglevel_url,
"org.apache.impala.catalog.HdfsTable : DEBUG")
# Try setting an invalid glog level. # Try setting an invalid glog level.
set_glog_url = self.SET_GLOG_LOGLEVEL_URL + "?glog=foo" self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": "foo"},
self.get_and_check_status(set_glog_url, "Bad glog level input") "Bad glog level input")
# Try a non-existent endpoint on log_level URL. # Try a non-existent endpoint on log_level URL.
bad_loglevel_url = self.SET_GLOG_LOGLEVEL_URL + "?badurl=foo" self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"badurl": "foo"})
self.get_and_check_status(bad_loglevel_url)
@pytest.mark.execute_serially @pytest.mark.execute_serially
def test_uda_with_log_level(self): def test_uda_with_log_level(self):
@@ -264,15 +284,15 @@ class TestWebPage(ImpalaTestSuite):
to 3. Running this test serially not to interfere with other tests setting the log to 3. Running this test serially not to interfere with other tests setting the log
level.""" level."""
# Check that the log_level end points are accessible. # Check that the log_level end points are accessible.
self.get_and_check_status(self.SET_GLOG_LOGLEVEL_URL) self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL)
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL) self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL)
# Set log level to 3. # Set log level to 3.
set_glog_url = (self.SET_GLOG_LOGLEVEL_URL + "?glog=3") self.post_and_check_status(self.SET_GLOG_LOGLEVEL_URL, {"glog": 3}, "val(3)")
self.get_and_check_status(set_glog_url, "val(3)")
# Check that Impala doesn't crash when running a query that aggregates. # Check that Impala doesn't crash when running a query that aggregates.
self.client.execute("select avg(int_col) from functional.alltypessmall") self.client.execute("select avg(int_col) from functional.alltypessmall")
# Reset log level. # Reset log level.
self.get_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, "Current backend log level: ") self.post_and_check_status(self.RESET_GLOG_LOGLEVEL_URL, {},
"Current backend log level: ")
def test_catalog(self, cluster_properties, unique_database): def test_catalog(self, cluster_properties, unique_database):
"""Tests the /catalog and /catalog_object endpoints.""" """Tests the /catalog and /catalog_object endpoints."""

3
www/form-hidden-inputs.tmpl
View File

@@ -1,3 +1,6 @@
{{# __common__.csrf_token }}
<input type='hidden' name='csrf_token' value='{{ __common__.csrf_token }}' />
{{/ __common__.csrf_token }}
{{# __common__.hostname }} {{# __common__.hostname }}
<input type='hidden' name='scheme' value='{{ __common__.scheme }}' /> <input type='hidden' name='scheme' value='{{ __common__.scheme }}' />
<input type='hidden' name='host' value='{{ __common__.hostname }}' /> <input type='hidden' name='host' value='{{ __common__.hostname }}' />

8
www/log_level.tmpl
View File

@@ -31,7 +31,7 @@ under the License.
<h5>Current frontend log level:</h5> <h5>Current frontend log level:</h5>
<span style="white-space: pre-line">{{get_java_loglevel_result}}</span> <span style="white-space: pre-line">{{get_java_loglevel_result}}</span>
<br> <br>
<form action="set_java_loglevel">{{>www/form-hidden-inputs.tmpl}} <form action="set_java_loglevel" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="form-group" name="level"> <div class="form-group" name="level">
<input type="text" class="form-control" name="class" placeholder="e.g. org.apache.impala.analysis.Analyzer"> <input type="text" class="form-control" name="class" placeholder="e.g. org.apache.impala.analysis.Analyzer">
<br> <br>
@@ -51,7 +51,7 @@ under the License.
</div> </div>
</div> </div>
</form> </form>
<form action="reset_java_loglevel">{{>www/form-hidden-inputs.tmpl}} <form action="reset_java_loglevel" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="col-xs-20"> <div class="col-xs-20">
<button type="submit" class="btn btn-primary btn-sm">Reset Frontend Log Levels</button> <button type="submit" class="btn btn-primary btn-sm">Reset Frontend Log Levels</button>
<strong>{{reset_java_loglevel_result}}</strong> <strong>{{reset_java_loglevel_result}}</strong>
@@ -62,7 +62,7 @@ under the License.
<h2>Backend log level configuration (glog)</h2> <h2>Backend log level configuration (glog)</h2>
<h5>Current backend log level: <span id="glog_text"></span></h5> <h5>Current backend log level: <span id="glog_text"></span></h5>
<form action="set_glog_level">{{>www/form-hidden-inputs.tmpl}} <form action="set_glog_level" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="form-group" name="level"> <div class="form-group" name="level">
<div class="col-xs-20"> <div class="col-xs-20">
<label>Log level:</label> <label>Log level:</label>
@@ -76,7 +76,7 @@ under the License.
</div> </div>
</div> </div>
</form> </form>
<form action="reset_glog_level">{{>www/form-hidden-inputs.tmpl}} <form action="reset_glog_level" method="post">{{>www/form-hidden-inputs.tmpl}}
<div class="col-xs-20"> <div class="col-xs-20">
<button type="submit" class="btn btn-primary btn-sm">Reset Backend Log Level</button> <button type="submit" class="btn btn-primary btn-sm">Reset Backend Log Level</button>
<strong>&nbsp &nbsp Default backend log level: {{default_glog_level}}</strong> <strong>&nbsp &nbsp Default backend log level: {{default_glog_level}}</strong>