After going through the nginx-sticky-module and understanding the requirements for session based load balancing, we were convinced that the cookie based approach implemented earlier had several drawbacks, including -
1. The most obvious one, as stated on the nginx-sticky-module page itself, is that it requires cookies to be enabled by the browser. If that fails, the load balancing either falls back to the regular round robin mechanism or returns a bad gateway error depending on its configuration.
2. This cookie based approach isn't completely transparent. To ensure persistence, both the client and server are involved, they are both aware of the cookie being stored on the client side. If there is a server side processing simply based on the presence of a cookie in the incoming request, the logic fails as the load balancer intervenes to add another cookie. So yea, not completely transparent!
3. The third and more significant drawback, is the session timeout. At the application level, the session may last for a few minutes or more, there might also be an upper limit on the duration of the session. On the other hand, the cookie being stored for persistence has an associated lifetime, which will definitely be different from the actual user sessions, usually set to the length of the longest session by the developer! As mentioned in our previous blog, this will cause the load balancing to get skewed, causing the requests to go the the same server even after the user session has expired.
4. Another drawback of this module, is that the client side gets too much control. This is the consequence of the transparency problem mentioned earlier. Since the client gets the digest of the server ip in the cookie, the client can track all the sever side mappings, by simply sending a cookie free request each time - the nginx load balancer would allot a different server each time using the default round robin approach. The concept of a reverse proxy to abstract the backend servers is somewhat lost. The client can then use the information to flood one particular server with requests, succeeding in a DOS attack!
5. Https, by using TLS, encrypts all the data, including the headers. Only the hostname isn't encrypted.
For instance : the first part of the URL (https://www.google.com) is still visible as it builds the connection. The second part (/herearemygetparameters/1/2/3/4) is protected by TLS. So if the application uses https and if the termination proxy is located behind the load balancer, this approach fails as the cookies become inaccessible.
After identifying the above shortcomings, we wanted to be able to counter as many of them in our approach, if not all, using query parameters.
Our approach involves the use of 2 modules -
1. The first module is a load balancer, similar to the nginx-sticky-module using cookies. The main idea is to fetch the query parameters from the incoming url request. Extract the session id, which is the unique identifier for a client, from the query parameters. This is then matched against our own Hash Map which stores the mappings of the session id (key) and the server id(value). If a match is found, the client's request is redirected to the appropriate server. If there are no query parameters, or if there is no session id in the query parameters, or if the session id does not match with any in the hash map, a server is selected in round robin fashion.
For a new request (or otherwise) , if a new session id is assigned by the server, it has to be mapped to the server id. This can be done only after the response message has been built, clearly indicating the need of a second module to process the response sent by the server.
2. The second module, a filter, parses the response body to look for a new session id. This parsing can either involve :
a. Parsing only for form action urls and examining the query parameters contained in them (to add them to the HashMap). This approach is restrictive and only addresses form based requests.
b. Parsing for all urls and only storing the session id information from those which belong to the same domain.
As the number of user sessions increase, there is also a need to clean up some of the old mappings to optimize space. This wasn't a necessity in case of the cookie based approach as the the number of mappings was equal to the number of the backend servers in the system. With our approach, the number of mappings keep increasing with new user sessions, hence the filter module also has to clear some of the mappings after a configured timeout. For the same reason, we decided to associate a new mapping with a time attribute as well.
This alternative approach for sticky sessions counters some of the drawbacks of the cookie based approach.
- It doesn't rely on the client's browser to enable cookies.
- It is transparent on the client's side, we use a cookies only to share information across modules, before and after the server's response.
- The application is in control of the timeout, the requests are sent to the same server so long as the session id is the same in the incoming request and within the timeout. The onus of the session id is on the application, if the server session has expired, then the server will no longer generate a response with the same session id.
- The client can still access the session id which is used as a unique identifier and can flood by sending many requests with the same session id. But we cannot control the server to which the requests are sent like in the case of the sticky module.
- Lastly, since this method relies on parsing the query parameters as well as the response body, https is still a limitation if the termination proxy lies beyond the load balancer, as only the hostname is not encrypted.
After going through the nginx-sticky-module and understanding the requirements for session based load balancing, we were convinced that the cookie based approach implemented earlier had several drawbacks, including -
1. The most obvious one, as stated on the nginx-sticky-module page itself, is that it requires cookies to be enabled by the browser. If that fails, the load balancing either falls back to the regular round robin mechanism or returns a bad gateway error depending on its configuration.
2. This cookie based approach isn't completely transparent. To ensure persistence, both the client and server are involved, they are both aware of the cookie being stored on the client side. If there is a server side processing simply based on the presence of a cookie in the incoming request, the logic fails as the load balancer intervenes to add another cookie. So yea, not completely transparent!
3. The third and more significant drawback, is the session timeout. At the application level, the session may last for a few minutes or more, there might also be an upper limit on the duration of the session. On the other hand, the cookie being stored for persistence has an associated lifetime, which will definitely be different from the actual user sessions, usually set to the length of the longest session by the developer! As mentioned in our previous blog, this will cause the load balancing to get skewed, causing the requests to go the the same server even after the user session has expired.
4. Another drawback of this module, is that the client side gets too much control. This is the consequence of the transparency problem mentioned earlier. Since the client gets the digest of the server ip in the cookie, the client can track all the sever side mappings, by simply sending a cookie free request each time - the nginx load balancer would allot a different server each time using the default round robin approach. The concept of a reverse proxy to abstract the backend servers is somewhat lost. The client can then use the information to flood one particular server with requests, succeeding in a DOS attack!
5. Https, by using TLS, encrypts all the data, including the headers. Only the hostname isn't encrypted.
For instance : the first part of the URL (https://www.google.com) is still visible as it builds the connection. The second part (/herearemygetparameters/1/2/3/4) is protected by TLS. So if the application uses https and if the termination proxy is located behind the load balancer, this approach fails as the cookies become inaccessible.
After identifying the above shortcomings, we wanted to be able to counter as many of them in our approach, if not all, using query parameters.
Our approach involves the use of 2 modules -
1. The first module is a load balancer, similar to the nginx-sticky-module using cookies. The main idea is to fetch the query parameters from the incoming url request. Extract the session id, which is the unique identifier for a client, from the query parameters. This is then matched against our own Hash Map which stores the mappings of the session id (key) and the server id(value). If a match is found, the client's request is redirected to the appropriate server. If there are no query parameters, or if there is no session id in the query parameters, or if the session id does not match with any in the hash map, a server is selected in round robin fashion.
For a new request (or otherwise) , if a new session id is assigned by the server, it has to be mapped to the server id. This can be done only after the response message has been built, clearly indicating the need of a second module to process the response sent by the server.
2. The second module, a filter, parses the response body to look for a new session id. This parsing can either involve :
a. Parsing only for form action urls and examining the query parameters contained in them (to add them to the HashMap). This approach is restrictive and only addresses form based requests.
b. Parsing for all urls and only storing the session id information from those which belong to the same domain.
As the number of user sessions increase, there is also a need to clean up some of the old mappings to optimize space. This wasn't a necessity in case of the cookie based approach as the the number of mappings was equal to the number of the backend servers in the system. With our approach, the number of mappings keep increasing with new user sessions, hence the filter module also has to clear some of the mappings after a configured timeout. For the same reason, we decided to associate a new mapping with a time attribute as well.
This alternative approach for sticky sessions counters some of the drawbacks of the cookie based approach.
- It doesn't rely on the client's browser to enable cookies.
- It is transparent on the client's side, we use a cookies only to share information across modules, before and after the server's response.
- The application is in control of the timeout, the requests are sent to the same server so long as the session id is the same in the incoming request and within the timeout. The onus of the session id is on the application, if the server session has expired, then the server will no longer generate a response with the same session id.
- The client can still access the session id which is used as a unique identifier and can flood by sending many requests with the same session id. But we cannot control the server to which the requests are sent like in the case of the sticky module.
- Lastly, since this method relies on parsing the query parameters as well as the response body, https is still a limitation if the termination proxy lies beyond the load balancer, as only the hostname is not encrypted.
Few inputs
ReplyDeletePt 2: Why should server fail just because LB has introduced some additional cookies. This will happen if LB is removing server cooking but not otherwise. LB can remove its cookies and then pass it on to server.
Pt 3: The real challenge for LB is to figure out when session has expired. can LB really identify the cookies used by server and when these have been expired.
Pt 4: This needs to be relooked into. Client (Browser) does not interpret/understand the cookie value. LB can set the the cookie value of server side IP mapping in some encrypted form which will prevent from client to interpret. This problem can exist if client can interpret how cookies are being defined by LB and their structure and semantics. So, this can be easily avoided.
Pt 5: Please recheck. Even the hostname header in HTTPS communication will be encrypted. There is no way a man in the middle can interjet it. LB can at best derive some canonical hostname from IP address by using inverse DNS.
you may need to relook load balancing from a new TCP connection point of view. If LB is using TCP Connetion between client and server, then LB has nothing to do with it. Please be clear on what is the assumption. Is it that TCP connection is terminating on LB and establishing new connection with the server? In case, yes, then LB has a bigger role to play. In case, no, LB really has not much work to do for session based LB. It is quite possible to have a new TCP Connection but same session to continue from application perspective.
session id approach: The main challenge is parsing of the response content. This will become more cumbersome when these are embedded as part of some javascript. If the content are encrypted, or encoded, that will make it further complicated.
Point 2) That was just an example showing no transparency. Yes sir, in case the LB removes its cookies and sends, then there is transparency on the server side. But the sticky module does not do that. So it is a problem with sticky module and not cookie based approach as a whole.
DeletePoint 3) I guess its possible. Have an option to configure the name of the cookie used by the backend server. When the response comes, check the life time of that cookie and we can know when it will expire. But this has to be done by a filter module and not a load balancer. This approach seems a little similar to our url session id approach. Here we are parsing the output for session id in url and there, we would be looking at expiry time of cookies and adding mappings.
This could be another possible approach. But it will still have the cookie disabled problem.
Point 4) Yes sir, we wont know the structure or the semantics. But since the client can edit the cookie value, during a DDOS attack, all the computers can set the same cookie value and flood the same server bypassing the load balancing. This problem would probably be there for any method that uses data sent from the client to perform load balancing.
Point 5) We worded it very wrongly. What we meant was, man in the middle could find out the hostname because it is required for setting up the connection. Once the connection has been made, everything is encrypted and man in the middle cannot find out anything. Both the cookie based and session based approach will not work in case of HTTPS if the TCP connection does not end at LB.
Yes sir, we are assuming that the TCP connection ends at LB and LB makes a new connection to the backend server.
For now, we will be assuming that the links are plain text and in in the action attribute of form.