Proven Tips and Tricks for UI Developer

 In this blog, I will walk you through the guidelines / necessary points to consider while developing an UI application.

Checkout complete blog at https://github.com/shivajimutkule/ui-developer-tips-and-tricks

 

Migration of Standalone Apache Spark Applications to Azure Databricks

In this blog, we will explore how to migrate standalone Apache Spark applications to Azure Databricks. Azure Databricks provides Spark based envoronment is minutes with autoscale and interactive workspace.

Complete blog is posted at https://www.gslab.com/blogs/migration-of-spark-to-databricks

Migration of Standalone Spark on Kubernetes to gain maximum efficiency from containers

In this blog, we will explore how to migrate standalone Apache Spark applications to Spark on Kubernetes. Kubernetes helps in optimizing the resource usage and costs by autoscaling the resources.

Complete blog is posted at https://www.gslab.com/blogs/migration-of-spark-to-kubernetes

Asynchronous Programming with Node JS

Node.js is server side JavaScript built on top of V8 JavaScript engine. Node uses non-blocking, single threaded, event driven I/O model. Ryan Dahl (creator of Node.js) chose JavaScript because of the following reasons:

1. It has no concept of threads

2. Model of concurrency is completely based around events

We have used node.js as a backend for live auctioning application. ‘Thousands of concurrent bid update requests may come to server’ keeping this in mind we have selected Node. Node best suits for the real time data applications. Next section highlights characteristic features of the Node.

Why node.js?

Node allows us to achieve high throughput and scalable platform for network applications. Node provide following benefits

• Non-blocking I/O
• Based on chrome's V8 engine (FAST!)
• 15,000+ third party modules
• Active community (IRC, mailing lists, twitter, github)
• Open source
• built-in HTTP server library
• One language for frontend and backend
• JavaScript is the language of the web 

Some of the challenges we faced while working on Node

1. Asynchronous Programming

We start learning programming with C language which follows synchronous programming culture. But node behaves very differently. It makes asynchronous calls whenever there is file, database or network operation. Let us see effect of asynchronous on following code snippet.

Problem statement:

Purpose of this code is to find email address associated with each userid. We use userid to get email address from LDAP server and print it on console along with respective user names. getEmail(userid) is a function which takes userid and returns email address from LDAP server.

Usual Approach:
user=[ { userid: 'gs-xyz', uname: 'XYZ' },{ userid: 'gs-pqr',uname: 'PQR'} ]; //Array of userid along with username

for(var i=0; i<user.length; i++)  {
   console.log('In for loop: [i=' + i + ']');
   getEmail(user[i].userid, function(email){
     console.log('In callback function: [i=' + i + ' Email=' + email + ']');
   });
}

Output:
In for loop: [i=0]
In for loop: [i=1]

In callback function: [i=2 Email=xyz@gmail.com]
In callback function: [i=2 Email=pqr@ gmail.com]

Explanation:

getEmail() is called asynchronously by node. Node registers a callback function in event loop and iterate over the for loop. When LDAP server returns email address, node starts executing callback method. Output shows correct order of email addresses but loop control variable i is set to length of array (because execution of for loop is completed and loop gets terminated when i>=user.length). So, we can't access usernames from user array.

There are two solutions to overcome above problem.

Solution 1:
user=[ { userid: 'xyz', uname: 'XYZ' },{ userid: 'gs-pqr',uname: 'PQR'}];

for(var i=0; i<user.length; i++) {
   console.log('In for loop: [i=' + i + ']');
   (function(i){
      getEmail(user[i].userid, function(email){
          console.log('In callback function: [i=' + i + ' Email=' + email + ']');
      });
   })(i);
}

Explanation:

An anonymous function is called with loop control variable, so respective username can be used with email address. This solves problem by keeping local copy of i with each asynchronous call.

Solution 2:

You may write separate function and call it from for loop. Keep in mind that all variables used inside the callback are within scope.

user=[ { userid: 'gs-xyz', uname: 'XYZ' },{ userid: 'gs-pqr',uname: 'PQR'} ];

function fromForLoop(i)
{
   getEmail(user[i].userid, function(email){
   console.log('In callback function: [i=' + i + ' Email=' + email + ']');
});
}

for(var i=0; i<user.length; i++) {
   console.log('In for loop: [i=' + i + ']');
   fromForLoop(i);
}

Output:
In for loop: [i=0]
In for loop: [i=1]

In callback function: [i=0 Email=xyz@ gmail.com]
In callback function: [i=1 Email=pqr@ gmail.com]

Now, loop control variable can be used to access respective username by replacing the log statement in callback as

console.log('In callback function: [Username='+user[i].uname+' Email='+email.email+']');

Output:
In for loop: [i=0]
In for loop: [i=1]

In callback function: [Username=XYZ Email=xyz@ gmail.com]
In callback function: [Username=PQR Email=pqr@ gmail.com]

Solution 3:
This is the practical solution of above problem. In this approach JavaScript promises are used.

var Q = require('q'); // Promise library

//we have usernames and userids in userlist object. we need to find out email id with respect to each userid
userlist =[ { userid: 'xyz', username: 'XYZ' },{ userid: 'gs-pqr', username: 'PQR'}];

//getEmail internally makes network call and return promise
//I have simulated network call with following function
var getEmail = function(userid){
var defered = Q.defer();

setTimeout(function(){
defered.resolve({email: userid + "@gmail.com", userid: userid});
},1000);

return defered.promise;
};

/*this function takes userlist as input,
 calls getEmail() function asynchronously for each userid and returns userlist with email */

var getEmailList = function(userlist){
var promises = [];
var promise;
for(var i=0; i<userlist.length; i++) {
  promise = getEmail(userlist[i].userid);
  promises.push(promise);
}

return Q.all(promises).then(function(list){
for(var i=0; i<userlist.length; i++) {
userlist[i].email = list[i].email;
}
return userlist;
});
}

getEmailList(userlist).then(function(list){
console.log(list);  //final output
});

Output:

[ { userid: 'xyz', username: 'XYZ', email: 'xyz@gmail.com' },

  { userid: 'gs-pqr', username: 'PQR', email: 'gs-pqr@gmail.com' } ]

2. Callback hell

Callback could be another challenge. As node is asynchronous, operations that are need to be executed after asynchronous call are written in callback method. If callback chain becomes longer it is called Callback hell. It creates hard to understand and debug code.

Configuring REST Adapter in Ember.js

As the Ember.js website states, it is a framework for creating ‘ambitious’ web applications. What it does not state is that the REST APIs called from the Ember application should not be ‘over-ambitious’ in nature. If the APIs follow REST guidelines and standards then Ember will love it and embrace it but if those APIs don’t follow these guidelines then Ember will make you suffer.

This blog is to help people to avoid the latter and hence an earnest effort in explaining why sometimes we have to use and configure ember’s DS.RESTAdapter.

For the sake of a practical example, let’s consider a student model with attributes like first name, last name and age:

App.Student = DS.Model.extend({

  firstName: DS.attr('string'),

  lastName: DS.attr('string'),

  age: DS.attr('number')

});

Following table shows REST standard APIs and corresponding non-standard backend APIs (because most of the times, the backend APIs are not in our control):

Action	HTTP Verb	Standard REST APIs	Non-Standard REST APIs
Find	GET	/student/:id	/get-student/:id
Find All	GET	/student	/all-students
Update	PUT	/student/:id	/put-student/:id
Create	POST	/student	/set-student
Delete	DELETE	/student/:id	/delete-student/:id

The above (Non-standard) backend APIs would fail with ember. The following steps show how customizations can be done:

1.     Host customization

Host customization can be done by setting ‘host’ property on the adapter.

App.StudentAdapter = DS.RESTAdapter.extend({

  host: 'http://example.com'

}); 

Requests for App.Student will point to url http://example.com/

2.     Path customization

Path customization can be done by setting ‘namespace’ property on the adapter.

App.StudentAdapter = DS.RESTAdapter.extend({

  namespace: 'v1/api'

});

3.     Header customization

Header customization can be done by setting ‘headers’ object on the adapter.

App.StudentAdapter = DS.RESTAdapter.extend({

  headers: {

    "API_KEY": "secret key",

    "ANOTHER_HEADER": "Some header value"

  }

}); 

API_KEY and ANOTHER_HEADER headers will send with every request to the server. This may be useful for enabling http basic authentication.

4.     HTTP method customization

Http PUT method is used to update record. Suppose backend implements same functionality with POST method. This would be done by implementing ‘upadateRecord’ method of REST Adapter. 

App.StudentAdapter = DS.RESTAdapter.extend({

        updateRecord: function(store, type, record) {

            var newdata={};

            var serializer = store.serializerFor(type.typeKey);

            serializer.serializeIntoHash(newdata, type, record);

 return this.ajax(this.buildURL(type.typeKey, null, record), "POST", { data: newdata });

        }

});

Now, let’s consider other use cases:

Say, if backend provides update functionality with ‘GET’ method, then we would have to use “GET” instead of “POST” in above code. Same logic applies for other functionality also.

5.     URL customization

Suppose backend provides different URL words for each action as shown in table 1, i.e. instead of /student it uses /get-student then the adapter for student should be written as:

App.StudentAdapter = DS.RESTAdapter.extend({

           //method will be invoked for getting single record

           find: function(store, type, id, record) {

       return this.ajax(“get-student/” + id, 'GET');

}

});

… and so on for the other methods.

Hope the above information helps you in your endeavor to use the power of Ember and at the same time keep your application flexible enough to accept any type of API URLs.