So back in few days I was applying for Masters in China and there a requirement in there universities admission to get acceptance letter from universities professor for admission, china is big country and they got a lot of universities so I went to some of their top university website and searched for professors, and the problem was that there university website got only list of professor not specified any field so I have to go through every professor profile and find my interested field professor, Well there was too much repetition then I thought why do we do programming to overcome repetition, then I came up with the idea of using crawler and sending auto emails to all professors.

alt text

Getting data from websites

So I got data of professor list from Shanghai Jiao Tong University website, here the code for getting data

var c = new Crawler({
  maxConnections: 10,
  // This will be called for each crawled page
  callback: function (error, res, done) {
    if (error) {
      console.log(error);
    } else {
      if (res.statusCode == 200) {
        console.log(i++);
        var $ = res.$;
        // $ is Cheerio by default
        //a lean implementation of core jQuery designed specifically for the server
        let disc = $(".teacher-description").text().toLowerCase();
        if (disc.includes("data mining") || disc.includes("natural language") || disc.includes("machine learning") {
          let obj1 = {
            name: $(".teacher-show-name").text(),
            field: $(".teacher-description").text(),
            email: $(".teacher-show-email").text()
          };
          // fs.appendFileSync('file.txt' , data);
          //   updateSpreadsheet(obj1);
          arrObj.push([obj1.name.trim(), obj1.email.trim(), obj1.field.trim()]);
        }
      }
    }
    done(console.log("done"));
  },
});

Now I got all my desired data in arrObj

Saving data into google spreadsheet

Now I saved data to google spreadsheet so I can send auto emails to professors, here the code to save data to google spreadsheet:

sheets.spreadsheets.values.append({
spreadsheetId: "1RtPpJRRK6y5Eeux_xrs3kXglg2IaE04RWV_crM",
range: "Sheet2!A2",
valueInputOption: "RAW",
insertDataOption: "INSERT_ROWS",
resource: {
// values: [[data.name, data.email, data.field]]
values: data
},
auth: oAuth2Client
},
(err, response) => {
if (err) return console.error(err);
else console.log("response", response.data);
}

here’s google spreadsheet link to follow further there documentations https://developers.google.com/sheets/api/guides/values

Sending Auto Emails to Professor

Now I got data into spreadsheet and have to send email to all professor for that google got an option called google spreadsheet script so you have to add a script for that. Here’s code of script :

function sendEmails() {
  var sheet = SpreadsheetApp.getActiveSheet();
  var startRow = 2; // First row of data to process
  var numRows = 15; // Number of rows to process
  // Fetch the range of cells A2:B3
  var file1 = DriveApp.getFileById('*****');
  //  var file2 = DriveApp.getFilesByName('developer1.pdf');
  var file = DriveApp.getFileById('***');
  //  var file = file2.next();
  var dataRange = sheet.getRange(startRow, 1, numRows, 3);
  // Fetch values for each row in the Range.
  var data = dataRange.getValues();
  for (i in data) {
    var row = data[i];
    var name = row[0]; // First column
    var emailAddress = row[1]; // Second column
    var field = row[2]; // Third column
    var message = "<p dir=\"ltr\"><strong>Respected Prof. Dr. " + name + "</strong></p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\">I am John Ellan from [Country Name].&nbsp; I have done my Bachelor&rsquo;s degree in the field of Computer Science from YOur university with CGPA: <strong>3.83/4.0</strong>.&nbsp; I have One and a Half years experience of Software Development in a well reputed software houses company name .</p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\">Having gone through your profile, I am quite inspired your research work on the &ldquo;<strong>" + field + "</strong>&rdquo; which is the similar to my interests.</p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\">I am solely interested in doing <strong>Master</strong> under your kind supervision, all the expense of my research will be sponsored by <strong>Chinese government Scholarship</strong> or any other available scholarship at your University if I am able to get the acceptance from you. I have attached my Curriculum vitae (CV), Transcript, and I would be happy to send you any additional materials on demand.</p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\">Early thanks for your consideration and I would wait for your kind response.</p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\"><span style=\"font-size: 15pt;\"><strong>Best Regards,</strong></span></p>" +
      "<p dir=\"ltr\">&nbsp;</p>" +
      "<p dir=\"ltr\"><span style=\"font-size: 13pt;\"><strong>name,</strong></span></p>" +
      "<p dir=\"ltr\"><span style=\"font-size: 12pt;\">uni name,</span></p>" +
      "<p dir=\"ltr\"><span style=\"font-size: 12pt;\">address.</span></p>" +
      "<p dir=\"ltr\"><span style=\"font-size: 12pt;\">Email: <a href=\"your email\" target=\"_blank\">email</a></span></p>";
    var subject = 'Request for Acceptance letter for Master Program';
    MailApp.sendEmail({
      to: emailAddress, subject: subject,
      htmlBody: message,
      attachments: [file.getAs(MimeType.PDF), file1.getAs(MimeType.JPEG)]
    });
  }
}

So this how I did a lot work through just programming in node.js and javascript. Here’s github link of project : https://github.com/SalahudinMalik/nodejs_crawler Thanks for reading, hope it helps you solve your repetition problem.