I'd like to output some values from this site. With the browser inspector I located the URL to the JSON that has that information that I want. Once with the JSON, I can extract the values just fine.

The issue is that the URL to the JSON only works for a limited amount of time. If I try to access it later (via browser or via script's UrlFetch), I get a:

{
"status": 401,
"response": "unauthorized"
}

Since I would like to check on the target JSON periodically, I would like to find a way to authorize the request. Any ideas?

Thanks!

PS: This is what I've tried so far (google apps script):

function getUserAndJSON(){
  var url = ';conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc';

  var opt = {
    "method": "GET", 
    "muteHttpExceptions": true,
    "escaping":false,
    "headers":JSON.parse(user)
  }
  var str = UrlFetchApp.fetch(url,opt).getContentText();
  Logger.log(str); //error 401
}

I'd like to output some values from this site. With the browser inspector I located the URL to the JSON that has that information that I want. Once with the JSON, I can extract the values just fine.

The issue is that the URL to the JSON only works for a limited amount of time. If I try to access it later (via browser or via script's UrlFetch), I get a:

{
"status": 401,
"response": "unauthorized"
}

Since I would like to check on the target JSON periodically, I would like to find a way to authorize the request. Any ideas?

Thanks!

PS: This is what I've tried so far (google apps script):

function getUserAndJSON(){
  var url = 'https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc';

  var opt = {
    "method": "GET", 
    "muteHttpExceptions": true,
    "escaping":false,
    "headers":JSON.parse(user)
  }
  var str = UrlFetchApp.fetch(url,opt).getContentText();
  Logger.log(str); //error 401
}

• javascript
• json
• web-scraping

• can u tell where is this url located on the site, I need to see it working and then can inspect the request header it's sending. As u can see that ur url is having endpoint and body but to see headers you need to see the request header. U can do it in browser also by opeing ur network window and the see where this api is making the request. Let me know if this works or any other help is needed. – 1UC1F3R616 Commented Apr 5, 2020 at 10:17
• @1UC1F3R616 The url is the first link on the question under 'this site'. I have opened up the network tab under the dev tools, but I fail to find the file where the request header is being sent. Help would be appreciated. Cheers – jlo Commented Apr 6, 2020 at 19:13
• problem is I don't see where is this question, the whole site is in some other language. Open under network tab, then click on any api, now on side there will appear options like Response, Request Header and so on. You click them and then inspect it. – 1UC1F3R616 Commented Apr 9, 2020 at 8:38
• Please provide the entire 401 status response including headers. This will give you a hint regarding the type of authentication and then you can work with it to generate a token every time you get a 401 message. An example of a fairly use authentication mechanism: en.wikipedia/wiki/Basic_access_authentication – ofirule Commented Apr 9, 2020 at 21:06
• 1 @jlo click on them and then a window will pop up on side, I am uisng Firefox. Use Postman to test apis, in python there is a request module u can send headers and cookies in it. – 1UC1F3R616 Commented Apr 11, 2020 at 13:49

1 Answer 1

One solution to this problem is to simply use the option 'copy as... > fetch' within the browser inspector (in my case Opera), which gives out the following (somewhat different headers than the ones sent by the browser):

fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc", {"credentials":"omit","headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});

Which I used in GAS in the following way:

function testGetJSON(){
 var str = UrlFetchApp.fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc",
 {"credentials":"omit",
"headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});
  Logger.log(str.getContentText());
}

Which works perfectly.

Thank you all for your tips and suggestions!