I am working on a project which requires data from a website. The problem with that data is that it is organised into a lot of different webpages with serial numbering in the URL. To tackle this problem, I wrote a simple script in Tampermonkey which loops through those pages and obtains the data in a string variable in the script.
Now comes the real problem, how do I store this data. I know I can't write to a file on my PC but can the data be displayed onto a separate tab in my browser so that I can copy and paste it into a local file when the loop is done? I wish to append a string into storage each loop
I do not want to use GM_setValue because I want the data in raw text format (like a .txt file)
However, if it can be written directly into a file on my PC without the use of an external library, that'd be preferred.
I am working on a project which requires data from a website. The problem with that data is that it is organised into a lot of different webpages with serial numbering in the URL. To tackle this problem, I wrote a simple script in Tampermonkey which loops through those pages and obtains the data in a string variable in the script.
Now comes the real problem, how do I store this data. I know I can't write to a file on my PC but can the data be displayed onto a separate tab in my browser so that I can copy and paste it into a local file when the loop is done? I wish to append a string into storage each loop
I do not want to use GM_setValue because I want the data in raw text format (like a .txt file)
However, if it can be written directly into a file on my PC without the use of an external library, that'd be preferred.
Share Improve this question edited Dec 5, 2015 at 6:18 AvZ asked Dec 5, 2015 at 6:04 AvZAvZ 1,0354 gold badges16 silver badges25 bronze badges 02 Answers
Reset to default 18I know I can't write to a file on my PC
Here's some good news for you: Yes you can!
var a = document.createElement("a");
a.href = "data:text,Hello World!"; //content
a.download = "Hello.txt"; //file name
a.click();
http://jsfiddle.net/DerekL/jgfhwfL0/
First open up your localhost page, master.html (http://localhost:8080/master.html):
<html>
<head>
<script>
window.addEventListener("load", function(){
//This is just like working with threads but in JavaScript
if(location.search.length){
//receiver
var query = decodeURIComponent(location.search.slice(1));
var data = JSON.parse(query).value;
//handle data
localStorage.storage = +localStorage.storage + data;
location.href = "about:blank"; //exit
}else{
//master
sum = document.getElementById("sum");
sum.innerText = localStorage.storage = "0";
window.addEventListener("storage", function(){
//data received from receiver
sum.innerText = localStorage.storage;
});
}
});
</script>
</head>
<body>
Sum: <span id="sum"></span>
</body>
</html>
Then you can start sending data to it in any webpages:
var frame = document.createElement("iframe");
frame.src = 'http://localhost:8080/master.html?{"value":90}'; //port 8080
document.body.appendChild(frame);
The sum counter should update automatically upon receiving data.
One approach is running a localhost HTTP server, exposing routes and using them to read and write files and generally do all of the back-end things you need for your userscript(s).
Here's a simple proof of concept echo using Node 17 and Express, but the same can be done with any back-end technology that you prefer.
server.js:
const cors = require("cors"); // "^2.8.5"
const express = require("express"); // "^4.18.1"
const fs = require("fs").promises;
const path = require("path");
const dataFile = "data.json";
const app = express();
app
.set("port", process.env.PORT || 5000)
.use(cors())
.use(express.json())
.get("/", (req, res) => {
fs.readFile(path.join(__dirname, dataFile))
.then(file => res.send(file.toString()))
.catch(err => res.status(404).send(err.message));
})
.post("/", (req, res) => {
const p = path.join(__dirname, dataFile);
fs.writeFile(p, JSON.stringify(req.body))
.then(file => res.sendStatus(200))
.catch(err => res.status(400).send(err.message));
})
.listen(app.get("port"), () =>
console.log(`server listening on port ${app.get("port")}`)
);
Tampermonkey script:
const url = "http://localhost:5000";
const writeData = payload =>
fetch(url, {
method: "POST",
headers: {
"Accept": "application/json",
"Content-Type": "application/json",
},
body: JSON.stringify(payload)
})
.then(response => {
if (!response.ok) {
throw Error(response.status);
}
});
const readData = () =>
fetch(url)
.then(response => {
if (!response.ok) {
throw Error(response.status);
}
return response.json();
});
writeData({hello: "world"})
.then(readData)
.then(data => console.log(data)) // => {hello: 'world'}
.catch(err => console.error(err));
There's nothing special here other than enabling cross-origin access and the novelty of sending requests from a userscript rather than a typical client such as a webpage or app.
Use caution since this exposes the local file system to potentially malicious scripts running on an unknown website.