Edge Collective

P2P / DIY Server

Repo is here.

A useful associted 'PUT' script is here

ESP32 --> Server

Adding headers to esp32 explanation here

Making a PUT request here.

Using ArduinoJSON with HTTPClient here.

Creating nested json objects in ArduinoJSON here.

Work Log


21 OCT 2020

Fixed sqlite3 missing module.

Got an ESP32 to post data via JSON, here.


23 OCT 2020

Trying to use this fastify basic auth module as part of the codebase.

... implemented!


24 OCT 2020

Some next steps:


25 OCT 2020

Making a python PUT script.

Reference on python requests here.

Setting the authorization header in curl here.

Issue: when using basic auth in current basicAuth branch, it is putting the username password authentication method on top of every request -- need to do it just for the home page.

Guide to using basic auth here.

Combining fastify static with fastify basic auth here.

Using multiple authentication options with fastify here

addhook onresponse by route here

fastify and mongodb here

Might just need to find a way to serve up web page from directly in fastify, using basicAuthFix branch

Nice getting started with fastify document.

Currently the action is in 'main.js' ...

Looks like there is a way to addHook to specific routes here

latest idea: keep basicAuth branch as-is, but use multiple authentication methods in 'PUT' script

'ctrl shift i' will insert timestamp in VS Code given installation of an extension here

Adding multiple authorization headers via postman here

Combining multiple authorizations via curl here

Setting multiple headers in curl -- useful! -- here

issues:

  1. can't use multiple authorization schemes, in particular not if one of them is Basic Auth
  2. not sure how to combine basic auth with fastify-static

interesting angle mentioned in "Transform response of fastify-static served files", which led to fastify point-of-view templates idea.

Ah -- I think I've got it working with the basicAuth branch, now!


2020-11-29 08:51:40

Adding authentication to NodeJS, here

Using Auth0 and Fastify here -- ah, blocked --

But there's a repo, here.


2020-11-29 16:43:16

Jupyter dashboard via voila -- gallery of examples

gesis notebooks, here

guide to voila, here

Dashboarding options in Jupyter discussed here

User authentication with nodejs and passport here

Rolling your own authentication, here.

React and Auth0 ...


2020-12-05 14:21:37

Node + Express + Postgresql

react + node + express + postgresql -- short tutorial

react + node + express + postgresql -- thorough tutorial

express + postgresql -- from node guide

Creating a web application in your home with a Raspberry Pi, Express and PostgreSQL -- this looks like a great place to begin!

Getting started with authentication in Node.js with Passport and PostgresQL

Sequelize -- JSON that creates a database table

suggestion to use multi-row inserts to save time

designing a restful api with node and postgres

ahhh -- part of a great series that seems to do all the things! including graphing! -- the series is here

using mqtt vs REST


2020-12-05 20:47:42

following tutorial here, putting in Documents/able

sudo apt-get install postgresql

psuser psps

sensor2=# INSERT INTO users (username, password) VALUES ('user1', 'bubbles');

INSERT INTO sensors (name) values ('NodeMCU ESP32');


2020-12-15 08:53:18

Building a simple API with Nodejs, Expressjs, PostgreSQL DB, and JWT - 3

node + postgresql -- nice tutorial

node + postgresql + jwt


2020-12-15 15:12:26

Begin here:

then add security here:

Update: actually, begin with second tutorial ...

Installing postgresql on ubuntu here

React + node + express + mongo w/ login page here


2020-12-15 15:53:43

Okay, here's the real deal:

Node.js Express & PostgreSQL: CRUD Rest APIs example with Sequelize

Prior exapmel:

node, express, postgresql using sequelize here

Wow, really simple example here

postgres and ubuntu tutorial here


2020-12-15 17:12:48

nodejs express mongodb jwt here


2020-12-16 11:26:24

as another approach w/ node + postgres, trying out this tutorial

Nice intro tutorial on nodejs and postgres here

try out the github code here

worked.

New homework! This is the thing to try:

"creating a web application in your home using nodes, rpi, express, postgresql" -- here

NOTE: 'createdb -0 username dbname' should have -O instead of -0

Associated github repo is here

Grand. And then can add users / JWT with something like this post

The first post for that series is here

Full code for the latter is here

And perhaps this shows how to combine the above with a front end -- here

Good example!

The following pairing of backend and frontend seems to work as a baseline structure. Running both, they even find one another on the proper ports.

Backend w/ node + express + postgresql + JWT

Tutorial here with associated code on github here

Frontend

React front end here, with github repo here

Node + Express + Postgres + Sequelize

Tutorial here. Nice explanation.

Another nice tutorial here, with associated code on github here

Ahhhh, really nice tutorial on using sequelize, here


2020-12-16 13:31:56

using quasar, vue, and chartjs here


Key links

https://scotch.io/tutorials/getting-started-with-node-express-and-postgres-using-sequelize#toc-sequelize-setup

https://bezkoder.com/node-js-jwt-authentication-postgresql/

https://www.codementor.io/@olawalealadeusi896/building-a-simple-api-with-nodejs-expressjs-and-postgresql-db-masuu56t7


2020-12-17 08:04:23

arg for postgresql for iot here


(HICCUP IN NOTES FLOW)


2020-12-15 17:31:58

make your own iot data platform

mern stack here

mean stack jwt here

MERN full stack here

ooooooh this is good MERN! here

another good example here

this worked!

https://github.com/vishalnagda1/mern-jwt-auth

broader tutorial on MERN here

MERN CRUD app tutorial here

this looks like a nice example, using postgres -- here


MERN:

okay so this tutorial lets you create student records and view them -- here

whereas this tutorial does user login stuff.


2020-12-15 20:59:54

this pair of server (node + mongo + jwt) + client (angular) worked:

reading about fastify here


2020-12-16 09:25:10

this might be my starting place -- node express postgresql ubuntu here

also see post here

doing this in ~/Documents/postgres-express-react-node-tutorial

notes on postgres and user auth here

used this technique to change postgres password to 'i8buddha': linked here

sequelize import is not a function' -- https://stackoverflow.com/questions/62917111/sequelize-import-is-not-a-function

associated github repo is here


2020-12-16 10:16:16

In the end, a nice codebase for postgres express and node is here. Note that this blog post is based on an earlier blog post by Michael Herman, here.

with the github repo here

used this technique to change postgres password to 'i8buddha': linked here

which needed to be changed in server/config/config.json as per:

{ "development": { "username": "postgres", "password": "i8buddha", "database": "todos-dev", "host": "127.0.0.1", "port": 5432, "dialect": "postgres" }, "test": { "username": "postgres", "password": "i8buddha", "database": "todos-test", "host": "127.0.0.1", "port": 5432, "dialect": "postgres" } }

this was the only change needed in the github repo above.

clone, then install with npm install, and run with npm run start:dev

result: working todo api w/ sequelize.


2020-12-16 11:26:24

as another approach w/ node + postgres, trying out this tutorial

Nice intro tutorial on nodejs and postgres here

try out the github code here

worked.

New homework! This is the thing to try:

"creating a web application in your home using nodes, rpi, express, postgresql" -- here

NOTE: 'createdb -0 username dbname' should have -O instead of -0

Associated github repo is here

Grand. And then can add users / JWT with something like this post

The first post for that series is here

Full code for the latter is here

And perhaps this shows how to combine the above with a front end -- here

Good example!

The following pairing of backend and frontend seems to work as a baseline structure. Running both, they even find one another on the proper ports.

Backend w/ node + express + postgresql + JWT

Tutorial here with associated code on github here

Frontend

React front end here, with github repo here

Node + Express + Postgres + Sequelize

Tutorial here. Nice explanation.

Another nice tutorial here, with associated code on github here

Ahhhh, really nice tutorial on using sequelize, here


2020-12-16 13:02:19

ingesting real time data w/ nodejs and timescale db here

rike in berlin here

getting a csv download from nodejs here

rike -- project -- maybe she'd help with corona -- here


multiple users and reflections here

this is a meaty post here

Key links

https://scotch.io/tutorials/getting-started-with-node-express-and-postgres-using-sequelize#toc-sequelize-setup

https://bezkoder.com/node-js-jwt-authentication-postgresql/

https://www.codementor.io/@olawalealadeusi896/building-a-simple-api-with-nodejs-expressjs-and-postgresql-db-masuu56t7

this adds content to the node + express + postgres + sequelize here w/ associated github repo here

follow this first!
and see if you can make it appropriate for sensors ...

and maybe if you can do a front end for it ... looks like they do have a front end in React example at the end here -- w/ associated github repo here

NOTE: have changed default port to something other than 8080, should go back to that ...

so, plan of attack:

  1. return to that .bio tutorial on sensors, and see how they set up the tables. use that as an inspiration for how to create the postgresql table structure.
  2. figure out best next thing to implement above; 'todo' or 'tutorials' examples; with or without jwt
  3. add jwt + users using the other tutorial.

postgres tutorial here

sudo -i -u postgres

psql

\q


2020-12-16 21:08:14

next steps:

latest running code experimented with was here.

going to experiment with the '.bio' code asap.

.bio tutorial

creating new tables

CREATE TABLE users ( user_id SERIAL PRIMARY KEY, username VARCHAR(50), password VARCHAR(255), is_admin BOOLEAN DEFAULT FALSE );

CREATE TABLE feeds( feed_id SERIAL PRIMARY KEY, name VARCHAR(255), private_key VARCHAR(255) );

CREATE TABLE measurements( id SERIAL PRIMARY KEY, feed_id INT, celcius FLOAT, humidity FLOAT, created TIMESTAMP DEFAULT NOW(), CONSTRAINT feed FOREIGN KEY(feed_id) REFERENCES feeds(feed_id) );

INSERT INTO users (username, password) VALUES ('bob', 'jones');

INSERT INTO feeds (name) values ('NodeMCU ESP32');

checking string status here


2020-12-17 20:03:39

newfeed branch is here

going to try for csv next using tutorial here

another tutorial on exporting csv w/ node and express here

nice bezcoder tutorial here


2020-12-17 22:09:41

'heltec' is the latest branch. accepts data from a heltec; includes firmware.

Server code is here; firmware is in 'firmware' directory.


2020-12-18 16:23:00

ah, interesting code snippet:

// https://github.com/auth0/express-jwt // Secure "protected" endpoints with JWT middleware app.use('/protected', jwtMiddleware({ secret: SECRET_TOKEN, // Use the same token that we used to sign the JWT above // Let's allow our clients to provide the token in a variety of ways getToken: function (req) { if (req.headers.authorization && req.headers.authorization.split(' ')[0] === 'Bearer') { // Authorization: Bearer g1jipjgi1ifjioj // Handle token presented as a Bearer token in the Authorization header return req.headers.authorization.split(' ')[1]; } else if (req.query && req.query.token) { // Handle token presented as URI param return req.query.token; } else if (req.cookies && req.cookies.token) { // Handle token presented as a cookie parameter return req.cookies.token; } // If we return null, we couldn't find a token. // In this case, the JWT middleware will return a 401 (unauthorized) to the client for this request return null; } }));

// A simple protected route for demo purposes app.get('/protected/data', function (req, res) { console.log(req.user); // => { _id: <S


2020-12-18 16:26:03

Secure Rest api in nodejs

using OAuth

node api key

api keys here

nice tutorial on development of the stack here

very nice tutorial on postgres + express dev here

basic passport api key here

passport and nestjs integration here

this looks like a good strategy here

nice strategy for token authentication and whatnot here

--

bezcode node + postgresql + express + react here


2020-12-19 11:35:42

inserting html form data into postgresql using nodejs here


2021-01-08 15:34:05

capturing form data with nodejs and express here

working with forms in express here

Forms example #1

even better version, with working code, here

github repo here


2021-01-08 15:57:43

This is the code to use and modify for habitat, here, when trying to get forms up and running.

Next step is to get habitat up and running -- the 'heltec' code -- which will likely require reconnecting / reconfiguring the postgres database ...


2021-01-08 19:32:35

Forms exmaple with SQLITE3

Read HTML Form Data Using GET and POST Method in Node.js

forking their example here

issue with sqlite3 -- covered here

seems that this approach worked, i.e.:

npm install -g node-pre-gyp npm install sqlite3 --save-dev


2021-01-09 16:40:05

Capturing which button was pressed in form submit here

Got basic forms working in the 'form' branch:

https://github.com/edgecollective/habitat/tree/form

Going to try templates next. Also:

understandin gnodejs express templating engines


2021-01-09 17:44:55

Got basic templating working: https://github.com/edgecollective/habitat/tree/template

TODOS:

Going to need to figure out how to create a new table, and new fields. Follow tutorial by Trickett (habitat creator) here


2021-01-10 10:57:12

setting up postgresql server ... creating tables ...

then follow instructions here

sudo -i -u postgres psql

can exit via \q

CREATE TABLE users ( user_id SERIAL PRIMARY KEY, username VARCHAR(50), password VARCHAR(255), is_admin BOOLEAN DEFAULT FALSE );

CREATE TABLE feeds( feed_id SERIAL PRIMARY KEY, name VARCHAR(255), public_key VARCHAR(255), private_key VARCHAR(255) );

CREATE TABLE measurements( id SERIAL PRIMARY KEY, feed_id INT, co2 FLOAT, tempC FLOAT, humidity FLOAT, mic FLOAT, auxPressure FLOAT, auxTempC FLOAT, aux001 FLOAT, aux002 FLOAT,
created TIMESTAMP DEFAULT NOW(), CONSTRAINT feed FOREIGN KEY(feed_id) REFERENCES feeds(feed_id) );

INSERT INTO users (username, password) VALUES ('bob', 'jones');

INSERT INTO feeds (name) values ('NodeMCU ESP32');

pcat999

how to handle: relation does not exist -- here


2021-01-10 12:09:36

update: relation didn't exist b/c hadn't created the tables in the proper database.

PORT=3000 DB_NAME=hab1 DB_USER=postgres DB_PASSWORD=pcat999 DB_HOST=localhost DB_PORT=5432


2021-01-10 12:30:59

Created a 'feed creation' landing page in the 'createfeed-landing' branch of habitat on github.com/edgecollective


2021-01-10 12:48:01

using serial primary keys postgresql here

postgresql random primary key here

unique identifier in your postgresql database here

advice here is not to use the public key as the unique identifier -- here


2021-01-11 11:06:05

error handling in express here

TODO: refactor things based on the suggestions in that post.

in particular:

app.get('/user/:id', async function (req, res, next) {
  var user = await getUserById(req.params.id)
  res.send(user)
})

2021-01-11 11:16:08

notes on api keys:

nice article here

carto's policy here

maybe should now have authorization key in the header. should just be a matter of checking on the "POST" ... getting a header. can implement it after this round.


2021-01-11 11:25:41

changing name of table in postgres

ALTER TABLE table_name 
RENAME COLUMN column_name TO new_column_name;

tests:


2021-01-11 12:23:53

writing an async function here

guide to async programming here


2021-01-11 12:32:41

'use-pubkey' is the latest branch.

using 'async await' in express here

and: definition of 'async / await' here

w/ postgresql here

started to use promises ...

even better now would be to create a 'getFeedPrivateKey() function and use that in postNewMeasurement()


2021-01-11 17:13:28

have now started some 'async' code in branch 'async'. that's the latest working branch.

TODO: the 'new feed' api should return JSON; the page we land on after creating a new feed in the form should leverage that api.


2021-01-11 17:25:43

beginner's guide to pug here

cami -- simon's kid


2021-01-12 19:02:12

instructions for setting up postgres for habitat on digitalocean

install postgres

sudo apt update
sudo apt install postgresql postgresql-contrib

sudo -i -u postgres createdb bay1 psql \c bay1

then:

CREATE TABLE users ( user_id SERIAL PRIMARY KEY, username VARCHAR(50), password VARCHAR(255), is_admin BOOLEAN DEFAULT FALSE );

CREATE TABLE feeds( feed_id SERIAL PRIMARY KEY, name VARCHAR(255), public_key VARCHAR(255), private_key VARCHAR(255) );

CREATE TABLE measurements( id SERIAL PRIMARY KEY, feed_id INT, co2 FLOAT, tempC FLOAT, humidity FLOAT, mic FLOAT, auxPressure FLOAT, auxTempC FLOAT, aux001 FLOAT, aux002 FLOAT,
created TIMESTAMP DEFAULT NOW(), CONSTRAINT feed FOREIGN KEY(feed_id) REFERENCES feeds(feed_id) );

check that all is well:

\dt

change the user password:

ALTER USER postgres WITH PASSWORD 'pcat999'

\q exit

see better guide here

need to make sure .env file is proper:

PORT=3000 DB_NAME=hab1 DB_USER=postgres DB_PASSWORD=pcat999 DB_HOST=localhost DB_PORT=5432

key guide here for deploying nodejs server on digitalocean.

has section for adding additional apps on other ports!

setting up nginx on digitalocean guide here

how to set up a firewall on ubuntu here


2021-01-13 08:05:11

Its creation was motivated by recent work (e.g. by Jimenez et al.) indicating that monitoring indoor CO2 levels can (in some contexts) help to assess the risk of airborne SARS-CoV-2 transmission.


2021-01-13 11:59:26

html to jade thing here


2021-01-13 19:20:23

Interesting way to do a drop-down menu via Pug, here:

div(class="choices") h2 p I'm voting for: select(class="form-control") option 1 option 2 option 3 option 4 option 5


2021-01-14 08:49:52

good example of chart setup here: view-source:https://www.chartjs.org/samples/latest/charts/line/basic.html

Reference for p2p-farm-server chartjs setup here

perhaps-useful chartjs formatting here


2021-01-21 11:20:38

Structure for a basic todo list app here: https://codeburst.io/creating-a-todo-app-with-node-js-express-and-postgresql-database-6dbf80c3fe9c

Associated code on github here: https://github.com/chisom5/nodejs-todoList

Idea:

Idea: add nodemon for auto restart of bayou while developing

Using React-like library 'Riot' here: https://hub.packtpub.com/simple-todo-list-web-application-nodejs-express-and-riot/, with associated code here: https://github.com/revington/riot-express-todo-list}

Drawing a grid on a canvas here https://codereview.stackexchange.com/questions/114702/drawing-a-grid-on-canvas

postgres tutorial again here: https://able.bio/rhett/creating-a-web-application-in-your-home-with-a-raspberry-pi-express-and-postgresql--3c90a372


2021-01-21 11:38:40

Simpler todo app approach here -- these are the elements I think I need -- https://medium.com/@atingenkay/creating-a-todo-app-with-node-js-express-8fa51f39b16f

And on github: https://github.com/missating/nodejs-todo

Okay -- TODO: redo that last repo, using pug. Then create feedmaps -- the equiv of bayou but by pulling in new feeds from various sources. depending on the type of feed, can have various functionality. so, add a 'feed type' variable -- e.g. bayou-co2 is one such type.


2021-01-21 12:10:58

Iteration in pug here: https://pugjs.org/language/iteration.html


2021-01-21 12:42:26

feed-map can now display a simple list of feeds here: https://github.com/edgecollective/feed-map/commit/4d3a522ba01f0bae15721fe0420100dcf76fd1a8


2021-01-21 17:22:56

Creating dynamic ides in a loop with pug here https://www.dev-tips-and-tricks.com/create-dynamic-ids-in-a-loop-with-pug-jade

Update: this commit shows the basic functionality!

https://github.com/edgecollective/feed-map/commit/765a9eec9b101e0e507075ec3b0e6aae8b9fec3b

Added nicer formatting:

Associated git commit is here: https://github.com/edgecollective/feed-map/commit/6f449c02024013035f1d42cd264505fb0066018e


2021-01-21 18:28:59

Nice reference for doing the map, via p2p-server code, here:

https://gitlab.com/dwblair/p2p-farm-server/-/blob/mappin/public/console.html


2021-01-21 18:39:13

Got image map working! at this commit -- https://github.com/edgecollective/feed-map/commit/72e90916c5bf5b21d748a5e4e54451f93a25c654


2021-01-21 18:46:53

Got markers working! commit -- https://github.com/edgecollective/feed-map/commit/2e4281fc405246c231ce817b35ee4f99f7d9b80c


2021-01-21 19:01:20

More formatting tweaks -- https://github.com/edgecollective/feed-map/commit/2b81f2f1154ddf1e6ffa11d2d5cd1b8ecb0260ce


2021-01-22 11:16:31

Created a 'checkbox' functionality, and also have charts graphed separately, in this commit and branch: https://github.com/edgecollective/feed-map/commit/e75252d43b287c398a6e82fa02e1599dfe307711


2021-01-22 14:14:58

Note: this is the nodejs-todo app I built feedmap upon, which has a nice 'delete note' functionality: https://github.com/missating/nodejs-todo

This code might be useful re: deleting items in postgres: https://codeburst.io/creating-a-todo-app-with-node-js-express-and-postgresql-database-6dbf80c3fe9c

Here's a nice guide on using knex -- https://medium.com/@tobie.tsuzuki/getting-started-with-node-js-express-and-knex-5640f595df98

Interesting post on sequelize vs knex vs objection.js here: https://stackoverflow.com/questions/56028287/how-is-node-js-knex-similar-different-to-sequelize

And an interesting link here: https://www.jakso.me/blog/objection-to-orm-hatred -- looks like objection.js might be a great idea.


2021-01-22 14:31:57

Overlay line on chart.js https://stackoverflow.com/questions/28076525/overlay-line-on-chart-js-graph


2021-01-22 14:43:27

Good example of chartjs datasets: https://tobiasahlin.com/blog/chartjs-charts-to-get-you-started/#2-line-chart


2021-01-22 18:34:46

Now plotting multiple co2 plots! at this commit: https://github.com/edgecollective/feed-map/commit/91adae48079a21fcf9fcc286ccd85a7ac2a68e91


2021-01-22 18:42:23

Hosting via google drive here: https://stackoverflow.com/questions/10311092/displaying-files-e-g-images-stored-in-google-drive-on-a-website


2021-01-22 18:56:25

Add asylum map:

git commit: https://github.com/edgecollective/feed-map/commit/fce79dc266a8ee5696a7da75c2a7177722128f80


2021-01-23 10:15:25

Revisiting my own documentation / repo for a minimal node + sqlite endpoint here https://github.com/edgecollective/nodejs-sqlite-endpoint