AutoDoc for WebWidgets Python Code

This is the documentation for the Python scripts that are used to communicate with the WebWidgets.io server. This information is automatically extracted from the code, to prevent documentation rot. Text in code format results from evaluating a code expression.

This data was generated by user ec2-user on 2024-03-19. If you are reading this online, please note that you can run this AutoDocTool.py script on your local machine as well.

There are three scripts that you can use to push or pull info to the server. They are:

  • PullData
  • DataPush
  • CodePush

The omission of a "PullCode" script is very intentional. Widget developers are expected and empowered to backup and manage their source code using whatever methods they prefer (source control like Git, DropBox, emailing a zip file to yourself, etc). In contrast, the WebWidgets server keeps backups of the DB files on your behalf, so you do not need to worry about backing them up unless you are very paranoid ("Only the paranoid surive" - Andy Grove).

Software Requirements

These scripts are written in vanilla Python 3 code with no dependencies. You should be able to download them from the GitHub repo and run them directly. Python is one of the most common programming languages in the world, and Python 3 is the modern version, so there is a good chance you already have it. To check, run this command: 

admins-mbp:script burfoot$ python3 --version
Python 3.6.5

The scripts also depend on curl, a very common utility for transferring data using HTTP. To check if you have curl, run: 

admins-mbp:script burfoot$ curl --version
curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
...

The WebWidgets framework uses SQLite (version 3) to store data. SQLite is the most widely deployed database engine in the world. To check if you have the right version, type: 

admins-mbp:script burfoot$ sqlite3 --version
3.28.0 2019-04-15 14:49:49 ....

Any 3.x version should work fine, you don't need exactly 3.28.0.

Configuring the Scripts

To configure the Python scripts properly, you will need to download a config file and put it in a special folder. For example, for the username ec2-user

  • Name of subfolder: .ssh
  • User home: /home/ec2-user/.ssh
  • /home/ec2-user/.ssh/ec2-user__widget.conf

The contents of this config file are simply a set of key/value pairs. The valid keys are:

  • accesshash
  • dbdir
  • codedir1
  • codedir2
  • codedir3
  • codedir4
  • codedir5

The access hash is a hashed version of your password that the scripts use to authenticate to the server. The DB dir is where your SQLite files will go. And the codedir1, codedir2 parameters are directories that the system will search for widget code - only one of them is necessary.

Pushing Widget Code

You can push code to the WebWidgets.io server using the CodePush script. The script takes at least one argument, the name of the widget that you wish to upload. The script works by looking up the code directories from the config file, and finding a subfolder with the same name as the widget. It then zips the contents of the subfolder, and sends the zip file to the server using curl. The server processes the upload, and emits some log data in response, which includes information about the old files being deleted. Here is a typical interaction: 

admins-mbp:script burfoot$ CodePush.py widgetname=links
Wrote upload file to path /opt/rawdata/servlet/dburfoot__links.widgetzip
Deleted old file /opt/userdata/widgetserve/dburfoot/links/widget.jsp
Deleted old file /opt/userdata/widgetserve/dburfoot/links/LinkDisplay.js
Extracted file /opt/userdata/widgetserve/dburfoot/links/widget.jsp, size is 11687
Extracted file /opt/userdata/widgetserve/dburfoot/links/LinkDisplay.js, size is 838
Prepended AuthInclude tag to Widget file widget.jsp

Notice that this upload process is extremely fast. The speed of this script means that widgets developers can make use of a very simple development process: edit a file, push it to the server, and reload to see the result. 

admins-mbp:script burfoot$ time CodePush.py widgetname=links
...
Extracted file /opt/userdata/widgetserve/dburfoot/links/LinkDisplay.js, size is 838
Prepended AuthInclude tag to Widget file widget.jsp

real    0m0.159s
user    0m0.084s
sys     0m0.028s

Pushing / Pulling DB Files

We also provide scripts for transferring DB data, called DataPush and PullData. These work very similarly to the script for pushing code. The main difference is that DB transfers can go both directions, which means there is some danger that you will overwrite your data. To avoid this situation, we strongly recommend that you avoid keeping your DB files on your local machine for an extended period of time. Instead, follow this workflow:

  • Start a Widget development session by grabbing one or more DB files
  • Make desired changes to the DB file(s) by using the sqlite3 console, or whatever DB-access program you prefer
  • Upload the modified DB file to the server, and reload the widget
  • Continue making changes, uploading, and reloading until you have the desired behavior
  • Delete the DB files from your local machine

The WebWidgets framework was very intentionally designed to allow and encourage people to use their SQLite databases in a flexible way. Many data sources cannot be accessed easily through the browser, and therefore Widget JavaScript code cannot get to the data. Instead, you might be able to download the data to your local machine. Then you can ingest it into your Widget with the following workflow:

  • Download both the Widget DB and your target data source to your local machine
  • Use a Python script (or some other form of software) to query the target data, transform it as necessary, and insert it into the Widget DB.
  • Upload the Widget DB back to the server
  • Use the target data in your Widget with the client-side JavaScript API

If you attempt to run PullData but you already have a DB file in the target location, the script will give you an error: 

admins-mbp:script burfoot$ PullData.py widgetname=links
**Error** : DB path /opt/userdata/db4widget/dburfoot/LINKS_DB.sqlite already exists, please delete or move first, or use deleteold=true

You can add the extra deleteold argument as suggested, or simply delete the file yourself.

When you upload a DB file, the server inspects the DB schema and regenerates the JavaScript code that is used to interface with the Widget framework. To demonstrate this, I've added a dummy table to my links database: 

sqlite> .schema
CREATE TABLE link_main (id int, cat_id int, short_desc varchar(150), link_url varchar(250), primary key(id));
CREATE TABLE link_categ (id int, short_code varchar(20), full_desc varchar(400), is_active smallint, primary key(id));
sqlite> create table dummy_link (id int, my_link varchar(30), primary key(id));

Now I upload it to the server, and it creates a new JS file for me. The other tables are not changed, so the JS code is not updated. 

admins-mbp:script burfoot$ DataPush.py widgetname=links
Defaulting to admin user name
Wrote upload file to path /opt/rawdata/servlet/dburfoot__links.sqlite
Saving DB file for widget dburfoot::links
Deleted old DB file /opt/userdata/db4widget/dburfoot/LINKS_DB.sqlite
Copied upload file to location /opt/userdata/db4widget/dburfoot/LINKS_DB.sqlite, size is 26624
Wrote autogen code to path /opt/userdata/widgetserve/dburfoot/autogenjs/links/DummyLink__001.js
New version of AutoGen code identical to previous for DB link_categ
New version of AutoGen code identical to previous for DB link_main