Saturday, May 27, 2023

tracing Python in JSON-based workflow is untenable

I added

def mytrace(frame, event, arg):
    if event == "call":
        print("call", frame.f_code.co_name, frame.f_locals)
    elif event == "return":
        print("return", frame.f_code.co_name, arg)
    return mytrace

sys.settrace(mytrace)

to https://github.com/allofphysicsgraph/proofofconcept/blob/gh-pages/v7_pickle_web_interface/flask/controller.py but the output wasn't that useful since I'm passing the entire database as JSON. The full JSON shows up in almost every argument and return value, making the output of the trace unreadable.

When I switch to the Neo4j/Cypher-based approach, the trace might be more usable.

Monday, May 1, 2023

Omniscient project management

The Physics Derivation Graph relies on a diverse set of skills. To speed up the development process we could identify separable tasks and then spread the tasks among a team of contributors. Collaboration requires coordination, and that coordination can be top-down or organic or a mixture.

This post focuses on the top-down approach and assumes an omniscient view. 

A standard data structure in project management is the Gantt chart. A Gantt chart uses information about tasks, task dependencies, task durations, and dates to create a visualization associated with a project.

task ID task description task duration [days] depends on tasks earliest start date
14235 something useful 3 N/A 2022-03-01
25532 hard work 2 [14235] N/A
3456252 keeping busy 3 [25532] N/A

That table can be visualized with tasks versus time:

Visualization of a Gantt chart. Four tasks. Task 2 and 3 depend on task 1 being completed. Task 4 depends on task 2 being completed.

 

That data structure doesn't account for staffing, skills, equipment, or budget. The  Gantt chart doesn't account for uncertainty of task duration, nor alternative paths.

Gantt charts present a single path

Project management involves contingency planning.  

IF this THEN
   that 
ELSE
   other

Every conditional statement is a branching of possible paths, each a separate Gantt chart.

A single Gantt chart is a snapshot of a single path.


Staffing, budget, equipment, skills, uncertainty

Augmenting the basic Gantt chart means extending the table data structure to something like
task ID task description task duration [days] depends on tasks earliest start date depends on equipment minimum skill set and level uncertainty in duration [days]
14235 something useful 3 N/A 2022-03-01 [Photoshop] photo editing, intermediate +/-1
25532 hard work 2 [14235] N/A [Excel] math, beginner; text editing, beginner +2
3456252 keeping busy 3 [25532] N/A Chrome browser clicking on links, beginner 0

That information needs to be augmented with a cost table for equipment:

equipment cost per day [USD] acquisition cost [USD]
Photoshop 0 100
Excel 0 20
Chrome browser 0 0

Lastly, we need people who can do the tasks.

person name hourly cost [USD] skill and level dates available
Alice 20 Excel, beginner; text editing, intermediate [2022-01-02, 2022-01-03, 2022-01-04]
Bob 15 Excel, intermediate; Math, beginner [2022-02-01, 2022-02-15, 2022-02-24]
Charlie 24 photo editing, beginner [2022-01-12, 2022-01-23, 2022-01-24]

Caveat: the above model is focused exclusively on experts doing tasks using equipment. The model does not account for managers, nor does the model account for support staff. The model could include any scope of tasks, but a boundary needs to be drawn somewhere to avoid becoming Seldon's psychohistory. The problem of tasks existing outside the model is the risk that out-of-model tasks block project progress or alter the project cost. Anything outside the model could be an invisible dependency.

Derived project views

The following views can be derived from the three tables above:
  • standard task Gantt chart
  • project duration histogram. (Each "task duration" has an uncertainty that can be aggregated and results in variance.)
  • per-person activity schedule -- who works on what when. A prioritization based on task dependencies and when people with skills are available
  • cost per day -- the spend rate

Blockers inhibiting the omniscient project management view

Even though the issue can be formulated into a set of data structures

Blockers are the amount of time needed to 

  • gather the relevant information and keep the information up-to-date as the situation evolves
  • document the information and tune the model
  • survey the skills of the workforce
  • have the workforce track their time spent on a task
  • define each task, along with a "definition of done"
  • track costs of the workforce and equipment
  • identifying conditional paths and tracking which alternative was selected

Blockers that aren't time-based:

  • inability to break project into atomic tasks (where "atomic" is based on skill set and skill level)
  • can break project into tasks, but unable to identify relevant skills for task

The centralization of this model is merely to construct a narrative. Software like Atlassian's Jira is a way of distributing the task tracking rather than try to administer the project in a centralized top-down approach.

Conclusion

The top-down omniscient view of project management is an unrealistic fantasy. However, it might be a helpful artifact for negotiation among decision makers. The alternative (where decision makers don't have a quantitative model to argue about) devolves into reliance on personal relationships, turf battles, and political factions. Bureaucratic processes evolve as a substitute for the lack of top-down omniscient view of project management. 

Saturday, April 22, 2023

intermittent Internal Server Error on webpage - activity log of diagnosis

My login to the server provides an overview,

Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-45-generic x86_64)

  System information as of Sat Apr 22 23:49:00 UTC 2023

  System load:                      0.09
  Usage of /:                       70.8% of 24.06GB
  Memory usage:                     72%
  Swap usage:                       0%
  Processes:                        131
  Users logged in:                  1

I first confirmed disk utilization.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            474M     0  474M   0% /dev
tmpfs            99M  1.2M   97M   2% /run
/dev/vda1        25G   18G  7.1G  71% /

I next looked at the error logs. When the error occurred, the fault trace ended with

  File "/home/appuser/app/controller.py", line 3424, in review_derivation
    dat = clib.read_db(path_to_db)
  File "/home/appuser/app/common_lib.py", line 142, in read_db
    for row in cur.execute("SELECT * FROM data"):
sqlite3.OperationalError: no such table: data
  
Probably a race condition in the Python? If so, this problem is due to my hack of writing the database to SQL. My motivation to resolve this issue is low because I had intended to switch to Neo4j. I haven't complete the migration from JSON/SQL to Neo4j yet.

I used top and saw 2 and 3 instances of gunicorn

I checked for redundant Docker containers running but saw only what's expected:

$ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED       STATUS       PORTS                                                                      NAMES
632ee101f714   v7_pickle_web_interface_nginx   "nginx -g 'daemon of…"   4 weeks ago   Up 4 weeks   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   v7_pickle_web_interface_nginx_1
80bb13ec66ca   v7_pickle_web_interface_flask   "gunicorn --reload -…"   4 weeks ago   Up 4 weeks   5000/tcp                                                                   v7_pickle_web_interface_flask_1

Sunday, March 26, 2023

OpenModelica on Mac using Docker -- command line and GUI

"Modelica is a non-proprietary, object-oriented, equation based language to conveniently model complex physical systems."

(source)

"Modelica is the language for solving systems of ordinary differential, algebraic, continuous, and discrete equations."

(source)

https://modelica.org/tools.html
https://openmodelica.org/

Source code: https://github.com/OpenModelica/OpenModelica

Running Modelica using Docker

In a terminal on your Mac, run:

ip=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}') 
Xhost +$ip 
docker run -it -v ~/.Xauthority:/root/.Xauthority -e DISPLAY=$ip:0  openmodelica/openmodelica:v1.20.0-gui /bin/bash
Source: Docker instructions, which also had the commands for linux:
docker run -it --rm -e "HOME=$HOME" -e "DISPLAY=$DISPLAY" --network=host --user "$UID" -v "$HOME:$HOME" -v "$PWD:$PWD" -w "$PWD" openmodelica/openmodelica:v1.20.0-gui
From https://openmodelica.org/useresresources/userdocumentation/ see https://openmodelica.org/doc/OpenModelicaUsersGuide/OpenModelicaUsersGuide-latest.pdf After launching the Docker container, in the command prompt I ran
root@54d835361dc8:/# OMShell-terminal
OMShell Copyright 1997-2022, Open Source Modelica Consortium (OSMC)
Distributed under OMSC-PL and GPL, see www.openmodelica.org

To get help on using OMShell and OpenModelica, type "help()" and press enter
Set shortOutput flag: true

>>> x := 1:12
{1,2,3,4,5,6,7,8,9,10,11,12}

>>> quit()
I launched the GUI using
# OMEdit 
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast

On Mac using XQuartz I see


Questions: https://stackoverflow.com/questions/tagged/openmodelica


Comment threads:

Saturday, February 4, 2023

Medical knowledge as a probabilistic decision tree

Medical reasoning can sound like a large decision tree ("if this then that"). This post documents the nuances that render a decision tree data structure inadequate. 

  • Nuances about input data
    • Extracting facts relevant to diagnosis is challenging. 
      • A parent of an infant may report their baby is not eating much, but does that mean the baby is refusing food, sleepy, crying, vomiting...
    • Looking merely at the current symptoms is inadequate. Patient history matters. Family history can matter.
    • distinguishing irrelevant information is important. Temporal correlation does not mean causation.
    • There are varying degrees of uncertainty about data quality. For example, I would discount the data from a person who does not tell their story well, and put a greater weight on a lab test. But for someone who tells their story well, the history features may sway more than their lab. 
    • How questions get asked matter
      • "Does your family have a history of arrhythmia?" will get a "no," but
        • "Have people in your family drowned or died in a car accident?" would get a yes because arrhythmia can lead to drowning and accidents.
        • "did any babies in the family have hearing problems?"
    • Patients don't know the precise medical jargon
      • "Does your family have a history heart attacks?" is likely actually a different medical cause when the patient doesn't know the jargon or the specifics about cholesterol.
    • Humans are not uniform, and the differences can matter. 
      • "male versus female" isn't an easy binary in all cases
      • "age" isn't a scalar positive number
        • age since birth ("post-gestational age")
        • age since conception ("gestational age")
        • cognitive ability versus temporal age versus physical dexterity

    • Decisions
      • A lot of the reasoning is probabilistic rather than definitive. The exact probabilities are also unknown -- one is dealing with relative probabilities. 
      • Discarding data should be weighted by the consequence of removing the data
    • Causal explanations based on physiology can be used to discard information, or give insight about additional symptoms to ask about. [Writing down all of physiology is a separate task.]
    • The differential list should include not only a ranking of most likely diagnoses but also diagnoses that one should not miss. This is one area where some doctors say that doctors are better than nurse practitioners. The nurse practitioner tells the patient the most likely diagnosis, but the doctor knows other diagnoses that are less likely, but should not be missed and counsel accordingly. In other words, just guessing based on a "most likely" diagnosis is insufficient. Are there "do not miss" outcomes that could be explanatory for the diagnosis?
    • Suggestions for action by the patient includes, "what to do if this other symptom arises" or "what to do if this symptom persists"



    The clinical decision analysis using decision tree
    doi 10.4178/epih/e2014025

    Sunday, December 18, 2022

    MONTEREY PHOENIX grammar, keywords, and common patterns

    MONTEREY PHOENIX grammar

    In this table A is an event and B and C are events in A
    description of pattern grammar rule
    Ordered sequence of events (B followed by C) A: B C;
    Alternative events (B or C) A: ( B | C );
    Optional event (B or no event at all) A: [ B ];
    Ordered sequence of zero or more events B A: (* B *);
    Ordered sequence of one or more events B A: (+ B +);
    Unordered set of events B and C (B and C may happen concurrently) A: { B, C };
    Unordered set of zero or more events B A: {* B *};
    Unordered set of one or more events B A: {+ B +};

     

    description of pattern grammar rule
    name of model SCHEMA
    Identifies an actor ROOT
    event precedence across swimlanes COORDINATE
    if and only if <->
    <!>
    a condition that each valid trace should satisfy ENSURE
    FOREACH
    BUILD
    DISJ
    MARK
    ONFAIL
    CHECK
    AFTER
    EXISTS
    MAP
    CLEAR
    SHOW
    message box that prints text SAY("hello world")

    Common pattern: branching logic in swimlane

    ROOT Driver: enters_car (starts_car | exits_car);
    starts_car:  move_forward
                 stops_car
                 exits_car;
    

    Common pattern: dependency across swimlanes

    COORDINATE
    $a: turn_steering_wheel_right FROM driver,
    $b: right FROM front_tires
    DO 
    ADD $a PRECEDES $b; 
    OD;
    

    Common pattern: shared events

    Driver, Car SHARE ALL move_forward, 
                          turn_left, 
                          turn_right;
    

    MONTEREY PHOENIX model checker

    In this blog post I discuss a model involving two actors: a car and a driver. Each actor has an associated set of actions they can take.

    For a video walkthrough of this code, see https://youtu.be/r6aPKem6WLs

    Using the web interface https://firebird.nps.edu/, run this model

    SCHEMA using_a_car /* model title */
    
    /* Actor Behaviors */
    
    /* purple text is an MP keyword,
       actors are green text,
       atomic events are blue */
    
    /* with space as the separator, 
       the events (blue) are an ordered sequence */
    
    ROOT Driver: enters_car 
                 starts_car
                 move_forward turn_left turn_right 
                 stops_car
                 exits_car;
    
    ROOT Car: off
              starting 
              running_idle
              move_forward
              turn_left
              turn_right
              shutting_down;
    

    Introduce branching logic

    SCHEMA using_a_car /* model title */
    
    /* Actor Behaviors */
    
    /* purple text is an MP keyword,
       actors are green text,
       atomic events are blue,
       composit events are orange */
    
    /* 
          ( A | B ) are alternative events 
    */
    
    ROOT Driver: enters_car (starts_car | exits_car);
    starts_car:  move_forward turn_left turn_right 
                 stops_car
                 exits_car;
    
    ROOT Car: off
              starting 
              running_idle
              move_forward
              turn_left
              turn_right
              shutting_down;
    

    Events among actors are coordinated

    SCHEMA using_a_car /* model title */
    
    /* Actor Behaviors */
    
    /* purple text is an MP keyword,
       actors are green text,
       atomic events are blue,
       composit events are orange */
    
    ROOT Driver: enters_car (starts_car | exits_car);
    starts_car:  move_forward turn_left turn_right 
                 stops_car
                 exits_car;
    
    ROOT Car: off
              starting 
              running_idle
              move_forward
              turn_left
              turn_right
              shutting_down;
    
    /* the two processes are related by specific events */
    
    COORDINATE
    $a: starts_car FROM Driver,
    $b: starting FROM Car
    DO
    ADD $a PROCEEDS $b;
    OD;
    
    COORDINATE
    $a: stops_car FROM Driver,
    $b: shutting_down FROM Car
    DO
    ADD $a PROCEEDS $b;
    OD;
    

    Actors may take zero or more actions

    SCHEMA using_a_car /* model title */
    
    /* Actor Behaviors */
    
    /* purple text is an MP keyword,
       actors are green text,
       atomic events are blue,
       composit events are orange */
    
    /* 
        {* *} is an unordered set of zero or more events 
    */
    
    ROOT Driver: enters_car (starts_car | exits_car);
    starts_car:  {* ( move_forward | 
                      turn_left | 
                      turn_right ) *}
                 stops_car
                 exits_car;
    
    ROOT Car: off
              starting 
              running_idle
              {* ( move_forward | 
                   turn_left | 
                   turn_right ) *}
              shutting_down;
    
    COORDINATE
    $a: starts_car FROM Driver,
    $b: starting FROM Car
    DO
    ADD $a PROCEEDS $b;
    OD;
    
    COORDINATE
    $a: stops_car FROM Driver,
    $b: shutting_down FROM Car
    DO
    ADD $a PROCEEDS $b;
    OD;
    

    Events are shared among actors

    SCHEMA using_a_car /* model title */
    
    /* Actor Behaviors */
    
    /* purple text is an MP keyword,
       actors are green text,
       atomic events are blue,
       composit events are orange */
    
    ROOT Driver: enters_car (starts_car | exits_car);
    starts_car:  {* ( move_forward | 
                      turn_left | 
                      turn_right ) *}
                 stops_car
                 exits_car;
    
    ROOT Car: off
              starting 
              running_idle
              {* ( move_forward | 
                   turn_left | 
                   turn_right ) *}
              shutting_down;
    
    Driver, Car SHARE ALL move_forward, 
                          turn_left, 
                          turn_right;
    
    COORDINATE
    $a: starts_car FROM Driver,
    $b: starting FROM Car
    DO
    ADD $a PROCEEDS $b;
    OD;
    
    COORDINATE
    $a: stops_car FROM Driver,
    $b: shutting_down FROM Car
    DO
    ADD $a PROCEEDS $b;
    OD;