Monday, July 20, 2020

United States Schools Should Only Reopen When It's Safe

We had a very long school board session last week in Dublin, CA. They’ve been about 4-6 hours consistently over the last year, not just from COVID. My wife watches most of them. It’s like reality TV. There are characters, plot lines, and rivalries. Our local school board is in turmoil as half of our town is newer construction on the east side of town and half of the town lives in established neighborhoods from the 1980’s and prior. We’re building a 2nd high school because our existing one has 3k kids and growing. The east side of town wants to keep the tax money flowing into new construction of schools in the east, such as the new high school. The west side wants a balance of investment as our existing schools are older and need renovation.

Last week the school board revealed a change in plans to not reopen our schools for in-person learning. The prior plan-of-record had each family choose among three fully supported options:
* In-person 5 days a week
* In-person 2 days a week, remote 3
* Full remote
You could choose to move towards more remote later, but you could not move towards more in-person until the school board’s assessment of what it takes to keep school safe changes. The updated plan is for full remote learning for everyone for the 1st 6 weeks of the school year, and likely beyond that. I believe this is the best solution right now given the safety risks to kids and staff.

I believe it’s important to establish guiding principles for how to address school during COVID-19. Here are mine:
1. Prioritize safety over inconvenience.
2. Public health experts define safety.
3. Do what works.

My views are impacted by my wife’s work as a nurse in San Quentin prison for the last month part time. About 2 months ago they had zero cases. Then there was a botched prison transfer where COVID-positive prisoners were brought in due to incompetent testing and transfer protocols. Within a short time there was a huge outbreak among the 4k prisoners. San Quentin prison went on full lockdown with prisoners unable to leave their cells except for very stringent protocols. They try to minimize any ability for prisoners and staff to interact to minimize the chance of spreading. The positivity rate is now 59% (see below) despite going on lockdown with a much lower positivity rate. When you are indoors with infected people, COVID-19 will spread. Just as it did on cruise ships, restaurants, buses and church choir practice. Gatherings indoors are not safe without many precautions. You don’t have to be within 6 feet.

Here is a fascinating map of how COVID-19 spread from one dining session in a restaurant in China where they did extensive contact tracing. Note that the air conditioning airflow impacted who got infected and who didn’t. It wasn’t strictly based on distance alone.

This diagram created from using Closed Circuit TV cameras in Hunan China shows how a bus passenger was infected with COVID-19 despite not boarding the bus until 30min after the infected passenger had got off the bus.

South Korea just reported a study indicating children 10 years and older spread COVID-19 just as well as adults.

I understand the reasons why many people want our schools to open to onsite teaching. I want to do it as soon as it’s safe, which I don’t think it is now. For example, the recent Economist article “Let Them Learn” has basic facts wrong. They suggest a policy with remote-only learning does more harm than the risk of COVID-19. The Economist claims do not make sense in the context of this new study from South Korea.

The South Korea shows that 10 year olds and above are just as likely to spread the disease as infected adults, and therefore I believe the Economist claim is wrong to say that since kids aren’t “especially likely” to spread it more than adults and therefore it is safe. The Economist's claim about school staff from Sweden not getting COVID-19 at higher rates in other jobs is not comforting. We do not want school staff and volunteers to contract COVID-19 from school at all. We cannot have family members getting infected by kids. If the staff and volunteers don’t feel it’s safe, then many of them won’t return and we cannot have schools be effective without healthy staff and volunteers.

There are models that are known-to-work, such as Taiwan. Taiwan has children in schools and their economy is largely back to normal. Their response involves a collection of measures including strict quarantine protocols for those crossing the national border, widespread mask wearing with enforced fines, widely available cheap and frequent testing with fast results (say 48 hours and under), and contact tracing.

Meanwhile in the United States we are not enforcing quarantine during border crossings. We do not have widespread testing that gets results under 48 hours. We do not have a culture of wearing face coverings or national government guidance to temporarily step up face coverings to get the virus under control. Our outbreaks are currently too large to do effective contact tracing. If we want to make schools safe and stay safe, I believe we need to have a national plan that addresses these problems. Thinking about school as an isolated scenario from other parts of life doesn’t make sense.

Some politicians are trying to force schools to reopen. I am confident that regardless of political affiliation, families of school-age children will resist the push to send their kids to school given the COVID-19 risks. It just doesn’t make any sense right now. We should adopt policies known-to-work by other advanced countries and recognize that we can have a thriving economy and schools only we put in place a system that suppresses the virus and keeps our people safe.

Friday, January 2, 2015

boot2docker Work-around for API Limit Error

update Jan 3rd, 2015: see bottom of this post for follow-up tweets from docker project 

i recently got a new laptop and had to reinstall some software. i kept having trouble with boot2docker. after running boot2docker up followed by docker version i kept getting an error about an API versions mismatch between the docker client and server. sure enough boot2docker ssh confirmed that i had an old server despite using the new boot2docker 1.4.1 download package. the standard boot2docker update procedure did not work with some cryptic command about an API rate limit being reached: FATA[0000] Error response from daemon: client and server don't have same version (client : 1.16, server: 1.15)

it turns out that boot2docker relies on github URLs to determine the latest version and too many unauthenticated users were invoking github from my NAT'd originating IP. boot2docker issue 481 is currently tracking this.
thankfully there is an easy work-around already available that works with the latest 1.4.1 release of docker shown below.
now my local docker workflow is all happy again.

update Jan 3rd, 2015: the docker team followed up yesterday explaining that the docker machine project was the preferred going forward approach and that boot2docker would be sunset. i was able to get it working, but the distributed machine binary requires a custom build of docker from a personal repository, which is certainly not ideal if you're concerned about security. they explained that was the path forward until new identity auth work was merged into docker with .

Monday, February 3, 2014

Remote Dependencies, Convenience, Risk and Other Considerations for Operating Distributed Systems

One deeply held principle by experienced distributed system operators that I have worked with is that you should have no external dependencies to your software other than the ties to minimum requirements of the OS such as common system libraries, utilities, and the kernel of the base OS. This approach should enable recreating a distributed system deployment without any dependencies on the outside world. When something goes wrong, you should have control over your own destiny. Reliance on any external dependency that is managed or hosted by someone else introduces risk that something outside your system can affect your ability to restore and recreate the system any time you need to.
To use a simple metaphor, imagine your system is represented by Jenga blocks and it falls over as Jenga towers inevitably do. However, instead of being able to rebuild your tower you find out that a mandatory required component at the base of your tower is missing or unavailable now no matter what you cannot rebuild the tower exactly how it was before. Your new tower is going to behave differently in unexpected ways and you might topple over because you do not understand all the behaviors when using different building blocks combined in a different way.
Some of the original designers of the software deployment project for Cloud Foundry named BOSH (Mark LucovskyVadim SpiwakDerek Collison) embraced this principle and tried to create a prescriptive framework that encouraged this approach. They had experience managing large scale distributed systems at Google (the web services APIs). Kent Skaar also did similar for SaaS provider Zendesk. Given a software release that references specific versions of multiple software packages (known as a BOSH release), an instantiation of that release (a BOSH deployment) can be reconstructed at any time with the deployment configuration (a BOSH deployment manifest), the base OS images (the BOSH stemcells) and the software release (the BOSH packages and job templates for applying configuration). at any point in time, properly implemented BOSH releases of large scale distributed systems can be recreated without external dependencies. That means this holds true even when the internet is unavailable.
BOSH does give you the framework hooks to break out of this prescriptive principle and use external dependencies or at least external dependency formats if you choose to for convenience or other reasons. Dr Nic Williams recently implemented tooling to use apt packages instead of compiling from source. another example: some of the Pivotal big data software intentionally targets CentOS/RHEL only and therefore only ships rpm packages rather than compiling Hadoop. A guiding principle is that you should be mindful of the tradeoffs you are making of convenience vs risk and tying your release to only one OS distributor.
Examples of the tradeoffs:
  • relying on an externally hosted package manager like apt-get could affect the availability or correctness of that dependency when you need it most
  • relying on debian packages could prevent someone from using your release unmodified with a CentOS image
A recent real-world example demonstrated the risk of an external dependency changing unexpectedly. The coreos/etcdproject that Cloud Foundry is using for storing stateful configuration data for the new Cloud Foundry Health Manager codebase had one of the dependencies (goraft/raft) force push to master of their git repository that overwrote some git history required by git to work properly. This situation has limited the flexibility of some users to make code modifications on several previous releases of Cloud Foundry without some tedious intervention.
A common reaction when learning about Cloud Foundry BOSH is to question the prescriptive guidance to compile from source when commonly used distributed package management systems exist in the Linux distributions. My recommendation is to understand the tradeoffs involved and make the best choice for your situation. You should explicitly call out external dependencies if you have them in your system. When you tower inevitably falls over, know how to rebuild it.

Saturday, February 1, 2014

How to Find Java Mission Control on OSX 10.9

i was excited to show some of my colleagues how great Java Mission Control is to debug, troubleshoot and monitor local and remote Java applications. On my OSX 10.9 laptop i installed the latest Oracle JDK 7, which now includes Java Mission Control with the JDK, and i expected to be able to type jmc on the command line. that didn't work and resulted in a command not found! OSX Finder wouldn't find jmc it either. i found a hint on an OTN community thread. the Java installer only put installation files in the obscure and hard to find /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/ instead of also placing a shortcut in /usr/local/bin which should link to /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/jmc that should change when you use /usr/libexec/java_home. you can see the install location by running this command:
$ find /Library/Java -name jmc
/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/bin/jmc
/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/lib/missioncontrol/Java Mission Control.app/Contents/MacOS/jmc

the solution i chose was to create this executable file in /usr/local/bin/jmc

#!/bin/bash

/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/bin/jmc

maybe it's because i had JDK 7 installed originally before Mission Control was included with it. i'm not sure, but i hope this helps someone else.

Thursday, December 19, 2013

Friendly BOSH Labels in vCenter

thanks to the BOSH team for showing me how to do this. Cloud Foundry BOSH automatically tags vSphere deployed VMs with various attributions including the job and index. this way instead of justing have a GUID as the name in vCenter, you can add additional columns that are already populated with the BOSH job and job index. we'll be adding this to the documentation for pivotal cf soon. click the image to see the full size.


Thursday, December 12, 2013

Send Interactive Commands to a Cloud Foundry App with websocketd

on my way home from work tonight i saw the tweet below about connecting STDIN and STDOUT from remote processes with websocket. i tried it out quickly locally and it worked streaming count.sh output of numbers from 1 to 10 sent to STDOUT 1 second apart over websocket using localhost. can i apply this to cloud foundry easily?  it turns out the answer is yes!
you can easily include the small linux 64bit websocketd binary and this bash.sh script with your app and remotely send commands over websocket that will execute in the app container and stream the STDOUT from the command back over websocket to the browser. this is helpful for sending commands like rake db:migrate or to explore the linux container file system after the buildpack has run. see the screenshot and video below. for the impatient, skip to about 3:00 of the short demo.

more instructions are on github. also see the websocketd project.




Sunday, December 8, 2013

Be Direct When You Communicate

"Be direct when communicating" is a common theme I've been hearing the last few days in Pivotal leadership discussions and other places.



When I listened to the Twitter CEO Dick Costolo fireside chat with PandoMonthly (start listening at around 35:30) it crystalized how important this is and how easy it is to fall out of this to appease the feelings of someone you're meeting with. Dick describes how he has a management / leadership training class where they do exercises for this and how many experienced people still yield to the temptation to "migrate along the Y-axis" and give up clarity instead of "optimize for the X-axis." Doing this definitely takes practice, but I'm going to remind myself to think of these axes in my communication. It's totally great if the person feels great about the discussion, but how they feel about it is not as important as receiving and understanding the message. There are certainly many conversations where someone will feel badly about the message, and that's a fine outcome if they receive the message and were unlikely to feel good about it under any circumstances.